[jira] [Commented] (FLINK-16641) Announce sender's backlog to solve the deadlock issue without exclusive buffers
[ https://issues.apache.org/jira/browse/FLINK-16641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350801#comment-17350801 ] Piotr Nowojski commented on FLINK-16641: Hey [~kevin.cyj], sorry for the delay. I'm currently busy with investigation of some performance issue. Once I get over with that, I will get back to this topic. > Announce sender's backlog to solve the deadlock issue without exclusive > buffers > --- > > Key: FLINK-16641 > URL: https://issues.apache.org/jira/browse/FLINK-16641 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Network >Reporter: Zhijiang >Assignee: Yingjie Cao >Priority: Major > Labels: pull-request-available > Fix For: 1.14.0 > > > This is the second ingredient besides FLINK-16404 to solve the deadlock > problem without exclusive buffers. > The scenario is as follows: > * The data in subpartition with positive backlog can be sent without doubt > because the exclusive credits would be feedback finally. > * Without exclusive buffers, the receiver would not request floating buffers > for 0 backlog. But when the new backlog is added into such subpartition, it > has no way to notify the receiver side without positive credits ATM. > * So it would result in waiting for each other between receiver and sender > sides to cause deadlock. The sender waits for credit to notify backlog and > the receiver waits for backlog to request floating credits. > To solve the above problem, the sender needs a separate message to announce > backlog sometimes besides existing `BufferResponse`. Then the receiver can > get this info to request floating buffers to feedback. > The side effect brought is to increase network transport delay and throughput > regression. We can measure how much it effects in existing micro-benchmark. > It might probably bear this effect to get a benefit of fast checkpoint > without exclusive buffers. We can give the proper explanations in respective > configuration options to let users make the final decision in practice. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22763) avro-confluent format does not allow for authorization credentials to be supplied to Confluent Schema Registry
[
https://issues.apache.org/jira/browse/FLINK-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samuel Fiddis updated FLINK-22763:
--
Description:
In PyFlink, attempting to connect to a avro-confluent kafka stream where the
Confluent Schema Registry requires authorization does not work.
Table API definition:
{code:java}
ddl_kafka_avro_confluent_source = f"""
CREATE TABLE gtt_records(
**table columsn**
) WITH (
'connector' = 'kafka',
'topic' = 'topic.avro-v1',
'properties.bootstrap.servers' =
'pkc-ldvj1.ap-southeast-2.aws.confluent.cloud:9092',
'properties.security.protocol' = 'SASL_SSL',
'properties.sasl.mechanism' = 'PLAIN',
'properties.sasl.jaas.config' =
'org.apache.kafka.common.security.plain.PlainLoginModule required
username="{KAFKA_API_KEY}" password="{KAFKA_API_SECRET}";',
'properties.basic.auth.credentials.source' = 'USER_INFO',
'properties.basic.auth.user.info' = '{SR_API_KEY}:{SR_API_SECRET}',
'key.format' = 'avro-confluent',
'key.avro-confluent.schema-registry.url' =
'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
'key.fields' = '**key fields**',
'value.format' = 'avro-confluent',
'value.avro-confluent.schema-registry.url' =
'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
'value.fields-include' = 'ALL',
'key.avro-confluent.schema-registry.subject' = 'topic.avro-v1-key',
'value.avro-confluent.schema-registry.subject' = 'topic.avro-v1-value'
) """{code}
Attempting to run a job with this table as a source results in a 401 error for
the Confluent Schema Registry:
{code:java}
2021-05-19 04:50:21,830 WARN org.apache.flink.runtime.taskmanager.Task
[] - Source: TableSourceScan(table=[[default_catalog,
default_database, gtt_records]], fields=[unique, direction, window_ts,
road_number, link_number, carriageway, version_no, window_local_date,
window_local_time, poll_ts, duration, traffic_duration, distance, link_length])
-> Sink: Sink(table=[default_catalog.default_database.kafka_messages],
fields=[unique, direction, window_ts, road_number, link_number, carriageway,
version_no, window_local_date, window_local_time, poll_ts, duration,
traffic_duration, distance, link_length]) (1/1)#0
(7eddc3a42dbcad0fc313bb6bdfa2c922) switched from RUNNING to FAILED with failure
cause: java.io.IOException: Failed to deserialize Avro record.at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
at
org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
at
org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:119)
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)Caused
by: java.io.IOException: Could not find schema with id 11 in registry
at
org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.readSchema(ConfluentSchemaRegistryCoder.java:77)
at
org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:73)
at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:103)
... 9 moreCaused by:
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
Unauthorized; error code: 401at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:660)
at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:642)
at
[jira] [Updated] (FLINK-22763) avro-confluent format does not allow for authorization credentials to be supplied to Confluent Schema Registry
[
https://issues.apache.org/jira/browse/FLINK-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samuel Fiddis updated FLINK-22763:
--
Description:
In PyFlink, attempting to connect to a avro-confluent kafka stream where the
Confluent Schema Registry requires authorization does not work.
Table API definition:
{code:java}
ddl_kafka_avro_confluent_source = f"""
CREATE TABLE gtt_records(
**table columsn**
) WITH (
'connector' = 'kafka',
'topic' = 'topic.avro-v1',
'properties.bootstrap.servers' =
'pkc-ldvj1.ap-southeast-2.aws.confluent.cloud:9092',
'properties.security.protocol' = 'SASL_SSL',
'properties.sasl.mechanism' = 'PLAIN',
'properties.sasl.jaas.config' =
'org.apache.kafka.common.security.plain.PlainLoginModule required
username="{KAFKA_API_KEY}" password="{KAFKA_API_SECRET}";',
'properties.basic.auth.credentials.source' = 'USER_INFO',
'properties.basic.auth.user.info' = '{SR_API_KEY}:{SR_API_SECRET}',
'key.format' = 'avro-confluent',
'key.avro-confluent.schema-registry.url' =
'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
'key.fields' = '**key fields**',
'value.format' = 'avro-confluent',
'value.avro-confluent.schema-registry.url' =
'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
'value.fields-include' = 'ALL',
'key.avro-confluent.schema-registry.subject' =
'data.google-travel-time.avro-v1-key',
'value.avro-confluent.schema-registry.subject' =
'data.google-travel-time.avro-v1-value'
) """{code}
Attempting to run a job with this table as a source results in a 401 error for
the Confluent Schema Registry:
{code:java}
2021-05-19 04:50:21,830 WARN org.apache.flink.runtime.taskmanager.Task
[] - Source: TableSourceScan(table=[[default_catalog,
default_database, gtt_records]], fields=[unique, direction, window_ts,
road_number, link_number, carriageway, version_no, window_local_date,
window_local_time, poll_ts, duration, traffic_duration, distance, link_length])
-> Sink: Sink(table=[default_catalog.default_database.kafka_messages],
fields=[unique, direction, window_ts, road_number, link_number, carriageway,
version_no, window_local_date, window_local_time, poll_ts, duration,
traffic_duration, distance, link_length]) (1/1)#0
(7eddc3a42dbcad0fc313bb6bdfa2c922) switched from RUNNING to FAILED with failure
cause: java.io.IOException: Failed to deserialize Avro record.at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
at
org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
at
org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:119)
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)Caused
by: java.io.IOException: Could not find schema with id 11 in registry
at
org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.readSchema(ConfluentSchemaRegistryCoder.java:77)
at
org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:73)
at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:103)
... 9 moreCaused by:
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
Unauthorized; error code: 401at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:660)
at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.getId(RestService.java:642)
[jira] [Commented] (FLINK-22762) cannot use set table.sql-dialect=hive;
[ https://issues.apache.org/jira/browse/FLINK-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350797#comment-17350797 ] JasonLee commented on FLINK-22762: -- Hi xueluo, I tested it according to your above process and no exception was found. Is there any other information? > cannot use set table.sql-dialect=hive; > --- > > Key: FLINK-22762 > URL: https://issues.apache.org/jira/browse/FLINK-22762 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.0 > Environment: flink 1.13 > hive 3.12 >Reporter: xueluo >Priority: Major > Attachments: image-2021-05-25-10-11-49-944.png, > image-2021-05-25-10-14-22-111.png > > > sh sql-client.sh > *CREATE CATALOG* myhive *WITH* ( > 'type' *=* 'hive', > 'default-database' = 'default', > 'hive-conf-dir' = '/data/hive/conf/' > ); > > USE CATALOG myhive; > set table.sql-dialect=hive; > then > show tables; or any command error > !image-2021-05-25-10-14-22-111.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22763) avro-confluent format does not allow for authorization credentials to be supplied to Confluent Schema Registry
[
https://issues.apache.org/jira/browse/FLINK-22763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Samuel Fiddis updated FLINK-22763:
--
Summary: avro-confluent format does not allow for authorization credentials
to be supplied to Confluent Schema Registry (was: avro-confluent format does
not allow for authorization credentials to be supplied to Confluent Schema
Registery)
> avro-confluent format does not allow for authorization credentials to be
> supplied to Confluent Schema Registry
> --
>
> Key: FLINK-22763
> URL: https://issues.apache.org/jira/browse/FLINK-22763
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Formats (JSON, Avro, Parquet, ORC,
> SequenceFile), Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Samuel Fiddis
>Priority: Minor
>
> In PyFlink, attempting to connect to a avro-confluent kafka stream where the
> Confluent Schema Registry requires authorization does not work.
> Table API definition:
> {code:java}
> ddl_kafka_avro_confluent_source = f"""
> CREATE TABLE gtt_records(
> **table columsn**
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic.avro-v1',
> 'properties.bootstrap.servers' =
> 'pkc-ldvj1.ap-southeast-2.aws.confluent.cloud:9092',
> 'properties.security.protocol' = 'SASL_SSL',
> 'properties.sasl.mechanism' = 'PLAIN',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.plain.PlainLoginModule required
> username="{KAFKA_API_KEY}" password="{KAFKA_API_SECRET";',
> 'properties.basic.auth.credentials.source' = 'USER_INFO',
> 'properties.basic.auth.user.info' = '{SR_API_KEY}:{SR_API_SECRET}',
> 'key.format' = 'avro-confluent',
> 'key.avro-confluent.schema-registry.url' =
> 'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
>
> 'key.fields' = '**key fields**',
> 'value.format' = 'avro-confluent',
> 'value.avro-confluent.schema-registry.url' =
> 'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
>
> 'value.fields-include' = 'ALL',
> 'key.avro-confluent.schema-registry.subject' =
> 'data.google-travel-time.avro-v1-key',
> 'value.avro-confluent.schema-registry.subject' =
> 'data.google-travel-time.avro-v1-value'
> ) """{code}
>
> Attempting to run a job with this table as a source results in a 401 error
> for the Confluent Schema Registery:
>
> {code:java}
> 2021-05-19 04:50:21,830 WARN org.apache.flink.runtime.taskmanager.Task
> [] - Source: TableSourceScan(table=[[default_catalog,
> default_database, gtt_records]], fields=[unique, direction, window_ts,
> road_number, link_number, carriageway, version_no, window_local_date,
> window_local_time, poll_ts, duration, traffic_duration, distance,
> link_length]) -> Sink:
> Sink(table=[default_catalog.default_database.kafka_messages], fields=[unique,
> direction, window_ts, road_number, link_number, carriageway, version_no,
> window_local_date, window_local_time, poll_ts, duration, traffic_duration,
> distance, link_length]) (1/1)#0 (7eddc3a42dbcad0fc313bb6bdfa2c922) switched
> from RUNNING to FAILED with failure cause: java.io.IOException: Failed to
> deserialize Avro record.at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
> at
> org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
> at
> org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
> at
> org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:119)
> at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
> at
> org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)Caused
> by: java.io.IOException: Could not find schema with id 11 in registry
> at
> org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.readSchema(ConfluentSchemaRegistryCoder.java:77)
> at
>
[jira] [Created] (FLINK-22763) avro-confluent format does not allow for authorization credentials to be supplied to Confluent Schema Registery
Samuel Fiddis created FLINK-22763:
-
Summary: avro-confluent format does not allow for authorization
credentials to be supplied to Confluent Schema Registery
Key: FLINK-22763
URL: https://issues.apache.org/jira/browse/FLINK-22763
Project: Flink
Issue Type: Bug
Components: API / Python, Formats (JSON, Avro, Parquet, ORC,
SequenceFile), Table SQL / API
Affects Versions: 1.13.0
Reporter: Samuel Fiddis
In PyFlink, attempting to connect to a avro-confluent kafka stream where the
Confluent Schema Registry requires authorization does not work.
Table API definition:
{code:java}
ddl_kafka_avro_confluent_source = f"""
CREATE TABLE gtt_records(
**table columsn**
) WITH (
'connector' = 'kafka',
'topic' = 'topic.avro-v1',
'properties.bootstrap.servers' =
'pkc-ldvj1.ap-southeast-2.aws.confluent.cloud:9092',
'properties.security.protocol' = 'SASL_SSL',
'properties.sasl.mechanism' = 'PLAIN',
'properties.sasl.jaas.config' =
'org.apache.kafka.common.security.plain.PlainLoginModule required
username="{KAFKA_API_KEY}" password="{KAFKA_API_SECRET";',
'properties.basic.auth.credentials.source' = 'USER_INFO',
'properties.basic.auth.user.info' = '{SR_API_KEY}:{SR_API_SECRET}',
'key.format' = 'avro-confluent',
'key.avro-confluent.schema-registry.url' =
'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
'key.fields' = '**key fields**',
'value.format' = 'avro-confluent',
'value.avro-confluent.schema-registry.url' =
'https://{SR_API_KEY}:{SR_API_SECRET}@psrc-lzvd0.ap-southeast-2.aws.confluent.cloud',
'value.fields-include' = 'ALL',
'key.avro-confluent.schema-registry.subject' =
'data.google-travel-time.avro-v1-key',
'value.avro-confluent.schema-registry.subject' =
'data.google-travel-time.avro-v1-value'
) """{code}
Attempting to run a job with this table as a source results in a 401 error for
the Confluent Schema Registery:
{code:java}
2021-05-19 04:50:21,830 WARN org.apache.flink.runtime.taskmanager.Task
[] - Source: TableSourceScan(table=[[default_catalog,
default_database, gtt_records]], fields=[unique, direction, window_ts,
road_number, link_number, carriageway, version_no, window_local_date,
window_local_time, poll_ts, duration, traffic_duration, distance, link_length])
-> Sink: Sink(table=[default_catalog.default_database.kafka_messages],
fields=[unique, direction, window_ts, road_number, link_number, carriageway,
version_no, window_local_date, window_local_time, poll_ts, duration,
traffic_duration, distance, link_length]) (1/1)#0
(7eddc3a42dbcad0fc313bb6bdfa2c922) switched from RUNNING to FAILED with failure
cause: java.io.IOException: Failed to deserialize Avro record.at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:106)
at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:46)
at
org.apache.flink.api.common.serialization.DeserializationSchema.deserialize(DeserializationSchema.java:82)
at
org.apache.flink.streaming.connectors.kafka.table.DynamicKafkaDeserializationSchema.deserialize(DynamicKafkaDeserializationSchema.java:119)
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:179)
at
org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:142)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:826)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:269)Caused
by: java.io.IOException: Could not find schema with id 11 in registry
at
org.apache.flink.formats.avro.registry.confluent.ConfluentSchemaRegistryCoder.readSchema(ConfluentSchemaRegistryCoder.java:77)
at
org.apache.flink.formats.avro.RegistryAvroDeserializationSchema.deserialize(RegistryAvroDeserializationSchema.java:73)
at
org.apache.flink.formats.avro.AvroRowDataDeserializationSchema.deserialize(AvroRowDataDeserializationSchema.java:103)
... 9 moreCaused by:
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
Unauthorized; error code: 401at
org.apache.flink.avro.registry.confluent.shaded.io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
at
[GitHub] [flink] wuchong closed pull request #10880: [FLINK-15469][SQL] Update UpsertStreamTableSink and RetractStreamTabl…
wuchong closed pull request #10880: URL: https://github.com/apache/flink/pull/10880 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] cuspymd commented on a change in pull request #15986: [FLINK-22155][table] EXPLAIN statement should validate insert and query separately
cuspymd commented on a change in pull request #15986:
URL: https://github.com/apache/flink/pull/15986#discussion_r638441450
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/operations/SqlToOperationConverter.java
##
@@ -844,16 +843,19 @@ private Operation convertShowViews(SqlShowViews
sqlShowViews) {
return new ShowViewsOperation();
}
-/** Convert EXPLAIN statement. */
-private Operation convertExplain(SqlExplain sqlExplain) {
-Operation operation = convertSqlQuery(sqlExplain.getExplicandum());
-
-if (sqlExplain.getDetailLevel() != SqlExplainLevel.EXPPLAN_ATTRIBUTES
-|| sqlExplain.getDepth() != SqlExplain.Depth.PHYSICAL
-|| sqlExplain.getFormat() != SqlExplainFormat.TEXT) {
-throw new TableException("Only default behavior is supported now,
EXPLAIN PLAN FOR xx");
+/** Convert RICH EXPLAIN statement. */
+private Operation convertRichExplain(SqlRichExplain sqlExplain) {
+Operation operation;
+SqlNode sqlNode = sqlExplain.getStatement();
+if (sqlNode instanceof RichSqlInsert) {
+operation = convertSqlInsert((RichSqlInsert) sqlNode);
+} else if (sqlNode instanceof SqlSelect) {
+operation = convertSqlQuery(sqlExplain.getStatement());
Review comment:
Can't `sqlNode` be passed as a parameter?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-22760) HiveParser::setCurrentTimestamp fails with hive-3.1.2
[ https://issues.apache.org/jira/browse/FLINK-22760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Li reassigned FLINK-22760: -- Assignee: 罗宇侠 (was: Rui Li) > HiveParser::setCurrentTimestamp fails with hive-3.1.2 > - > > Key: FLINK-22760 > URL: https://issues.apache.org/jira/browse/FLINK-22760 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: Rui Li >Assignee: 罗宇侠 >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22762) cannot use set table.sql-dialect=hive;
[ https://issues.apache.org/jira/browse/FLINK-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-22762: Component/s: (was: API / Core) Connectors / Hive > cannot use set table.sql-dialect=hive; > --- > > Key: FLINK-22762 > URL: https://issues.apache.org/jira/browse/FLINK-22762 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.13.0 > Environment: flink 1.13 > hive 3.12 >Reporter: xueluo >Priority: Major > Attachments: image-2021-05-25-10-11-49-944.png, > image-2021-05-25-10-14-22-111.png > > > sh sql-client.sh > *CREATE CATALOG* myhive *WITH* ( > 'type' *=* 'hive', > 'default-database' = 'default', > 'hive-conf-dir' = '/data/hive/conf/' > ); > > USE CATALOG myhive; > set table.sql-dialect=hive; > then > show tables; or any command error > !image-2021-05-25-10-14-22-111.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] paul8263 commented on a change in pull request #15927: [FLINK-22639][runtime] ClassLoaderUtil cannot print classpath of Flin…
paul8263 commented on a change in pull request #15927:
URL: https://github.com/apache/flink/pull/15927#discussion_r638417597
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoadersTest.java
##
@@ -208,6 +212,50 @@ public void testRepeatedParentFirstPatternClass() throws
Exception {
childClassLoader.close();
}
+@Test
+public void testGetClassLoaderInfo() throws Exception {
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+String formattedURL = ClassLoaderUtil.formatURL(childCodePath);
+
+assertEquals(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+"URL ClassLoader:" + formattedURL);
+
+childClassLoader.close();
+}
+
+@Test
+public void testGetClassLoaderInfoWithClassLoaderClosed() throws Exception
{
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+childClassLoader.close();
+
+assertThat(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+is(startsWith("Cannot access classloader info due to an
exception.")));
Review comment:
It was fixed in c927703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] paul8263 commented on a change in pull request #15927: [FLINK-22639][runtime] ClassLoaderUtil cannot print classpath of Flin…
paul8263 commented on a change in pull request #15927:
URL: https://github.com/apache/flink/pull/15927#discussion_r638417384
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoadersTest.java
##
@@ -208,6 +212,50 @@ public void testRepeatedParentFirstPatternClass() throws
Exception {
childClassLoader.close();
}
+@Test
+public void testGetClassLoaderInfo() throws Exception {
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+String formattedURL = ClassLoaderUtil.formatURL(childCodePath);
+
+assertEquals(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+"URL ClassLoader:" + formattedURL);
+
+childClassLoader.close();
Review comment:
I think there is no need to ensure it is closed. So I keep it the same
as other test cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-22663) Release YARN resource very slow when cancel the job after some NodeManagers shutdown
[
https://issues.apache.org/jira/browse/FLINK-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350763#comment-17350763
]
Yang Wang commented on FLINK-22663:
---
The reason why decreasing the value of "taskmanager.registration.timeout" could
work is the TaskManager containers running on the dead node tried to connect
with the Flink ResourceManager. The timeout could control when it will give up
retrying and then exit.
Checking the logs of live NodeManager could help to confirm that when the
TaskManager is terminated.
Maybe we could introduce a config option to control whether to clean up running
containers when stopping the NMClient. By default, it is configured to true,
because this will make the TaskManager containers terminated faster.
> Release YARN resource very slow when cancel the job after some NodeManagers
> shutdown
>
>
> Key: FLINK-22663
> URL: https://issues.apache.org/jira/browse/FLINK-22663
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.2
>Reporter: Jinhong Liu
>Priority: Major
> Labels: YARN
>
> When I test flink on YARN, there is a case that may cause some problems.
> Hadoop Version: 2.7.3
> Flink Version: 1.12.2
> I deploy a flink job on YARN, when the job is running I stop one NodeManager,
> after one or two minutes, the job is auto recovered. But in this situation,
> if I cancel the job, the containers cannot be released immediately, there are
> still some containers that are running include the app master. About 5
> minutes later, these containers exit, and about 10 minutes later the app
> master exit.
> I check the log of app master, seems it try to stop the containers on the
> NodeManger which I have already stopped.
> {code:java}
> 2021-05-14 06:15:17,389 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job class
> tv.freewheel.reporting.fastlane.Fastlane$ (da883ab39a7a82e4d45a3803bc77dd6f)
> switched from state CANCELLING to CANCELED.
> 2021-05-14 06:15:17,389 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Stopping
> checkpoint coordinator for job da883ab39a7a82e4d45a3803bc77dd6f.
> 2021-05-14 06:15:17,390 INFO
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] -
> Shutting down
> 2021-05-14 06:15:17,408 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Job
> da883ab39a7a82e4d45a3803bc77dd6f reached globally terminal state CANCELED.
> 2021-05-14 06:15:17,409 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Shutting
> down cluster with state CANCELED, jobCancelled: true, executionMode: DETACHED
> 2021-05-14 06:15:17,409 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting
> YarnJobClusterEntrypoint down with application status CANCELED. Diagnostics
> null.
> 2021-05-14 06:15:17,409 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting
> down rest endpoint.
> 2021-05-14 06:15:17,420 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Stopping the JobMaster for job class
> tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f).
> 2021-05-14 06:15:17,422 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing
> cache directory
> /tmp/flink-web-af72a00c-0ddd-4e5e-a62c-8244d6caa552/flink-web-ui
> 2021-05-14 06:15:17,432 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> http://ip-10-23-19-197.ec2.internal:43811 lost leadership
> 2021-05-14 06:15:17,432 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down
> complete.
> 2021-05-14 06:15:17,436 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Shut down cluster because application is in CANCELED, diagnostics null.
> 2021-05-14 06:15:17,436 INFO org.apache.flink.yarn.YarnResourceManagerDriver
> [] - Unregister application from the YARN Resource Manager with
> final status KILLED.
> 2021-05-14 06:15:17,458 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending
> SlotPool.
> 2021-05-14 06:15:17,458 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Close ResourceManager connection
> 493862ba148679a4f16f7de5ffaef665: Stopping JobMaster for job class
> tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f)..
> 2021-05-14 06:15:17,458 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping
> SlotPool.
> 2021-05-14 06:15:17,482 INFO
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl[] - Waiting for
> application to be
[GitHub] [flink] flinkbot edited a comment on pull request #15994: [hotfix] [docs] Fix typo in JDBC Execution Options Documentation
flinkbot edited a comment on pull request #15994: URL: https://github.com/apache/flink/pull/15994#issuecomment-847462495 ## CI report: * 627c38a4894445368e1a9d249dff9a588d089e02 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18296) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15771: [FLINK-18934] Idle stream does not advance watermark in connected stream
flinkbot edited a comment on pull request #15771: URL: https://github.com/apache/flink/pull/15771#issuecomment-826939649 ## CI report: * 77caa970a0350585ebeb226e2d823e74b12511be Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18295) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] paul8263 commented on a change in pull request #15927: [FLINK-22639][runtime] ClassLoaderUtil cannot print classpath of Flin…
paul8263 commented on a change in pull request #15927:
URL: https://github.com/apache/flink/pull/15927#discussion_r638418295
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoadersTest.java
##
@@ -208,6 +212,64 @@ public void testRepeatedParentFirstPatternClass() throws
Exception {
childClassLoader.close();
}
+@Test
+public void testGetClassLoaderInfo() {
+final String className = FlinkUserCodeClassLoadersTest.class.getName();
+final String parentFirstPattern = className.substring(0,
className.lastIndexOf('.'));
+
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {parentFirstPattern},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+String formattedURL = null;
+try {
+formattedURL = ClassLoaderUtil.formatURL(childCodePath);
+} catch (IOException e) {
+// Should not fail here
+fail();
+}
+
+assertEquals(
+ClassLoaderUtil.getUserCodeClassLoaderInfo(childClassLoader),
+"URL ClassLoader:" + formattedURL);
+}
+
+@Test
+public void testGetClassLoaderInfoWithClassLoaderClosed() {
+final String className = FlinkUserCodeClassLoadersTest.class.getName();
+final String parentFirstPattern = className.substring(0,
className.lastIndexOf('.'));
+
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {parentFirstPattern},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+try {
+childClassLoader.close();
+} catch (IOException e) {
+fail("Cannot close child classloader");
+}
Review comment:
It was fixed in c927703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] paul8263 commented on a change in pull request #15927: [FLINK-22639][runtime] ClassLoaderUtil cannot print classpath of Flin…
paul8263 commented on a change in pull request #15927:
URL: https://github.com/apache/flink/pull/15927#discussion_r638418219
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/execution/librarycache/FlinkUserCodeClassLoadersTest.java
##
@@ -208,6 +212,64 @@ public void testRepeatedParentFirstPatternClass() throws
Exception {
childClassLoader.close();
}
+@Test
+public void testGetClassLoaderInfo() {
+final String className = FlinkUserCodeClassLoadersTest.class.getName();
+final String parentFirstPattern = className.substring(0,
className.lastIndexOf('.'));
+
+final ClassLoader parentClassLoader = getClass().getClassLoader();
+
+final URL childCodePath =
getClass().getProtectionDomain().getCodeSource().getLocation();
+
+final URLClassLoader childClassLoader =
+FlinkUserCodeClassLoaders.childFirst(
+new URL[] {childCodePath},
+parentClassLoader,
+new String[] {parentFirstPattern},
+NOOP_EXCEPTION_HANDLER,
+true);
+
+String formattedURL = null;
+try {
+formattedURL = ClassLoaderUtil.formatURL(childCodePath);
+} catch (IOException e) {
+// Should not fail here
+fail();
+}
Review comment:
It was fixed in c927703
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-21439) Adaptive Scheduler: Add support for exception history
[
https://issues.apache.org/jira/browse/FLINK-21439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350748#comment-17350748
]
John Phelan commented on FLINK-21439:
-
Do we need to create a {{FailureHandlingResult}} in order to create a
{{RootExceptionHistoryEntry}}?
> Adaptive Scheduler: Add support for exception history
> -
>
> Key: FLINK-21439
> URL: https://issues.apache.org/jira/browse/FLINK-21439
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.13.0
>Reporter: Matthias
>Assignee: John Phelan
>Priority: Major
> Labels: pull-request-available, reactive
> Time Spent: 3h
> Remaining Estimate: 0h
>
> {{SchedulerNG.requestJob}} returns an {{ExecutionGraphInfo}} that was
> introduced in FLINK-21188. This {{ExecutionGraphInfo}} holds the information
> about the {{ArchivedExecutionGraph}} and exception history information.
> Currently, it's a list of {{ErrorInfos}}. This might change due to ongoing
> work in FLINK-21190 where we might introduced a wrapper class with more
> information on the failure.
> The goal of this ticket is to implement the exception history for the
> {{AdaptiveScheduler}}, i.e. collecting the exceptions that caused restarts.
> This collection of failures should be forwarded through
> {{SchedulerNG.requestJob}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15877: [FLINK-22612][python] Restructure the coders in PyFlink
flinkbot edited a comment on pull request #15877: URL: https://github.com/apache/flink/pull/15877#issuecomment-836433504 ## CI report: * 44a0ac6333880616e97e1401aa9e2f38c0de087e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18288) * be2f68b4cd7086048d5a16c1ab5ee1661908a0c0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18297) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-22762) cannot use set table.sql-dialect=hive;
[ https://issues.apache.org/jira/browse/FLINK-22762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350731#comment-17350731 ] xingyuan cheng commented on FLINK-22762: I will pay attention to this issue > cannot use set table.sql-dialect=hive; > --- > > Key: FLINK-22762 > URL: https://issues.apache.org/jira/browse/FLINK-22762 > Project: Flink > Issue Type: Bug > Components: API / Core >Affects Versions: 1.13.0 > Environment: flink 1.13 > hive 3.12 >Reporter: xueluo >Priority: Major > Attachments: image-2021-05-25-10-11-49-944.png, > image-2021-05-25-10-14-22-111.png > > > sh sql-client.sh > *CREATE CATALOG* myhive *WITH* ( > 'type' *=* 'hive', > 'default-database' = 'default', > 'hive-conf-dir' = '/data/hive/conf/' > ); > > USE CATALOG myhive; > set table.sql-dialect=hive; > then > show tables; or any command error > !image-2021-05-25-10-14-22-111.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22038) Update TopN node to be without rowNumber if rowNumber field is projected out after TopN
[
https://issues.apache.org/jira/browse/FLINK-22038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andy updated FLINK-22038:
-
Labels: (was: stale-assigned)
> Update TopN node to be without rowNumber if rowNumber field is projected out
> after TopN
> ---
>
> Key: FLINK-22038
> URL: https://issues.apache.org/jira/browse/FLINK-22038
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: Andy
>Assignee: Andy
>Priority: Major
> Attachments: image-2021-03-30-16-03-09-876.png
>
>
> As describe in article
> [sql_queries|https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql/queries.html#no-ranking-output-optimization],
> an optimization way to improve performance of TopN is omitting rownum field
> in the outer SELECT clause of the Top-N query.
> However, some queries generated unexpected plan, even though we have followed
> the instructions in the documentation.
> {code:java}
> @Test
> def testRowNumberFiltered(): Unit = {
> util.addDataStream[(String, Long, Long, Long)](
> "T", 'uri, 'reqcount, 'start_time, 'bucket_id)
> val sql =
> """
> |SELECT
> | uri,
> | reqcount,
> | start_time
> |FROM
> | (
> |SELECT
> | uri,
> | reqcount,
> | rownum_2,
> | start_time
> |FROM
> | (
> |SELECT
> | uri,
> | reqcount,
> | start_time,
> | ROW_NUMBER() OVER (
> |PARTITION BY start_time
> |ORDER BY
> | reqcount DESC
> | ) AS rownum_2
> |FROM
> | (
> |SELECT
> |uri,
> |reqcount,
> |start_time,
> |ROW_NUMBER() OVER (
> |PARTITION BY start_time, bucket_id
> |ORDER BY
> |reqcount DESC
> |) AS rownum_1
> |FROM T
> | )
> |WHERE
> | rownum_1 <= 10
> | )
> |WHERE
> | rownum_2 <= 10
> |)
> |""".stripMargin
> util.verifyExecPlan(sql)
> }
> {code}
> For example, we expect both outer and inner TopN could use without rowNumber
> optimization in the above queries, however inner TopN is not as we expected.
> The logical plan and physical plan as following, we could find even though
> the rowNumber field is projected out after inner topN, inner topN still with
> rowNumber.
>
> !image-2021-03-30-16-03-09-876.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22038) Update TopN node to be without rowNumber if rowNumber field is projected out after TopN
[
https://issues.apache.org/jira/browse/FLINK-22038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350730#comment-17350730
]
Andy commented on FLINK-22038:
--
Hi, I'm still working on this, sorry for late response.
> Update TopN node to be without rowNumber if rowNumber field is projected out
> after TopN
> ---
>
> Key: FLINK-22038
> URL: https://issues.apache.org/jira/browse/FLINK-22038
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Reporter: Andy
>Assignee: Andy
>Priority: Major
> Attachments: image-2021-03-30-16-03-09-876.png
>
>
> As describe in article
> [sql_queries|https://ci.apache.org/projects/flink/flink-docs-stable/dev/table/sql/queries.html#no-ranking-output-optimization],
> an optimization way to improve performance of TopN is omitting rownum field
> in the outer SELECT clause of the Top-N query.
> However, some queries generated unexpected plan, even though we have followed
> the instructions in the documentation.
> {code:java}
> @Test
> def testRowNumberFiltered(): Unit = {
> util.addDataStream[(String, Long, Long, Long)](
> "T", 'uri, 'reqcount, 'start_time, 'bucket_id)
> val sql =
> """
> |SELECT
> | uri,
> | reqcount,
> | start_time
> |FROM
> | (
> |SELECT
> | uri,
> | reqcount,
> | rownum_2,
> | start_time
> |FROM
> | (
> |SELECT
> | uri,
> | reqcount,
> | start_time,
> | ROW_NUMBER() OVER (
> |PARTITION BY start_time
> |ORDER BY
> | reqcount DESC
> | ) AS rownum_2
> |FROM
> | (
> |SELECT
> |uri,
> |reqcount,
> |start_time,
> |ROW_NUMBER() OVER (
> |PARTITION BY start_time, bucket_id
> |ORDER BY
> |reqcount DESC
> |) AS rownum_1
> |FROM T
> | )
> |WHERE
> | rownum_1 <= 10
> | )
> |WHERE
> | rownum_2 <= 10
> |)
> |""".stripMargin
> util.verifyExecPlan(sql)
> }
> {code}
> For example, we expect both outer and inner TopN could use without rowNumber
> optimization in the above queries, however inner TopN is not as we expected.
> The logical plan and physical plan as following, we could find even though
> the rowNumber field is projected out after inner topN, inner topN still with
> rowNumber.
>
> !image-2021-03-30-16-03-09-876.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22762) cannot use set table.sql-dialect=hive;
xueluo created FLINK-22762: -- Summary: cannot use set table.sql-dialect=hive; Key: FLINK-22762 URL: https://issues.apache.org/jira/browse/FLINK-22762 Project: Flink Issue Type: Bug Components: API / Core Affects Versions: 1.13.0 Environment: flink 1.13 hive 3.12 Reporter: xueluo Attachments: image-2021-05-25-10-11-49-944.png, image-2021-05-25-10-14-22-111.png sh sql-client.sh *CREATE CATALOG* myhive *WITH* ( 'type' *=* 'hive', 'default-database' = 'default', 'hive-conf-dir' = '/data/hive/conf/' ); USE CATALOG myhive; set table.sql-dialect=hive; then show tables; or any command error !image-2021-05-25-10-14-22-111.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zoucao commented on pull request #15987: [FLINK-22437][table-planner-blink]set parallelism for not null filter
zoucao commented on pull request #15987: URL: https://github.com/apache/flink/pull/15987#issuecomment-847478203 Hi @godfreyhe，if you are free please leave some comments about this pr. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15877: [FLINK-22612][python] Restructure the coders in PyFlink
flinkbot edited a comment on pull request #15877: URL: https://github.com/apache/flink/pull/15877#issuecomment-836433504 ## CI report: * 44a0ac6333880616e97e1401aa9e2f38c0de087e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18288) * be2f68b4cd7086048d5a16c1ab5ee1661908a0c0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15994: [hotfix] [docs] Fix typo in JDBC Execution Options Documentation
flinkbot edited a comment on pull request #15994: URL: https://github.com/apache/flink/pull/15994#issuecomment-847462495 ## CI report: * 627c38a4894445368e1a9d249dff9a588d089e02 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18296) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15994: [hotfix] [docs] Fix typo in JDBC Execution Options Documentation
flinkbot commented on pull request #15994: URL: https://github.com/apache/flink/pull/15994#issuecomment-847462495 ## CI report: * 627c38a4894445368e1a9d249dff9a588d089e02 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15994: [hotfix] [docs] Fix typo in JDBC Execution Options Documentation
flinkbot commented on pull request #15994: URL: https://github.com/apache/flink/pull/15994#issuecomment-847457097 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 627c38a4894445368e1a9d249dff9a588d089e02 (Tue May 25 01:10:55 UTC 2021) **Warnings:** * Documentation files were touched, but no `docs/content.zh/` files: Update Chinese documentation or file Jira ticket. * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] alec-brooks opened a new pull request #15994: Fix typo in JDBC Execution Options Documentation
alec-brooks opened a new pull request #15994: URL: https://github.com/apache/flink/pull/15994 The method is `withBatchSize` ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16027) kafka connector's 'connector.topic' property should be optional rather than required
[ https://issues.apache.org/jira/browse/FLINK-16027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16027: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > kafka connector's 'connector.topic' property should be optional rather than > required > > > Key: FLINK-16027 > URL: https://issues.apache.org/jira/browse/FLINK-16027 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Reporter: Bowen Li >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15861) Change TableEnvironmentImpl to StreamTableEnvironmentImpl in python blink batch mode
[ https://issues.apache.org/jira/browse/FLINK-15861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15861: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Change TableEnvironmentImpl to StreamTableEnvironmentImpl in python blink > batch mode > - > > Key: FLINK-15861 > URL: https://issues.apache.org/jira/browse/FLINK-15861 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Huang Xingbo >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.9.4, 1.14.0 > > > We need to change TableEnvironmentImpl to StreamTableaEnvironmentImpl in > python blink batch mode otherwise we can't register TableFunction and > AggregateFunction in the mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15922) Show "Warn - received late message for checkpoint" only when checkpoint actually expired
[ https://issues.apache.org/jira/browse/FLINK-15922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15922: --- Labels: auto-deprioritized-major usability (was: stale-major usability) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Show "Warn - received late message for checkpoint" only when checkpoint > actually expired > > > Key: FLINK-15922 > URL: https://issues.apache.org/jira/browse/FLINK-15922 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.10.0 >Reporter: Stephan Ewen >Priority: Minor > Labels: auto-deprioritized-major, usability > Fix For: 1.14.0 > > > The message "Warn - received late message for checkpoint" is shown frequently > in the logs, also when a checkpoint was purposefully canceled. > In those case, this message is unhelpful and misleading. > We should log this only when the checkpoint is actually expired. > Meaning that when receiving the message, we check if we have an expired > checkpoint for that ID. If yes, we log that message, if not, we simply drop > the message. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22440) UpsertKafkaTableITCase fail due to create topic timeout
[
https://issues.apache.org/jira/browse/FLINK-22440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22440:
---
Labels: stale-major test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> UpsertKafkaTableITCase fail due to create topic timeout
> ---
>
> Key: FLINK-22440
> URL: https://issues.apache.org/jira/browse/FLINK-22440
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=17132=logs=b0097207-033c-5d9a-b48c-6d4796fbe60d=e8fcc430-213e-5cce-59d4-6942acf09121=6594
> {code:java}
> Apr 24 01:53:06 org.junit.runners.model.TestTimedOutException: test timed out
> after 30 seconds
> Apr 24 01:53:06 at java.lang.Object.wait(Native Method)
> Apr 24 01:53:06 at java.lang.Thread.join(Thread.java:1252)
> Apr 24 01:53:06 at java.lang.Thread.join(Thread.java:1326)
> Apr 24 01:53:06 at
> org.apache.kafka.clients.admin.KafkaAdminClient.close(KafkaAdminClient.java:541)
> Apr 24 01:53:06 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:96)
> Apr 24 01:53:06 at
> org.apache.kafka.clients.admin.Admin.close(Admin.java:79)
> Apr 24 01:53:06 at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestBase.createTestTopic(KafkaTableTestBase.java:99)
> Apr 24 01:53:06 at
> org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase.testTemporalJoin(UpsertKafkaTableITCase.java:81)
> Apr 24 01:53:06 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Apr 24 01:53:06 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 24 01:53:06 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 24 01:53:06 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 24 01:53:06 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 24 01:53:06 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 24 01:53:06 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 24 01:53:06 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 24 01:53:06 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> Apr 24 01:53:06 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> Apr 24 01:53:06 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 24 01:53:06 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Apr 24 01:53:06 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Apr 24 01:53:06 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Apr 24 01:53:06 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22662) YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint fail
[
https://issues.apache.org/jira/browse/FLINK-22662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22662:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint fail
>
>
> Key: FLINK-22662
> URL: https://issues.apache.org/jira/browse/FLINK-22662
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Critical
> Labels: stale-critical, test-stability
>
> {code:java}
> 2021-05-14T00:24:57.8487649Z May 14 00:24:57 [ERROR]
> testKillYarnSessionClusterEntrypoint(org.apache.flink.yarn.YARNHighAvailabilityITCase)
> Time elapsed: 34.667 s <<< ERROR!
> 2021-05-14T00:24:57.8488567Z May 14 00:24:57
> java.util.concurrent.ExecutionException:
> 2021-05-14T00:24:57.8489301Z May 14 00:24:57
> org.apache.flink.runtime.rest.util.RestClientException:
> [org.apache.flink.runtime.rest.handler.RestHandlerException:
> org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (610ed4b159ece04c8ee2ec40e7d0c143)
> 2021-05-14T00:24:57.8493142Z May 14 00:24:57 at
> org.apache.flink.runtime.rest.handler.job.JobExecutionResultHandler.propagateException(JobExecutionResultHandler.java:94)
> 2021-05-14T00:24:57.8495823Z May 14 00:24:57 at
> org.apache.flink.runtime.rest.handler.job.JobExecutionResultHandler.lambda$handleRequest$1(JobExecutionResultHandler.java:84)
> 2021-05-14T00:24:57.8496733Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:884)
> 2021-05-14T00:24:57.8497640Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:866)
> 2021-05-14T00:24:57.8498491Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2021-05-14T00:24:57.8499222Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2021-05-14T00:24:57.853Z May 14 00:24:57 at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:234)
> 2021-05-14T00:24:57.8500872Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> 2021-05-14T00:24:57.8501702Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> 2021-05-14T00:24:57.8502662Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> 2021-05-14T00:24:57.8503472Z May 14 00:24:57 at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
> 2021-05-14T00:24:57.8504269Z May 14 00:24:57 at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:1079)
> 2021-05-14T00:24:57.8504892Z May 14 00:24:57 at
> akka.dispatch.OnComplete.internal(Future.scala:263)
> 2021-05-14T00:24:57.8505565Z May 14 00:24:57 at
> akka.dispatch.OnComplete.internal(Future.scala:261)
> 2021-05-14T00:24:57.8506062Z May 14 00:24:57 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:191)
> 2021-05-14T00:24:57.8506819Z May 14 00:24:57 at
> akka.dispatch.japi$CallbackBridge.apply(Future.scala:188)
> 2021-05-14T00:24:57.8507418Z May 14 00:24:57 at
> scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> 2021-05-14T00:24:57.8508373Z May 14 00:24:57 at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:73)
> 2021-05-14T00:24:57.8509144Z May 14 00:24:57 at
> scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> 2021-05-14T00:24:57.8509972Z May 14 00:24:57 at
> scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> 2021-05-14T00:24:57.8510675Z May 14 00:24:57 at
> akka.pattern.PromiseActorRef.$bang(AskSupport.scala:572)
> 2021-05-14T00:24:57.8511376Z May 14 00:24:57 at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:23)
> 2021-05-14T00:24:57.851Z May 14 00:24:57 at
> akka.pattern.PipeToSupport$PipeableFuture$$anonfun$pipeTo$1.applyOrElse(PipeToSupport.scala:21)
> 2021-05-14T00:24:57.8513090Z
[jira] [Updated] (FLINK-16024) support filter pushdown in jdbc connector
[ https://issues.apache.org/jira/browse/FLINK-16024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16024: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > support filter pushdown in jdbc connector > - > > Key: FLINK-16024 > URL: https://issues.apache.org/jira/browse/FLINK-16024 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: Bowen Li >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22312) YARNSessionFIFOSecuredITCase>YARNSessionFIFOITCase.checkForProhibitedLogContents due to the heartbeat exception with Yarn RM
[
https://issues.apache.org/jira/browse/FLINK-22312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22312:
---
Labels: stale-critical test-stability (was: test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> YARNSessionFIFOSecuredITCase>YARNSessionFIFOITCase.checkForProhibitedLogContents
> due to the heartbeat exception with Yarn RM
>
>
> Key: FLINK-22312
> URL: https://issues.apache.org/jira/browse/FLINK-22312
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Reporter: Guowei Ma
>Priority: Critical
> Labels: stale-critical, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16633=logs=fc5181b0-e452-5c8f-68de-1097947f6483=6b04ca5f-0b52-511d-19c9-52bf0d9fbdfa=26614
> {code:java}
> 2021-04-15T22:11:39.5648550Z java.io.InterruptedIOException: Call interrupted
> 2021-04-15T22:11:39.5649145Z at
> org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1483)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5649823Z at
> org.apache.hadoop.ipc.Client.call(Client.java:1435)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5650488Z at
> org.apache.hadoop.ipc.Client.call(Client.java:1345)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5651387Z at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:227)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5652193Z at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:116)
> ~[hadoop-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5652675Z at com.sun.proxy.$Proxy32.allocate(Unknown
> Source) ~[?:?]
> 2021-04-15T22:11:39.5653478Z at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
> ~[hadoop-yarn-common-2.8.3.jar:?]
> 2021-04-15T22:11:39.5654223Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.8.0_275]
> 2021-04-15T22:11:39.5654742Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:1.8.0_275]
> 2021-04-15T22:11:39.5655269Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:1.8.0_275]
> 2021-04-15T22:11:39.5655625Z ]
> 2021-04-15T22:11:39.5655853Z at org.junit.Assert.fail(Assert.java:88)
> 2021-04-15T22:11:39.5656281Z at
> org.apache.flink.yarn.YarnTestBase.ensureNoProhibitedStringInLogFiles(YarnTestBase.java:576)
> 2021-04-15T22:11:39.5656831Z at
> org.apache.flink.yarn.YARNSessionFIFOITCase.checkForProhibitedLogContents(YARNSessionFIFOITCase.java:86)
> 2021-04-15T22:11:39.5657360Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-04-15T22:11:39.565Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-04-15T22:11:39.5658252Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-15T22:11:39.5658723Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-15T22:11:39.5659311Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-15T22:11:39.5659780Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-15T22:11:39.5660248Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-15T22:11:39.5660829Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> 2021-04-15T22:11:39.5661247Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-15T22:11:39.5661652Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-04-15T22:11:39.5662006Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-04-15T22:11:39.5662379Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-04-15T22:11:39.5662812Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-04-15T22:11:39.5663260Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-04-15T22:11:39.5663935Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>
[jira] [Updated] (FLINK-18894) StateFun job stalls on stop-with-savepoint
[
https://issues.apache.org/jira/browse/FLINK-18894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18894:
---
Labels: auto-unassigned stale-critical (was: auto-unassigned)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 7 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> StateFun job stalls on stop-with-savepoint
> --
>
> Key: FLINK-18894
> URL: https://issues.apache.org/jira/browse/FLINK-18894
> Project: Flink
> Issue Type: Bug
> Components: Stateful Functions
>Affects Versions: statefun-2.1.0, statefun-2.2.0, statefun-3.0.0
>Reporter: Seth Wiesman
>Priority: Critical
> Labels: auto-unassigned, stale-critical
> Attachments: stacktrace.txt
>
>
> Stateful Function jobs stall when performing a stop with savepoint. The
> FunctionDispatchOperator never completes the sync portion of the savepoint.
> Taking a savepoint and then canceling in two separate steps works correctly,
> it is only the stop command that has issues.
> {code}
> curl -X POST localhost:8001/jobs/:jobid/stop -d '{"drain": false}'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15378) StreamFileSystemSink supported mutil hdfs plugins.
[
https://issues.apache.org/jira/browse/FLINK-15378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15378:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> StreamFileSystemSink supported mutil hdfs plugins.
> --
>
> Key: FLINK-15378
> URL: https://issues.apache.org/jira/browse/FLINK-15378
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / FileSystem, FileSystems
>Affects Versions: 1.9.2, 1.10.0
>Reporter: ouyangwulin
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.14.0
>
> Attachments: jobmananger.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> [As report from
> maillist|[https://lists.apache.org/thread.html/7a6b1e341bde0ef632a82f8d46c9c93da358244b6bac0d8d544d11cb%40%3Cuser.flink.apache.org%3E]]
> Request 1: FileSystem plugins not effect the default yarn dependecies.
> Request 2: StreamFileSystemSink supported mutil hdfs plugins under the same
> schema
> As Problem describe :
> when I put a ' filesystem plugin to FLINK_HOME/pulgins in flink', and the
> clas{color:#172b4d}s '*com.filesystem.plugin.FileSystemFactoryEnhance*'
> implements '*FileSystemFactory*', when jm start, It will call
> FileSystem.initialize(configuration,
> PluginUtils.createPluginManagerFromRootFolder(configuration)) to load
> factories to map FileSystem#**{color}FS_FACTORIES, and the key is only
> schema. When tm/jm use local hadoop conf A , the user code use hadoop conf
> Bin 'filesystem plugin', Conf A and Conf B is used to different hadoop
> cluster. and The Jm will start failed, beacuse of the blodserver in JM will
> load Conf B to get filesystem. the full log add appendix.
>
> AS reslove method:
> use schema and spec identify as key for ' FileSystem#**FS_FACTORIES '
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18640) Fix PostgresDialect doesn't quote the identifiers
[ https://issues.apache.org/jira/browse/FLINK-18640?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18640: --- Labels: pull-request-available stale-major (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Fix PostgresDialect doesn't quote the identifiers > - > > Key: FLINK-18640 > URL: https://issues.apache.org/jira/browse/FLINK-18640 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Ecosystem >Affects Versions: 1.9.1, 1.10.1 >Reporter: 毛宗良 >Priority: Major > Labels: pull-request-available, stale-major > > Flink jdbc throw exceptions when read a postgresql table with scheam, like > "ods.t_test". BY debugging the source code, I found a bug about dealing the > table name. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-15693) Stop receiving incoming RPC messages when RpcEndpoint is closing
[
https://issues.apache.org/jira/browse/FLINK-15693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15693:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Stop receiving incoming RPC messages when RpcEndpoint is closing
>
>
> Key: FLINK-15693
> URL: https://issues.apache.org/jira/browse/FLINK-15693
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.9.1, 1.10.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major
>
> When calling {{RpcEndpoint#closeAsync()}}, the system triggers
> {{RpcEndpoint#onStop}} and transitions the endpoint into the
> {{TerminatingState}}. In order to allow asynchronous clean up operations, the
> main thread executor is not shut down immediately. As a side effect, the
> {{RpcEndpoint}} still accepts incoming RPC messages from other components.
> I think it would be cleaner to no longer accept incoming RPC messages once we
> are in the {{TerminatingState}}. That way we would not worry about the
> internal state of the {{RpcEndpoint}} when processing RPC messages (similar
> to
> [here|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskExecutor.java#L952]).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15471) HA e2e check for empty .out files does not print specific error
[
https://issues.apache.org/jira/browse/FLINK-15471?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15471:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> HA e2e check for empty .out files does not print specific error
> ---
>
> Key: FLINK-15471
> URL: https://issues.apache.org/jira/browse/FLINK-15471
> Project: Flink
> Issue Type: Improvement
> Components: Test Infrastructure
>Affects Versions: 1.9.0
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.10.4, 1.11.4, 1.14.0
>
>
> {{common_ha.sh#verify_logs:}}
> {code}
> if ! check_logs_for_non_empty_out_files; then
> echo "FAILURE: Alerts found at the general purpose job."
> EXIT_CODE=1
> fi
> {code}
> Since check_logs_for_non_empty_out_files does only set EXIT_CODE without
> modifying the return value the check will never fail.
> While the test will still fail (since EXIT_CODE is later being evaluated), we
> may not actually print the error cause.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15740) Remove Deadline#timeLeft()
[
https://issues.apache.org/jira/browse/FLINK-15740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15740:
---
Labels: auto-deprioritized-major pull-request-available (was:
pull-request-available stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Remove Deadline#timeLeft()
> --
>
> Key: FLINK-15740
> URL: https://issues.apache.org/jira/browse/FLINK-15740
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream, Tests
>Reporter: Chesnay Schepler
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.14.0
>
> Time Spent: 40m
> Remaining Estimate: 0h
>
> As shown in FLINK-13662, {{Deadline#timeLeft()}} is conceptually broken since
> there is no reliable way to call said method while ensuring that
> a) the value is non-negative (desired since most time-based APIs reject
> negative values)
> b) the value sign (+,-) corresponds to preceding calls to {{#hasTimeLeft()}}
>
> As a result any usage of the following form is unreliable and obfuscating
> error messages.
> {code:java}
> while (deadline.hasTimeLeft()) {
> doSomething(deadline.timeLeft());
> } {code}
>
> All existing usage should be migrate to either
> {code:java}
> while (deadline.hasTimeLeft()) {
> doSomething();
> } {code}
> or
> {code:java}
> while (true) {
> doSomething(deadline.timeLeftIfAny());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15924) Detect and log blocking main thread operations
[
https://issues.apache.org/jira/browse/FLINK-15924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15924:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Detect and log blocking main thread operations
> --
>
> Key: FLINK-15924
> URL: https://issues.apache.org/jira/browse/FLINK-15924
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major
>
> When using the {{RpcEndpoint}} it is important that all operations which run
> on the main thread are never blocking. We have seen in the past that it is
> quite hard to always catch blocking operations in reviews and sometimes these
> changes caused instabilities in Flink. Once this happens it is not trivial to
> find the culprit which is responsible for the blocking operation.
> One way to make debugging easier is to add a monitor which detects and logs
> if a {{RpcEndpoint}} operation takes longer than {{n}} seconds for example.
> Depending on the overhead of this monitor one could even think about enabling
> it only via a special configuration (e.g. debug mode).
> A proper class to introduce this monitor could be the {{AkkaRpcActor}} which
> is responsible for executing main thread operations. Whenever we schedule an
> operation, we could start a timeout which if triggered and the operation has
> not been completed will log a warning.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15325) Input location preference which affects task distribution may make certain job performance worse
[ https://issues.apache.org/jira/browse/FLINK-15325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15325: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Input location preference which affects task distribution may make certain > job performance worse > - > > Key: FLINK-15325 > URL: https://issues.apache.org/jira/browse/FLINK-15325 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: Zhu Zhu >Priority: Minor > Labels: auto-deprioritized-major > Attachments: D58ADB03-7187-46B1-B077-91E5005FD463.png > > > When running TPC-DS jobs in a session cluster, we observed that sometimes > tasks are not evenly distributed in TMs. The root cause turned out to be that > the downstream tasks tend to be TM or host local with its input tasks. This > helps to reduce network shuffle. > However, in certain cases, like the topology presented in the attached image, > jamming the input task's TM and machine with downstream tasks would affect > the performance. In this case, respecting input location preferences is > causing troubles more than bringing benefits. > So I'm wondering whether we should introduce a config so that users can > disable input location preferences? -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16023) jdbc connector's 'connector.table' property should be optional rather than required
[ https://issues.apache.org/jira/browse/FLINK-16023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16023: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > jdbc connector's 'connector.table' property should be optional rather than > required > --- > > Key: FLINK-16023 > URL: https://issues.apache.org/jira/browse/FLINK-16023 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC >Reporter: Bowen Li >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > > jdbc connector's 'connector.table' property should be optional rather than > required. > connector should assume the table name in dbms is the same as that in Flink > when this property is not present > The fundamental reason is that such a design didn't consider integration with > catalogs. Once introduced catalog, the flink table's name should be just the > 'table''s name in corresponding external system. > cc [~ykt836] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18541) Verify the Elasticsearch authentication option in end to end test
[
https://issues.apache.org/jira/browse/FLINK-18541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18541:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Verify the Elasticsearch authentication option in end to end test
> -
>
> Key: FLINK-18541
> URL: https://issues.apache.org/jira/browse/FLINK-18541
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / ElasticSearch, Table SQL / Ecosystem, Tests
>Reporter: Jark Wu
>Priority: Major
> Labels: stale-major
>
> We introduced the authentication feature for Elasticsearch connector in
> FLINK-18361, i.e. the {{username}} and {{password}} options. However, it's
> hard to test the options in integration test, because the embedded
> Elasticsearch cluster doesn't support authentication. Therefore, we would
> like to verify this feature in end to end tests.
> Before adding the tests, in order to be able to use the new connector
> options, we should migrate the existing Elasticsearch e2e to Java testing
> framework FLINK-14620.
> As a reference about how to configure security:
> https://www.elastic.co/guide/en/elasticsearch/reference/7.x/configuring-security.html#configuring-security
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15524) Some links in Chinese documents points to English documents by mistake
[
https://issues.apache.org/jira/browse/FLINK-15524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15524:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Some links in Chinese documents points to English documents by mistake
> --
>
> Key: FLINK-15524
> URL: https://issues.apache.org/jira/browse/FLINK-15524
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.10.1, 1.11.1, 1.12.0
>Reporter: Yun Tang
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.10.4, 1.11.4, 1.14.0
>
>
> After sync with English documents, I noticed some links of the Chinese
> documents point to English documents by mistake.
>
> Take [chinese
> metrics#IO|https://ci.apache.org/projects/flink/flink-docs-master/zh/monitoring/metrics.html#io]
> for example, the link of {{Default shuffle service metrics}} actually points
> to English documents.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16116) Remove shading from oss filesystems build
[ https://issues.apache.org/jira/browse/FLINK-16116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16116: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Remove shading from oss filesystems build > - > > Key: FLINK-16116 > URL: https://issues.apache.org/jira/browse/FLINK-16116 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Affects Versions: 1.10.0 >Reporter: Yang Wang >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > > Since Flink will use plugin to load all the filesystem, the class conflict > will not be a problem. So just like S3, i suggest to remove the shading for > oss filesystem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21445) Application mode does not set the configuration when building PackagedProgram
[
https://issues.apache.org/jira/browse/FLINK-21445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-21445:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> Application mode does not set the configuration when building PackagedProgram
> -
>
> Key: FLINK-21445
> URL: https://issues.apache.org/jira/browse/FLINK-21445
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Deployment / Scripts,
> Deployment / YARN
>Affects Versions: 1.11.3, 1.12.1, 1.13.0
>Reporter: Yang Wang
>Assignee: Nicholas Jiang
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.11.4, 1.14.0, 1.12.5
>
>
> Application mode uses {{ClassPathPackagedProgramRetriever}} to create the
> {{PackagedProgram}}. However, it does not set the configuration. This will
> cause some client configurations not take effect. For example,
> {{classloader.resolve-order}}.
> I think we just forget to do this since we have done the similar thing in
> {{CliFrontend}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15928) Batch mode in blink planner caused IndexOutOfBoundsException error
[
https://issues.apache.org/jira/browse/FLINK-15928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15928:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Batch mode in blink planner caused IndexOutOfBoundsException error
> --
>
> Key: FLINK-15928
> URL: https://issues.apache.org/jira/browse/FLINK-15928
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.9.2
>Reporter: Fanbin Bu
>Priority: Minor
> Labels: auto-deprioritized-major
>
> Flink version: 1.9.2
> mode: Batch mode, running on EMR with YARN
> The following is the details:
>
> table source sample:
> class SnowflakeTableSource(val schema: TableSchema,
> val parallelism: Int,
> val fetchSize: Int,
> val query: String,
> val options: SnowflakeOptions
> )
> extends StreamTableSource[Row] {
> override def getDataStream(execEnv: StreamExecutionEnvironment):
> SingleOutputStreamOperator[Row] = {
> execEnv.createInput(getInputFormat, getReturnType).name("app_event_stream")
> }
> override def getReturnType: TypeInformation[Row] = schema.toRowType
> override def getTableSchema: TableSchema = schema
> override def isBounded: Boolean = true
> private def getInputFormat: JDBCInputFormat = {
> JDBCInputFormat.buildJDBCInputFormat
> .setDrivername(options.driverName)
> .setDBUrl(options.dbUrl)
> .setUsername(options.username)
> .setPassword(options.password)
> .setQuery(query)
> .setRowTypeInfo(getInputRowTypeInfo)
> .setFetchSize(fetchSize)
> .setParametersProvider(new
> GenericParameterValuesProvider(buildQueryParams(parallelism)))
> .finish
> }
> }
>
> Here is the sample setup code:
> val settings = EnvironmentSettings.newInstance()
> .useBlinkPlanner()
> .inBatchMode()
> .build()
> val tableEnv = TableEnvironment.create(settings)
> val configurations = tableEnv.getConfig.getConfiguration
> configurations.setString(
> TABLE_EXEC_RESOURCE_HASH_AGG_MEMORY.key,
> s"${Globals.TABLE_EXEC_RESOURCE_HASH_AGG_MEMORY} mb")
> tableEnv.registerTableSource(tableName, tableSource)
> queryResult = tableEnv.sqlQuery(sql)
> tableEnv.execute()
>
> Here is the sample SQL:
> select
> ip_address
> , hop_end(created_at, interval '30' second, interval '1' minute) as bucket_ts
> , sum(case when name = 'signin' then 1 else 0 end) as signin_count_1m
> , sum(case when name = 'signin_failure' then 1 else 0 end) as
> signin_failure_count_1m
> ...
> from events
> group by
> ip_address
> , hop(created_at, interval '30' second, interval '1' minute)
>
> Here is the stacktrace:
> java.lang.IndexOutOfBoundsException at
> org.apache.flink.core.memory.MemorySegment.getInt(MemorySegment.java:701) at
> org.apache.flink.table.dataformat.BinaryRow.getInt(BinaryRow.java:264) at
> HashWinAggWithKeys$538.endInput(Unknown Source) at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.endInput(OperatorChain.java:276)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.checkFinished(StreamOneInputProcessor.java:151)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:138)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:279)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.run(StreamTask.java:301)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:406)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:705) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:530) at
> java.lang.Thread.run(Thread.java:748)
>
> The fact that this same code works well with other sql and the stacktrace
> message suggests that this might be related to memory issue. And this only
> happens for blink planner in batch mode. I tried to use BatchTableEnvironment
> in old planner and it works.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22113) UniqueKey constraint is lost with multiple sources join in SQL
[
https://issues.apache.org/jira/browse/FLINK-22113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-22113:
---
Labels: pull-request-available stale-assigned (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issue is assigned but has not
received an update in 14, so it has been labeled "stale-assigned".
If you are still working on the issue, please remove the label and add a
comment updating the community on your progress. If this issue is waiting on
feedback, please consider this a reminder to the committer/reviewer. Flink is a
very active project, and so we appreciate your patience.
If you are no longer working on the issue, please unassign yourself so someone
else may work on it. If the "warning_label" label is not removed in 7 days, the
issue will be automatically unassigned.
> UniqueKey constraint is lost with multiple sources join in SQL
> --
>
> Key: FLINK-22113
> URL: https://issues.apache.org/jira/browse/FLINK-22113
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Fu Kai
>Assignee: Xu Guangheng
>Priority: Major
> Labels: pull-request-available, stale-assigned
> Fix For: 1.14.0
>
>
> Hi team,
>
> We have a use case to join multiple data sources to generate a continuous
> updated view. We defined primary key constraint on all the input sources and
> all the keys are the subsets in the join condition. All joins are left join.
>
> In our case, the first two inputs can produce *JoinKeyContainsUniqueKey*
> input sepc, which is good and performant. While when it comes to the third
> input source, it's joined with the intermediate output table of the first two
> input tables, and the intermediate table does not carry key constraint
> information(although the thrid source input table does), so it results in a
> *NoUniqueKey* input sepc. Given NoUniqueKey inputs has dramatic performance
> implications per the[ Force Join Unique
> Key|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Force-Join-Unique-Key-td39521.html#a39651]
> email thread, we want to know if there is any mitigation solution for this.
>
> Example:
> Take the example from
> [https://github.com/ververica/flink-sql-cookbook/blob/master/joins/05/05_star_schema.md]
> {code:java}
> CREATE TEMPORARY TABLE passengers (
> passenger_key STRING,
> first_name STRING,
> last_name STRING,
> update_time TIMESTAMP(3),
> PRIMARY KEY (passenger_key) NOT ENFORCED
> ) WITH (
> 'connector' = 'upsert-kafka',
> 'topic' = 'passengers',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'key.format' = 'raw',
> 'value.format' = 'json'
> );
> CREATE TEMPORARY TABLE stations (
> station_key STRING,
> update_time TIMESTAMP(3),
> city STRING,
> PRIMARY KEY (station_key) NOT ENFORCED
> ) WITH (
> 'connector' = 'upsert-kafka',
> 'topic' = 'stations',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'key.format' = 'raw',
> 'value.format' = 'json'
> );
> CREATE TEMPORARY TABLE booking_channels (
> booking_channel_key STRING,
> update_time TIMESTAMP(3),
> channel STRING,
> PRIMARY KEY (booking_channel_key) NOT ENFORCED
> ) WITH (
> 'connector' = 'upsert-kafka',
> 'topic' = 'booking_channels',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'key.format' = 'raw',
> 'value.format' = 'json'
> );
> CREATE TEMPORARY TABLE train_activities (
> scheduled_departure_time TIMESTAMP(3),
> actual_departure_date TIMESTAMP(3),
> passenger_key STRING,
> origin_station_key STRING,
> destination_station_key STRING,
> booking_channel_key STRING,
> PRIMARY KEY (booking_channel_key, origin_station_key,
> destination_station_key) NOT ENFORCED
> ) WITH (
> 'connector' = 'upsert-kafka',
> 'topic' = 'train_activities',
> 'properties.bootstrap.servers' = 'localhost:9092',
> 'key.format' = 'json',
> 'value.format' = 'json'
> );
> SELECT
> t.actual_departure_date,
> p.first_name,
> p.last_name,
> b.channel,
> os.city AS origin_station,
> ds.city AS destination_station
> FROM train_activities_1 t
> LEFT JOIN booking_channels b
> ON t.booking_channel_key = b.booking_channel_key
> LEFT JOIN passengers p
> ON t.passenger_key = p.passenger_key
> LEFT JOIN stations os
> ON t.origin_station_key = os.station_key
> LEFT JOIN stations ds
> ON t.destination_station_key = ds.station_key
> {code}
>
> The query will generate exeuction plan of:
>
> {code:java}
> Flink SQL> explain
> > SELECT
> >t.actual_departure_date,
> >p.first_name,
> >p.last_name,
> >b.channel,
> >os.city AS
[jira] [Updated] (FLINK-16277) StreamTableEnvironment.toAppendStream fails with Decimal types
[
https://issues.apache.org/jira/browse/FLINK-16277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16277:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> StreamTableEnvironment.toAppendStream fails with Decimal types
> --
>
> Key: FLINK-16277
> URL: https://issues.apache.org/jira/browse/FLINK-16277
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Benoît Paris
>Priority: Minor
> Labels: auto-deprioritized-major
> Attachments: DecimalType 38 18 Logical - stacktrace.txt,
> flink-test-schema-update.zip
>
>
> The following fails when there is a Decimal type in the underlying
> TableSource:
>
> {code:java}
> DataStream appendStream = tEnv.toAppendStream(
> asTable,
> asTable.getSchema().toRowType()
> );{code}
> Yielding the following error:
>
> ValidationException: Type ROW<`y` DECIMAL(38, 18)> of table field 'payload'
> does not match with the physical type ROW<`y` LEGACY('DECIMAL', 'DECIMAL')>
> of the 'payload' field of the TableSource return type
>
>
> Remarks:
> * toAppendStream is not ready for the new type system, does not accept the
> new DataTypes
> * The LegacyTypeInformationType transition type hinders things. Replacing it
> with the new DataTypes.DECIMAL type makes things work.
> * flink-json is not ready for the new type system, does not give the new
> DataTypes
>
> Workaround: reprocess TypeConversions.fromLegacyInfoToDataType's output to
> replace LegacyTypeInformationType types when they are of DECIMAL typeroot
> with the new types.
>
> Included is reproduction and workaround (activated by line 127) code, with
> java + pom + stacktrace files.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-22282) Move creation of SplitEnumerator to the SourceCoordinator thread
[ https://issues.apache.org/jira/browse/FLINK-22282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-22282: --- Labels: stale-assigned (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Move creation of SplitEnumerator to the SourceCoordinator thread > > > Key: FLINK-22282 > URL: https://issues.apache.org/jira/browse/FLINK-22282 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.12.2 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin >Priority: Critical > Labels: stale-assigned > Fix For: 1.14.0, 1.12.5 > > > Currently the creation of the SplitEnumerator is in the JM main thread. In > case the SplitEnumerator instantiation takes long, the job execution will > timeout. The fix is moving the SplitEnumerator creation to the coordinator > thread. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21755) Remove flink-runtime dependency from metric reporters
[ https://issues.apache.org/jira/browse/FLINK-21755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-21755: --- Labels: stale-assigned (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Remove flink-runtime dependency from metric reporters > - > > Key: FLINK-21755 > URL: https://issues.apache.org/jira/browse/FLINK-21755 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Major > Labels: stale-assigned > Fix For: 1.14.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15771: [FLINK-18934] Idle stream does not advance watermark in connected stream
flinkbot edited a comment on pull request #15771: URL: https://github.com/apache/flink/pull/15771#issuecomment-826939649 ## CI report: * d55f8912d2aea880925f4aa631467e8f9679d501 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18248) * 77caa970a0350585ebeb226e2d823e74b12511be Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18295) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15771: [FLINK-18934] Idle stream does not advance watermark in connected stream
flinkbot edited a comment on pull request #15771: URL: https://github.com/apache/flink/pull/15771#issuecomment-826939649 ## CI report: * d55f8912d2aea880925f4aa631467e8f9679d501 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18248) * 77caa970a0350585ebeb226e2d823e74b12511be UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-12675) Event time synchronization in Kafka consumer
[ https://issues.apache.org/jira/browse/FLINK-12675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350597#comment-17350597 ] Alexey Trenikhin commented on FLINK-12675: -- Does new Source interface support suspend/resume ? > Event time synchronization in Kafka consumer > > > Key: FLINK-12675 > URL: https://issues.apache.org/jira/browse/FLINK-12675 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Kafka >Reporter: Thomas Weise >Priority: Major > Labels: auto-unassigned, pull-request-available > Attachments: 0001-Kafka-event-time-alignment.patch > > > Integrate the source watermark tracking into the Kafka consumer and implement > the sync mechanism (different consumer model, compared to Kinesis). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #15993: Merge pull request #1 from apache/master
flinkbot edited a comment on pull request #15993: URL: https://github.com/apache/flink/pull/15993#issuecomment-847105250 ## CI report: * 59c93763d7b2c0fe314d429ea547ef09a8193863 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18291) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-22761) Cannot remove POJO fields
[
https://issues.apache.org/jira/browse/FLINK-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ygor Allan de Fraga updated FLINK-22761:
Description:
I tested a schema evolution in a state using POJO and no problem was found when
trying to add a new field, it was executed just fine. This same field was
removed from the POJO as it was just a test, but the application could not
restore the state due to an error.
Here is the error:
{code:java}
2021-05-24 13:05:31,958 WARN org.apache.flink.runtime.taskmanager.Task
[] - Co-Flat Map -> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de)
switched from RUNNING to FAILED.2021-05-24 13:05:31,958 WARN
org.apache.flink.runtime.taskmanager.Task [] - Co-Flat Map
-> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de) switched from RUNNING to
FAILED.java.lang.Exception: Exception while creating
StreamOperatorStateContext. at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
[zdata-flink-streams.jar:0.1] at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
[zdata-flink-streams.jar:0.1] at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_282]
Caused by: org.apache.flink.util.FlinkException: Could not restore keyed state
backend for CoStreamFlatMap_b101f370952ea85c2104e98dd54bf7f9_(3/3) from any of
the 1 provided restore options. at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[zdata-flink-streams.jar:0.1] ... 9 more
Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
unexpected exception. at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:361)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:587)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:93)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[zdata-flink-streams.jar:0.1] ... 9 more
Caused by: java.lang.NullPointerException at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.(PojoSerializer.java:119)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:184)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:56)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:170)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:56)
[jira] [Updated] (FLINK-22761) Cannot remove POJO fields
[
https://issues.apache.org/jira/browse/FLINK-22761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ygor Allan de Fraga updated FLINK-22761:
Description:
I tested a schema evolution in a state using POJO and no problem was found when
trying to add a new field, it was executed just fine. This same field was
removed from the POJO as it was just a test, but the application could not
restore the state due to an error.
Here is the error:
{code:java}
2021-05-24 13:05:31,958 WARN org.apache.flink.runtime.taskmanager.Task
[] - Co-Flat Map -> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de)
switched from RUNNING to FAILED.2021-05-24 13:05:31,958 WARN
org.apache.flink.runtime.taskmanager.Task [] - Co-Flat Map
-> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de) switched from RUNNING to
FAILED.java.lang.Exception: Exception while creating
StreamOperatorStateContext. at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
[zdata-flink-streams.jar:0.1] at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
[zdata-flink-streams.jar:0.1] at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_282]Caused by: org.apache.flink.util.FlinkException: Could not restore
keyed state backend for CoStreamFlatMap_b101f370952ea85c2104e98dd54bf7f9_(3/3)
from any of the 1 provided restore options. at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[zdata-flink-streams.jar:0.1] ... 9 moreCaused by:
org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected
exception. at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:361)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:587)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:93)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[zdata-flink-streams.jar:0.1] ... 9 moreCaused by:
java.lang.NullPointerException at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.(PojoSerializer.java:119)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:184)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:56)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:170)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:56)
[GitHub] [flink] flinkbot edited a comment on pull request #15993: Merge pull request #1 from apache/master
flinkbot edited a comment on pull request #15993: URL: https://github.com/apache/flink/pull/15993#issuecomment-847105250 ## CI report: * 59c93763d7b2c0fe314d429ea547ef09a8193863 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18291) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #15993: Merge pull request #1 from apache/master
flinkbot commented on pull request #15993: URL: https://github.com/apache/flink/pull/15993#issuecomment-847105250 ## CI report: * 59c93763d7b2c0fe314d429ea547ef09a8193863 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22761) Cannot remove POJO fields
Ygor Allan de Fraga created FLINK-22761:
---
Summary: Cannot remove POJO fields
Key: FLINK-22761
URL: https://issues.apache.org/jira/browse/FLINK-22761
Project: Flink
Issue Type: Bug
Components: API / Type Serialization System
Affects Versions: 1.12.1
Reporter: Ygor Allan de Fraga
I tested a schema evolution in a state using POJO and no problem was found when
trying to add a new field, it was executed just fine. This same field was
removed from the POJO as it was just a test, but the application could not
restore the state due to an error.
Here is the error:
{code:java}
2021-05-24 13:05:31,958 WARN org.apache.flink.runtime.taskmanager.Task
[] - Co-Flat Map -> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de)
switched from RUNNING to FAILED.2021-05-24 13:05:31,958 WARN
org.apache.flink.runtime.taskmanager.Task [] - Co-Flat Map
-> Map (3/3)#464 (e0e6d41a18214eab0a1d3c089d8672de) switched from RUNNING to
FAILED.java.lang.Exception: Exception while creating
StreamOperatorStateContext. at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:254)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.AbstractStreamOperator.initializeState(AbstractStreamOperator.java:272)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:425)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$2(StreamTask.java:535)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:525)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:565)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:755)
[zdata-flink-streams.jar:0.1] at
org.apache.flink.runtime.taskmanager.Task.run(Task.java:570)
[zdata-flink-streams.jar:0.1] at java.lang.Thread.run(Thread.java:748)
[?:1.8.0_282]Caused by: org.apache.flink.util.FlinkException: Could not restore
keyed state backend for CoStreamFlatMap_b101f370952ea85c2104e98dd54bf7f9_(3/3)
from any of the 1 provided restore options. at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:160)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[zdata-flink-streams.jar:0.1] ... 9 moreCaused by:
org.apache.flink.runtime.state.BackendBuildingException: Caught unexpected
exception. at
org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:361)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:587)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:93)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:328)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:168)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:345)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:163)
~[zdata-flink-streams.jar:0.1] ... 9 moreCaused by:
java.lang.NullPointerException at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.(PojoSerializer.java:119)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:184)
~[zdata-flink-streams.jar:0.1] at
org.apache.flink.api.java.typeutils.runtime.PojoSerializer.duplicate(PojoSerializer.java:56)
~[zdata-flink-streams.jar:0.1] at
[GitHub] [flink] flinkbot commented on pull request #15993: Merge pull request #1 from apache/master
flinkbot commented on pull request #15993: URL: https://github.com/apache/flink/pull/15993#issuecomment-847090768 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 59c93763d7b2c0fe314d429ea547ef09a8193863 (Mon May 24 14:44:36 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuhutongchao opened a new pull request #15993: Merge pull request #1 from apache/master
wuhutongchao opened a new pull request #15993: URL: https://github.com/apache/flink/pull/15993 1 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuhutongchao commented on pull request #15993: Merge pull request #1 from apache/master
wuhutongchao commented on pull request #15993: URL: https://github.com/apache/flink/pull/15993#issuecomment-847087303 1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #15725: [FLINK-21389] determine parquet schema from file instead of taking it from user
echauchot commented on pull request #15725: URL: https://github.com/apache/flink/pull/15725#issuecomment-847084989 @AHeise same comment as in https://github.com/apache/flink/pull/15156#issuecomment-847073898 ParquetInputFormat.java was removed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot edited a comment on pull request #15156: [FLINK-21393] [formats] Implement ParquetAvroInputFormat
echauchot edited a comment on pull request #15156: URL: https://github.com/apache/flink/pull/15156#issuecomment-847073898 @AHeise ParquetInputFormat base class was removed since I submitted my PR hence the compilation issues, commit ce3631af7313855f675e29b8faa386f6e5a2d43c removed it. This commit mentions "Use the filesystem connector with a Parquet format as a replacement". I guess it refers to https://ci.apache.org/projects/flink/flink-docs-master/docs/connectors/table/filesystem/ which is SQL based. But what if our pipeline pipeline does not use SQL but DataSet API ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] echauchot commented on pull request #15156: [FLINK-21393] [formats] Implement ParquetAvroInputFormat
echauchot commented on pull request #15156: URL: https://github.com/apache/flink/pull/15156#issuecomment-847073898 @AHeise ParquetInputFormat base class was removed since I submitted my PR hence the compilation issues, commit ce3631af7313855f675e29b8faa386f6e5a2d43c removed it. This commit mentions "Use the filesystem connector with a Parquet format as a replacement". What is the equivalent then ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15992: [FLINK-22759][docs] Correct the applicability of some RocksDB related options as per operator
flinkbot edited a comment on pull request #15992: URL: https://github.com/apache/flink/pull/15992#issuecomment-846929996 ## CI report: * b0bb65f033ea4817a3e33137430f794f4d7814f3 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18290) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15938: [FLINK-11103][runtime] Set a configurable default uncaught exception handler for all entrypoints
flinkbot edited a comment on pull request #15938: URL: https://github.com/apache/flink/pull/15938#issuecomment-842314070 ## CI report: * e7bacdbc8fb620fe2e6c81ff2c664dbd8ab6cc19 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18289) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15877: [FLINK-22612][python] Restructure the coders in PyFlink
flinkbot edited a comment on pull request #15877: URL: https://github.com/apache/flink/pull/15877#issuecomment-836433504 ## CI report: * 44a0ac6333880616e97e1401aa9e2f38c0de087e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18288) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on a change in pull request #15939: [FLINK-15390][Connectors/ORC][Connectors/Hive]List/Map/Struct types support for vectorized orc reader
lirui-apache commented on a change in pull request #15939:
URL: https://github.com/apache/flink/pull/15939#discussion_r636891521
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/vector/OrcArrayColumnVector.java
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.orc.vector;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.ColumnarArrayData;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.types.logical.ArrayType;
+
+import org.apache.hadoop.hive.ql.exec.vector.ListColumnVector;
+
+/** This column vector is used to adapt hive's ListColumnVector to Flink's
ArrayColumnVector. */
+public class OrcArrayColumnVector extends AbstractOrcColumnVector
+implements org.apache.flink.table.data.vector.ArrayColumnVector {
+
+private ListColumnVector hiveVector;
+private ArrayType type;
Review comment:
I don't see how this field is used
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/vector/OrcArrayColumnVector.java
##
@@ -0,0 +1,49 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.orc.vector;
+
+import org.apache.flink.table.data.ArrayData;
+import org.apache.flink.table.data.ColumnarArrayData;
+import org.apache.flink.table.data.vector.ColumnVector;
+import org.apache.flink.table.types.logical.ArrayType;
+
+import org.apache.hadoop.hive.ql.exec.vector.ListColumnVector;
+
+/** This column vector is used to adapt hive's ListColumnVector to Flink's
ArrayColumnVector. */
+public class OrcArrayColumnVector extends AbstractOrcColumnVector
+implements org.apache.flink.table.data.vector.ArrayColumnVector {
+
+private ListColumnVector hiveVector;
+private ArrayType type;
+private ColumnVector flinkVector;
Review comment:
Make it final
##
File path:
flink-formats/flink-orc/src/test/java/org/apache/flink/orc/OrcFileSystemITCase.java
##
@@ -164,4 +167,40 @@ public void testOrcFilterPushDown() throws
ExecutionException, InterruptedExcept
"select x, y from orcFilterTable where g = timestamp
'2020-01-01 05:20:00' and x = 'x10'",
Collections.singletonList(Row.of("x10", "10")));
}
+
+@Test
+public void testNestedTypes() throws ExecutionException,
InterruptedException {
+String path =
+
this.getClass().getClassLoader().getResource("test-data-nested.orc").getPath();
+super.tableEnv()
+.executeSql(
+String.format(
+"create table orcNestedTypesTable ("
++ "boolean1 boolean,"
++ "byte1 tinyint,"
++ "short1 smallint,"
++ "int1 int,"
++ "long1 bigint,"
++ "float1 float,"
++ "double1 double,"
++ "string1 string,"
++ "middle ROW>>,"
++ "list ARRAY>,"
++ "map MAP>"
++ ") with ("
+
[GitHub] [flink] flinkbot edited a comment on pull request #15987: [FLINK-22437][table-planner-blink]set parallelism for not null filter
flinkbot edited a comment on pull request #15987: URL: https://github.com/apache/flink/pull/15987#issuecomment-845883196 ## CI report: * d47f5f72fafd68587477c49fbe2b365a511a0cc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18240) Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18287) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #15980: [FLINK-22655]When using -i option to initialize SQL Client session It should be possible to annotate the script with --
flinkbot edited a comment on pull request #15980: URL: https://github.com/apache/flink/pull/15980#issuecomment-845673686 ## CI report: * 26d7d8b9c71a8ca8303fd0ee190f399c126c14db UNKNOWN * 40bdaf053a4781b5d2555be62850288d72b70729 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=18286) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JasonLeeCoding commented on pull request #15980: [FLINK-22655]When using -i option to initialize SQL Client session It should be possible to annotate the script with --
JasonLeeCoding commented on pull request #15980: URL: https://github.com/apache/flink/pull/15980#issuecomment-847018314 OK , Thanks for your review -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-22760) HiveParser::setCurrentTimestamp fails with hive-3.1.2
Rui Li created FLINK-22760: -- Summary: HiveParser::setCurrentTimestamp fails with hive-3.1.2 Key: FLINK-22760 URL: https://issues.apache.org/jira/browse/FLINK-22760 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Rui Li Assignee: Rui Li -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wenlong88 commented on pull request #14916: [FLINK-21345][Table SQL / Planner] Fix BUG of Union All join Temporal…
wenlong88 commented on pull request #14916: URL: https://github.com/apache/flink/pull/14916#issuecomment-846982062 LGTM -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zicat commented on pull request #14916: [FLINK-21345][Table SQL / Planner] Fix BUG of Union All join Temporal…
zicat commented on pull request #14916: URL: https://github.com/apache/flink/pull/14916#issuecomment-846980370 @wenlong88 Please help to review this PR again, thx. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16775) expose FlinkKafkaConsumer/FlinkKafkaProducer Properties for other system
[ https://issues.apache.org/jira/browse/FLINK-16775?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16775: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > expose FlinkKafkaConsumer/FlinkKafkaProducer Properties for other system > > > Key: FLINK-16775 > URL: https://issues.apache.org/jira/browse/FLINK-16775 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.10.0 >Reporter: jackylau >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > > i want to expose Properties FlinkKafkaConsumer/FlinkKafkaProducer Properties > such as getProperties for other system such as atlas and i think it is needed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18981) Support column comment for Hive tables
[ https://issues.apache.org/jira/browse/FLINK-18981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-18981: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support column comment for Hive tables > -- > > Key: FLINK-18981 > URL: https://issues.apache.org/jira/browse/FLINK-18981 > Project: Flink > Issue Type: New Feature > Components: Connectors / Hive >Reporter: Rui Li >Priority: Major > Labels: stale-major > Fix For: 1.14.0 > > > Start working on this once FLINK-18958 is done -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19016) Checksum mismatch when restore from RocksDB
[
https://issues.apache.org/jira/browse/FLINK-19016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-19016:
---
Labels: stale-major (was: )
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 30 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Checksum mismatch when restore from RocksDB
> ---
>
> Key: FLINK-19016
> URL: https://issues.apache.org/jira/browse/FLINK-19016
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.1, 1.12.2, 1.13.0, 1.13.1
>Reporter: Jiayi Liao
>Priority: Major
> Labels: stale-major
>
> The error stack is shown below:
> {code:java}
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for
> KeyedMapBundleOperator_44cfc1ca74b40bb44eed1f38f72b3ea9_(71/300) from any of
> the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:307)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:135)
> ... 6 more
> Caused by: org.apache.flink.runtime.state.BackendBuildingException: Caught
> unexpected exception.
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:333)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:580)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:291)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:121)
> ... 8 more
> Caused by: java.io.IOException: Error while opening RocksDB instance.
> at
> org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.openDB(RocksDBOperationUtils.java:74)
> at
> org.apache.flink.contrib.streaming.state.restore.AbstractRocksDBRestoreOperation.openDB(AbstractRocksDBRestoreOperation.java:131)
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromLocalState(RocksDBIncrementalRestoreOperation.java:214)
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreFromRemoteState(RocksDBIncrementalRestoreOperation.java:188)
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restoreWithoutRescaling(RocksDBIncrementalRestoreOperation.java:162)
> at
> org.apache.flink.contrib.streaming.state.restore.RocksDBIncrementalRestoreOperation.restore(RocksDBIncrementalRestoreOperation.java:148)
> at
> org.apache.flink.contrib.streaming.state.RocksDBKeyedStateBackendBuilder.build(RocksDBKeyedStateBackendBuilder.java:277)
> ... 12 more
> Caused by: org.rocksdb.RocksDBException: checksum mismatch
> at org.rocksdb.RocksDB.open(Native Method)
> at org.rocksdb.RocksDB.open(RocksDB.java:286)
> at
> org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.openDB(RocksDBOperationUtils.java:66)
> ... 18 more
> {code}
> The machine goes down because of hardware problem, then the job cannot
> restart successfully anymore. After digging a little bit, I found that
> RocksDB in Flink uses sync instead of fsync to synchronized the data with the
> disk. With sync operation, the RocksDB cannot guarantee that the current
> in-progress file can be persisted on disk in takeDBNativeCheckpoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19558) Umbrella Ticket for upgrading the REST API
[ https://issues.apache.org/jira/browse/FLINK-19558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-19558: --- Labels: stale-major (was: ) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Major but is unassigned and neither itself nor its Sub-Tasks have been updated for 30 days. I have gone ahead and added a "stale-major" to the issue". If this ticket is a Major, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Umbrella Ticket for upgrading the REST API > -- > > Key: FLINK-19558 > URL: https://issues.apache.org/jira/browse/FLINK-19558 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: Matthias >Priority: Major > Labels: stale-major > > This ticket functions as a collector for all kind of improvements of the REST > API requiring a API version upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16774) expose HBaseUpsertSinkFunction hTableName and schema for other system
[ https://issues.apache.org/jira/browse/FLINK-16774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16774: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > expose HBaseUpsertSinkFunction hTableName and schema for other system > - > > Key: FLINK-16774 > URL: https://issues.apache.org/jira/browse/FLINK-16774 > Project: Flink > Issue Type: Improvement > Components: Connectors / HBase >Affects Versions: 1.10.0 >Reporter: jackylau >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > Attachments: flink-atlas.pdf > > > i want to expose hTableName and schema of HBaseUpsertSinkFunction such as > getTableName, getTableScheme for other system such as atlas and i think it is > needed -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16698) fllink need catalog listener to do such as preCreate/PreDrop* afterCreate/AfterDrop* things
[ https://issues.apache.org/jira/browse/FLINK-16698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16698: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > fllink need catalog listener to do such as preCreate/PreDrop* > afterCreate/AfterDrop* things > --- > > Key: FLINK-16698 > URL: https://issues.apache.org/jira/browse/FLINK-16698 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.10.0 >Reporter: jackylau >Priority: Minor > Labels: auto-deprioritized-major > > In order to support other things such as atlas or authentication, i think > flink need catalog listener to do such as preCreate/PreDrop* > afterCreate/AfterDrop* things, just like spark/hive does -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16682) Make batch size configurable for orc reader
[ https://issues.apache.org/jira/browse/FLINK-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16682: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Make batch size configurable for orc reader > --- > > Key: FLINK-16682 > URL: https://issues.apache.org/jira/browse/FLINK-16682 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Reporter: Rui Li >Priority: Minor > Labels: auto-deprioritized-major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16552) Cannot include Option fields in any Table join
[
https://issues.apache.org/jira/browse/FLINK-16552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16552:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Cannot include Option fields in any Table join
> --
>
> Key: FLINK-16552
> URL: https://issues.apache.org/jira/browse/FLINK-16552
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Legacy Planner
>Affects Versions: 1.7.2
>Reporter: Jason Sinn
>Priority: Minor
> Labels: auto-deprioritized-major
>
> The table API currently fails joins where one of the tables has an option
> type, even though it is not in the join condition. A reproducible test case:
>
> {code:java}
> object TestJoinWithOption {
> case class JoinOne(joinKeyOne: String, otherFieldOne: Option[Int])
> case class JoinTwo(joinKeyTwo: String, otherFieldTwo: Option[Int])
> def main(args: Array[String]): Unit = {
> val sEnv = StreamExecutionEnvironment.getExecutionEnvironment
> val tEnv = TableEnvironment.getTableEnvironment(sEnv)
> val testStream1 = sEnv.fromCollection(Seq(JoinOne("key", Some(1
> val testStream2 = sEnv.fromCollection(Seq(JoinTwo("key", Some(2
> val t1 = tEnv.fromDataStream(testStream1)
> val t2 = tEnv.fromDataStream(testStream2)
> val result = t1.join(t2, "joinKeyOne = joinKeyTwo")
> result.toAppendStream[Row].print()
> sEnv.execute()
> }
> }
> {code}
> Result:
> {code:java}
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Type 'scala.Option' cannot be used in a join operation because it does not
> implement a proper hashCode() method.Exception in thread "main"
> org.apache.flink.table.api.ValidationException: Type 'scala.Option' cannot be
> used in a join operation because it does not implement a proper hashCode()
> method. at
> org.apache.flink.table.typeutils.TypeCheckUtils$.validateEqualsHashCode(TypeCheckUtils.scala:174)
> at
> org.apache.flink.table.typeutils.TypeCheckUtils$.validateEqualsHashCode(TypeCheckUtils.scala:153)
> at
> org.apache.flink.table.typeutils.TypeCheckUtils$$anonfun$validateEqualsHashCode$1.apply$mcVI$sp(TypeCheckUtils.scala:149)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) at
> org.apache.flink.table.typeutils.TypeCheckUtils$.validateEqualsHashCode(TypeCheckUtils.scala:147)
> at
> org.apache.flink.table.runtime.join.NonWindowJoin.(NonWindowJoin.scala:56)
> at
> org.apache.flink.table.runtime.join.NonWindowInnerJoin.(NonWindowInnerJoin.scala:45)
> at
> org.apache.flink.table.plan.nodes.datastream.DataStreamJoinToCoProcessTranslator.createJoinOperator(DataStreamJoinToCoProcessTranslator.scala:112)
> {code}
> It seems as though this issue has been brought up before in the streams API
> here: https://issues.apache.org/jira/browse/FLINK-2673
> Expected behaviour: Since the join condition does not contain the option, the
> resulting schema should look like this (Actually, this was created by
> result.printSchema)
> {code:java}
> root
> |-- joinKeyOne: String
> |-- otherFieldOne: Option[Integer]
> |-- joinKeyTwo: String
> |-- otherFieldTwo: Option[Integer] {code}
> Actual behaviour: Runtime exception is thrown above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16548) Expose consistent environment variable to identify the component name and resource id of jm/tm
[
https://issues.apache.org/jira/browse/FLINK-16548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16548:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Expose consistent environment variable to identify the component name and
> resource id of jm/tm
> --
>
> Key: FLINK-16548
> URL: https://issues.apache.org/jira/browse/FLINK-16548
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Reporter: hejianchao
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.14.0
>
>
> We proposed to expose environment variable to identify the component name and
> resource id of jm/tm.
> To be specified:
> - Expose {{FLINK_COMPONENT_NAME}}. For jm, it should be "jobmanager". For tm,
> it should be "taskexecutor".
> - Expose {{FLINK_COMPONENT_ID}}. For jm/tm, it should be the resource id.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16324) Checkpoint tasks progress would display 100% in web UI even not all tasks finished
[
https://issues.apache.org/jira/browse/FLINK-16324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16324:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Checkpoint tasks progress would display 100% in web UI even not all tasks
> finished
> --
>
> Key: FLINK-16324
> URL: https://issues.apache.org/jira/browse/FLINK-16324
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Web Frontend
>Affects Versions: 1.10.1, 1.11.1, 1.12.0
>Reporter: Yun Tang
>Priority: Minor
> Labels: auto-deprioritized-major
> Fix For: 1.10.4, 1.11.4, 1.14.0
>
>
> The checkpoint progress details would display 100% even not all tasks
> finished in web UI. e.g {{996/1000}} would display as {{100%}} .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16751) Expose bind port for Flink metric query service
[
https://issues.apache.org/jira/browse/FLINK-16751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16751:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Expose bind port for Flink metric query service
> ---
>
> Key: FLINK-16751
> URL: https://issues.apache.org/jira/browse/FLINK-16751
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination, Runtime / Metrics
>Affects Versions: 1.11.0
>Reporter: Till Rohrmann
>Priority: Minor
> Labels: auto-deprioritized-major
>
> With FLINK-15911 it is now possible to run Flink behind a NAT/with an
> unresolvable external address. However, due to FLINK-11127, the
> {{MetricFetcherImpl}} tries to connect to the {{TaskManagers}} instead of the
> other way around. If the TM is running behind a NAT, it would require to
> define an external port. At the moment, it is only possible to configure the
> external and bind port at the same time via
> {{metrics.internal.query-service.port}}. Hence, the external port always
> needs to be the same as the bind port if port forwarding should work.
> I think in order to properly solve this problem we either solve FLINK-11127
> or we introduce a {{metrics.internal.query-service.bind-port}} option which
> allows to configure a bind port. With such an option,
> {{metrics.internal.query-service.port}} would become the external port.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16421) Changing default catalog to hive without changing default database fails
[
https://issues.apache.org/jira/browse/FLINK-16421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16421:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Changing default catalog to hive without changing default database fails
>
>
> Key: FLINK-16421
> URL: https://issues.apache.org/jira/browse/FLINK-16421
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: Gyula Fora
>Priority: Minor
> Labels: auto-deprioritized-major
>
> The default database in Hive is caled "default" not "default_database". This
> causes an error when starting the SQL CLI with hive set as default catalog:
> {code:java}
> Caused by: org.apache.flink.table.catalog.exceptions.CatalogException: A
> database with name [default_database] does not exist in the catalog:
> [hive].Caused by: org.apache.flink.table.catalog.exceptions.CatalogException:
> A database with name [default_database] does not exist in the catalog:
> [hive]. at
> org.apache.flink.table.catalog.CatalogManager.setCurrentDatabase(CatalogManager.java:174)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.useDatabase(TableEnvironmentImpl.java:631)
> at java.util.Optional.ifPresent(Optional.java:159) at
> org.apache.flink.table.client.gateway.local.ExecutionContext.initializeCatalogs(ExecutionContext.java:561)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.initializeTableEnvironment(ExecutionContext.java:494)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:159)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext.(ExecutionContext.java:118)
> at
> org.apache.flink.table.client.gateway.local.ExecutionContext$Builder.build(ExecutionContext.java:744)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16468) BlobClient rapid retrieval retries on failure opens too many sockets
[
https://issues.apache.org/jira/browse/FLINK-16468?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16468:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> BlobClient rapid retrieval retries on failure opens too many sockets
>
>
> Key: FLINK-16468
> URL: https://issues.apache.org/jira/browse/FLINK-16468
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.8.3, 1.9.2, 1.10.0
> Environment: Linux ubuntu servers running, patch current latest
> Ubuntu patch current release java 8 JRE
>Reporter: Jason Kania
>Priority: Minor
> Labels: auto-deprioritized-major
>
> In situations where the BlobClient retrieval fails as in the following log,
> rapid retries will exhaust the open sockets. All the retries happen within a
> few milliseconds.
> {noformat}
> 2020-03-06 17:19:07,116 ERROR org.apache.flink.runtime.blob.BlobClient -
> Failed to fetch BLOB
> cddd17ef76291dd60eee9fd36085647a/p-bcd61652baba25d6863cf17843a2ef64f4c801d5-c1781532477cf65ff1c1e7d72dccabc7
> from aaa-1/10.0.1.1:45145 and store it under
> /tmp/blobStore-7328ed37-8bc7-4af7-a56c-474e264157c9/incoming/temp-0004
> Retrying...
> {noformat}
> The above is output repeatedly until the following error occurs:
> {noformat}
> java.io.IOException: Could not connect to BlobServer at address
> aaa-1/10.0.1.1:45145
> at org.apache.flink.runtime.blob.BlobClient.(BlobClient.java:100)
> at
> org.apache.flink.runtime.blob.BlobClient.downloadFromBlobServer(BlobClient.java:143)
> at
> org.apache.flink.runtime.blob.AbstractBlobCache.getFileInternal(AbstractBlobCache.java:181)
> at
> org.apache.flink.runtime.blob.PermanentBlobCache.getFile(PermanentBlobCache.java:202)
> at
> org.apache.flink.runtime.execution.librarycache.BlobLibraryCacheManager.registerTask(BlobLibraryCacheManager.java:120)
> at
> org.apache.flink.runtime.taskmanager.Task.createUserCodeClassloader(Task.java:915)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:595)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:530)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketException: Too many open files
> at java.net.Socket.createImpl(Socket.java:478)
> at java.net.Socket.connect(Socket.java:605)
> at org.apache.flink.runtime.blob.BlobClient.(BlobClient.java:95)
> ... 8 more
> {noformat}
> The retries should have some form of backoff in this situation to avoid
> flooding the logs and exhausting other resources on the server.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16452) Insert into static partition doesn't support order by or limit
[
https://issues.apache.org/jira/browse/FLINK-16452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16452:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> Insert into static partition doesn't support order by or limit
> --
>
> Key: FLINK-16452
> URL: https://issues.apache.org/jira/browse/FLINK-16452
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Rui Li
>Priority: Minor
> Labels: auto-deprioritized-major
>
> The following example would fail:
> {code}
> create table src (x int, y string);
> create table dest (x int) partitioned by (p string, q string);
> insert into dest partition (p='a') select * from src order by x limit 10;
> {code}
> The error is: {{"INSERT INTO PARTITION statement only support SELECT
> clause for now"}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16701) Elasticsearch sink support alias for indices.
[ https://issues.apache.org/jira/browse/FLINK-16701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16701: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Elasticsearch sink support alias for indices. > - > > Key: FLINK-16701 > URL: https://issues.apache.org/jira/browse/FLINK-16701 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Priority: Minor > Labels: auto-deprioritized-major > Fix For: 1.14.0 > > > This is related to > [FLINK-15400|https://issues.apache.org/jira/browse/FLINK-15400] FLINK-15400 > will only support dynamic index, and do not support the alias. Because > supporting alias both need in Streaming API and Table API, so I think split > the original design to two PRs make sense. > PR for FLINK-15400: > support dynamic index for ElasticsearchTableSink > PR for this issue: > support alias for Streaming API and Table API -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16506) SqlCreateTable can not get the original text when there exists non-ascii char in the column definition
[
https://issues.apache.org/jira/browse/FLINK-16506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16506:
---
Labels: auto-deprioritized-major (was: stale-major)
Priority: Minor (was: Major)
This issue was labeled "stale-major" 7 ago and has not received any updates so
it is being deprioritized. If this ticket is actually Major, please raise the
priority and ask a committer to assign you the issue or revive the public
discussion.
> SqlCreateTable can not get the original text when there exists non-ascii char
> in the column definition
> --
>
> Key: FLINK-16506
> URL: https://issues.apache.org/jira/browse/FLINK-16506
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: Terry Wang
>Priority: Minor
> Labels: auto-deprioritized-major
>
> We can reproduce this problem in FlinkSqlParserImplTest, add one more column
> definition
> ` x varchar comment 'Flink 社区', \n`
> ```
> @Test
> public void testCreateTableWithComment() {
> conformance0 = FlinkSqlConformance.HIVE;
> check("CREATE TABLE tbl1 (\n" +
> " a bigint comment 'test column comment
> AAA.',\n" +
> " h varchar, \n" +
> " x varchar comment 'Flink 社区', \n" +
> " g as 2 * (a + 1), \n" +
> " ts as toTimestamp(b, '-MM-dd HH:mm:ss'),
> \n" +
> " b varchar,\n" +
> " proc as PROCTIME(), \n" +
> " PRIMARY KEY (a, b)\n" +
> ")\n" +
> "comment 'test table comment ABC.'\n" +
> "PARTITIONED BY (a, h)\n" +
> " with (\n" +
> "'connector' = 'kafka', \n" +
> "'kafka.topic' = 'log.test'\n" +
> ")\n",
> "CREATE TABLE `TBL1` (\n" +
> " `A` BIGINT COMMENT 'test column comment
> AAA.',\n" +
> " `H` VARCHAR,\n" +
> " `X` VARCHAR COMMENT 'Flink 社区', \n" +
> " `G` AS (2 * (`A` + 1)),\n" +
> " `TS` AS `TOTIMESTAMP`(`B`, '-MM-dd
> HH:mm:ss'),\n" +
> " `B` VARCHAR,\n" +
> " `PROC` AS `PROCTIME`(),\n" +
> " PRIMARY KEY (`A`, `B`)\n" +
> ")\n" +
> "COMMENT 'test table comment ABC.'\n" +
> "PARTITIONED BY (`A`, `H`)\n" +
> "WITH (\n" +
> " 'connector' = 'kafka',\n" +
> " 'kafka.topic' = 'log.test'\n" +
> ")");
> }
> ```
> the actual unparse of x column will be ` X` VARCHAR COMMENT u&'Flink
> \793e\533a' instead of our expection.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16080) Translate "Table API & SQL" pages into Chinese
[ https://issues.apache.org/jira/browse/FLINK-16080?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16080: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 14, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. If the "warning_label" label is not removed in 7 days, the issue will be automatically unassigned. > Translate "Table API & SQL" pages into Chinese > -- > > Key: FLINK-16080 > URL: https://issues.apache.org/jira/browse/FLINK-16080 > Project: Flink > Issue Type: New Feature > Components: chinese-translation, Documentation >Reporter: Jark Wu >Assignee: Jark Wu >Priority: Major > Labels: pull-request-available, stale-assigned > Time Spent: 10m > Remaining Estimate: 0h > > Translate all pages under "Table API & SQL" section into Chinese, they are > all located under `docs/dev/table/`. > This is an umbrella issue to track all relative pages. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16612) Submit job through the rest api, job name will be lost
[ https://issues.apache.org/jira/browse/FLINK-16612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16612: --- Labels: Client JobName RESTful auto-deprioritized-major (was: Client JobName RESTful stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Submit job through the rest api, job name will be lost > -- > > Key: FLINK-16612 > URL: https://issues.apache.org/jira/browse/FLINK-16612 > Project: Flink > Issue Type: Bug > Components: Client / Job Submission >Affects Versions: 1.10.0 > Environment: In flink1.10 >Reporter: Junli Zhang >Priority: Minor > Labels: Client, JobName, RESTful, auto-deprioritized-major > Fix For: 1.10.4, 1.11.4, 1.14.0 > > Attachments: image-2020-03-16-18-04-59-891.png, > image-2020-03-16-18-06-05-051.png > > > Bug：Submit job through the rest api, job name will be lost > Reason：In method OptimizerPlanEnvironment.executeAsync(String jobName) > > !image-2020-03-16-18-04-59-891.png! > > Fix: change to : this.pipeline = createProgramPlan(jobName); > !image-2020-03-16-18-06-05-051.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16731) Support SHOW PARTITIONS table command in TableEnvironment and SQL Client
[ https://issues.apache.org/jira/browse/FLINK-16731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16731: --- Labels: auto-deprioritized-major (was: stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > Support SHOW PARTITIONS table command in TableEnvironment and SQL Client > - > > Key: FLINK-16731 > URL: https://issues.apache.org/jira/browse/FLINK-16731 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API, Table SQL / Client >Affects Versions: 1.10.0 >Reporter: Jun Zhang >Priority: Minor > Labels: auto-deprioritized-major > > Add a SHOW PARTITIONS TABLE command in sql client to support show the > partition information of the partition table -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21578) Closeable Sink Committer/GlobalCommitter were created to function in onestep during job graph composition
[
https://issues.apache.org/jira/browse/FLINK-21578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350348#comment-17350348
]
Yun Gao commented on FLINK-21578:
-
I'll continue to consider this issue in the next release.
> Closeable Sink Committer/GlobalCommitter were created to function in onestep
> during job graph composition
> -
>
> Key: FLINK-21578
> URL: https://issues.apache.org/jira/browse/FLINK-21578
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Kezhu Wang
>Priority: Major
> Labels: stale-major
> Fix For: 1.14.0
>
>
> Normally, functions/operators are created in job graph composition phase for
> serialization and transmission. Them are "opened" in flink cluster to
> function. This two steps procedure succeed in that there will be no
> resource-cleanup requirement in job graph composition phase.
> While {{Committer}} and {{GlobalCommitter}} has no such "open" operatin but
> they were created in job graph composition phase.
> Following are fixes I could image if we converge to "this is problematic".
> # Add {{open}} or similar method for these two classes.
> # Add {{hasCommitter}}, {{hasGlobalCommitter}} to {{Sink}} and make
> {{createCommitter}} and others not optional(enforce this in runtime).
> Personally, I am a bit preferring second approach for possible less code path
> touching in job graph composition phase. But first approach has advantage
> that it could be an no breaking change.
> There might be other approaches though.
> cc [~guoweima] [~gaoyunhaii] [~aljoscha] [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22663) Release YARN resource very slow when cancel the job after some NodeManagers shutdown
[
https://issues.apache.org/jira/browse/FLINK-22663?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350345#comment-17350345
]
Jinhong Liu commented on FLINK-22663:
-
It is really strange, but all the issues I mentioned before did happen and they
can repeat stably. Thanks again [~fly_in_gis]
> Release YARN resource very slow when cancel the job after some NodeManagers
> shutdown
>
>
> Key: FLINK-22663
> URL: https://issues.apache.org/jira/browse/FLINK-22663
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.2
>Reporter: Jinhong Liu
>Priority: Major
> Labels: YARN
>
> When I test flink on YARN, there is a case that may cause some problems.
> Hadoop Version: 2.7.3
> Flink Version: 1.12.2
> I deploy a flink job on YARN, when the job is running I stop one NodeManager,
> after one or two minutes, the job is auto recovered. But in this situation,
> if I cancel the job, the containers cannot be released immediately, there are
> still some containers that are running include the app master. About 5
> minutes later, these containers exit, and about 10 minutes later the app
> master exit.
> I check the log of app master, seems it try to stop the containers on the
> NodeManger which I have already stopped.
> {code:java}
> 2021-05-14 06:15:17,389 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Job class
> tv.freewheel.reporting.fastlane.Fastlane$ (da883ab39a7a82e4d45a3803bc77dd6f)
> switched from state CANCELLING to CANCELED.
> 2021-05-14 06:15:17,389 INFO
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - Stopping
> checkpoint coordinator for job da883ab39a7a82e4d45a3803bc77dd6f.
> 2021-05-14 06:15:17,390 INFO
> org.apache.flink.runtime.checkpoint.StandaloneCompletedCheckpointStore [] -
> Shutting down
> 2021-05-14 06:15:17,408 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Job
> da883ab39a7a82e4d45a3803bc77dd6f reached globally terminal state CANCELED.
> 2021-05-14 06:15:17,409 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher [] - Shutting
> down cluster with state CANCELED, jobCancelled: true, executionMode: DETACHED
> 2021-05-14 06:15:17,409 INFO
> org.apache.flink.runtime.entrypoint.ClusterEntrypoint[] - Shutting
> YarnJobClusterEntrypoint down with application status CANCELED. Diagnostics
> null.
> 2021-05-14 06:15:17,409 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shutting
> down rest endpoint.
> 2021-05-14 06:15:17,420 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Stopping the JobMaster for job class
> tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f).
> 2021-05-14 06:15:17,422 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Removing
> cache directory
> /tmp/flink-web-af72a00c-0ddd-4e5e-a62c-8244d6caa552/flink-web-ui
> 2021-05-14 06:15:17,432 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] -
> http://ip-10-23-19-197.ec2.internal:43811 lost leadership
> 2021-05-14 06:15:17,432 INFO
> org.apache.flink.runtime.jobmaster.MiniDispatcherRestEndpoint [] - Shut down
> complete.
> 2021-05-14 06:15:17,436 INFO
> org.apache.flink.runtime.resourcemanager.active.ActiveResourceManager [] -
> Shut down cluster because application is in CANCELED, diagnostics null.
> 2021-05-14 06:15:17,436 INFO org.apache.flink.yarn.YarnResourceManagerDriver
> [] - Unregister application from the YARN Resource Manager with
> final status KILLED.
> 2021-05-14 06:15:17,458 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Suspending
> SlotPool.
> 2021-05-14 06:15:17,458 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [] - Close ResourceManager connection
> 493862ba148679a4f16f7de5ffaef665: Stopping JobMaster for job class
> tv.freewheel.reporting.fastlane.Fastlane$(da883ab39a7a82e4d45a3803bc77dd6f)..
> 2021-05-14 06:15:17,458 INFO
> org.apache.flink.runtime.jobmaster.slotpool.SlotPoolImpl [] - Stopping
> SlotPool.
> 2021-05-14 06:15:17,482 INFO
> org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl[] - Waiting for
> application to be successfully unregistered.
> 2021-05-14 06:15:17,566 INFO org.apache.flink.runtime.history.FsJobArchivist
> [] - Job da883ab39a7a82e4d45a3803bc77dd6f has been archived at
> hdfs:/realtime/flink-archive/da883ab39a7a82e4d45a3803bc77dd6f.
> 2021-05-14 06:15:17,589 INFO
> org.apache.flink.runtime.entrypoint.component.DispatcherResourceManagerComponent
> [] - Closing components.
> 2021-05-14 06:15:17,590 INFO
>
