[GitHub] [flink] flinkbot commented on pull request #19416: FlinkSQL中创建的表结构信息在SparkSQL中不可见
flinkbot commented on PR #19416: URL: https://github.com/apache/flink/pull/19416#issuecomment-1094171778 ## CI report: * eec5d45d62c8991bdf7975ef9698e3b3308313cd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] whitecloud6688 opened a new pull request, #19416: FlinkSQL中创建的表结构信息在SparkSQL中不可见
whitecloud6688 opened a new pull request, #19416: URL: https://github.com/apache/flink/pull/19416 ### FlinkSQL中创建的表结构信息在SparkSQL中不可见 FlinkSQL中创建的表结构信息在SparkSQL中不可见、字段的注释信息不可见，导致在SparkSQL中不能查询FlinkSQL定义的表数据。请问是配置错误还是什么原因，谢谢。 **FlinkSQL中创建hudi表和mysql表，在FlinkSQL能够看到表结构信息。** [flink]$ bin/sql-client.sh Flink SQL> CREATE TABLE hudi_orders_mor_b ( > order_id INT comment '订单编号', > order_date TIMESTAMP(0) comment '订单时间', > customer_name STRING comment '客户名称', > price DECIMAL(10, 5) comment '价格', > product_id INT comment '产品编号', > order_status BOOLEAN comment '订单状态', > PRIMARY KEY (order_id) NOT ENFORCED > ) COMMENT '订单表' > WITH ( > 'connector' = 'hudi', > 'table.type' = 'MERGE_ON_READ', > 'path' = '/hive/warehouse/hzga.db/hudi_orders_mor_s', > 'read.streaming.enabled' = 'false', > 'read.streaming.check-interval' = '3', > 'hive_sync.enable' = 'true', > 'changelog.enable' = 'true' > ); [INFO] Execute statement succeed. Flink SQL> desc hudi_orders_mor_b; +---++---+---++---+ | name | type | null | key | extras | watermark | +---++---+---++---+ | order_id |INT | false | PRI(order_id) || | |order_date | TIMESTAMP(0) | true | || | | customer_name | STRING | true | || | | price | DECIMAL(10, 5) | true | || | |product_id |INT | true | || | | order_status |BOOLEAN | true | || | +---++---+---++---+ 6 rows in set Flink SQL> CREATE TABLE mysql_orders ( >order_id INT comment '订单编号', >order_date TIMESTAMP(0) comment '订单时间', >customer_name STRING comment '客户名称', >price DECIMAL(10, 5) comment '价格', >product_id INT comment '产品编号', >order_status BOOLEAN comment '订单状态', >PRIMARY KEY (order_id) NOT ENFORCED > ) COMMENT '订单表' > WITH ( >'connector' = 'mysql-cdc', >'hostname' = 'vm01', >'port' = '3306', >'username' = 'test', >'password' = 'passwd', >'database-name' = 'mydb', >'table-name' = 'orders' > ); [INFO] Execute statement succeed. Flink SQL> desc mysql_orders; +---++---+---++---+ | name | type | null | key | extras | watermark | +---++---+---++---+ | order_id |INT | false | PRI(order_id) || | |order_date | TIMESTAMP(0) | true | || | | customer_name | STRING | true | || | | price | DECIMAL(10, 5) | true | || | |product_id |INT | true | || | | order_status |BOOLEAN | true | || | +---++---+---++---+ 6 rows in set **在SparkSQL中，查看不到相应的表结构信息，同时无法读取表的数据** $ bin/spark-sql \ > --master spark://vm01:7077,vm02:7077,vm03:7077 \ > --conf 'spark.serializer=org.apache.spark.serializer.KryoSerializer' \ > --conf 'spark.sql.extensions=org.apache.spark.sql.hudi.HoodieSparkSessionExtension' \ > --conf 'spark.executor.cores=1' \ > --conf 'spark.cores.max=2' \ > --conf 'spark.executor.memory=2g' \ > --conf 'spark.driver.memory=2g' 22/04/10 10:59:50 WARN HiveConf: HiveConf of name hive.metastore.event.db.notification.api.auth does not exist 22/04/10 10:59:57 WARN metastore: Failed to connect to the MetaStore Server... Spark master: spark://vm01:7077,vm02:7077,vm03:7077, Application Id: app-20220410105953-0011 spark-sql (default)> use mydb1; ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8ANTLR Tool version 4.7 used for code generation does not match the current runtime version 4.8Response code Time taken: 4.222 seconds spark-sql (default)> desc hudi_orders_mor_b; col_name data_type comment Time taken: 0.325 seconds spark-sql (default)> desc mysql_orders; col_name data_type comment Time taken: 0.09 seconds spark-sql (default)> **版本信息：** spark-3.1.2-bin-hadoop3.2 flink-1.13.6 hudi-0.10.1 flink-cdc-2.2.0 apache-hive-3.1.2-bin zookeeper-3.4.14 hadoop-3.2.2 scala-2.12.10 java-1.8.0_321 **部分Jar包信息：** $ ls /opt/app/spark/jars/
[jira] [Commented] (FLINK-27009) Support SQL job submission in flink kubernetes opeartor
[ https://issues.apache.org/jira/browse/FLINK-27009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17520062#comment-17520062 ] Yufei Zhang commented on FLINK-27009: - I agree, submitting SQL jobs in application mode seems a big need for many users. Would be great to see a discussion in ML. > Support SQL job submission in flink kubernetes opeartor > --- > > Key: FLINK-27009 > URL: https://issues.apache.org/jira/browse/FLINK-27009 > Project: Flink > Issue Type: New Feature > Components: Kubernetes Operator >Reporter: Biao Geng >Priority: Major > > Currently, the flink kubernetes opeartor is for jar job using application or > session cluster. For SQL job, there is no out of box solution in the > operator. > One simple and short-term solution is to wrap the SQL script into a jar job > using table API with limitation. > The long-term solution may work with > [FLINK-26541|https://issues.apache.org/jira/browse/FLINK-26541] to achieve > the full support. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #19415: Illegal use of 'NULL'
flinkbot commented on PR #19415: URL: https://github.com/apache/flink/pull/19415#issuecomment-1094160917 ## CI report: * 66d896081ae9c9572972ec13cef8169d57391444 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] whitecloud6688 opened a new pull request, #19415: Illegal use of 'NULL'
whitecloud6688 opened a new pull request, #19415: URL: https://github.com/apache/flink/pull/19415 ### Illegal use of 'NULL' 需要将 NULL 赋值给一个字段，然后写入表，但报语法错误。版本：flink-1.13.6。 请问类似需求该怎么规避或解决呢？谢谢。 Flink SQL> select if(1>2,1,NULL) as col1; [ERROR] Could not execute SQL statement. Reason: org.apache.calcite.sql.validate.SqlValidatorException: Illegal use of 'NULL' Flink SQL> select NULL as col1; [ERROR] Could not execute SQL statement. Reason: org.apache.calcite.sql.validate.SqlValidatorException: Illegal use of 'NULL' Flink SQL> select NULL is NULL as col1; +++ | op | col1 | +++ | +I | true | +++ Received a total of 1 row -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-26486) The operator restart ingress is lost
[ https://issues.apache.org/jira/browse/FLINK-26486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] leinenglian closed FLINK-26486. --- Resolution: Fixed > The operator restart ingress is lost > > > Key: FLINK-26486 > URL: https://issues.apache.org/jira/browse/FLINK-26486 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Reporter: leinenglian >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14917) Add end to end performance testing framework
[ https://issues.apache.org/jira/browse/FLINK-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14917: --- Labels: auto-deprioritized-major pull-request-available stale-minor (was: auto-deprioritized-major pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add end to end performance testing framework > > > Key: FLINK-14917 > URL: https://issues.apache.org/jira/browse/FLINK-14917 > Project: Flink > Issue Type: New Feature > Components: Test Infrastructure >Reporter: Yu Li >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-minor > Time Spent: 10m > Remaining Estimate: 0h > > This is the umbrella issue for [FLIP-83: Flink End-to-end Performance Testing > Framework|https://cwiki.apache.org/confluence/display/FLINK/FLIP-83%3A+Flink+End-to-end+Performance+Testing+Framework] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15424) Make all AppendingState#add respect the java doc
[
https://issues.apache.org/jira/browse/FLINK-15424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15424:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Make all AppendingState#add respect the java doc
>
>
> Key: FLINK-15424
> URL: https://issues.apache.org/jira/browse/FLINK-15424
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Runtime / State Backends
>Affects Versions: 1.8.3, 1.9.1
>Reporter: Congxian Qiu
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, We have a java doc in
> {{[AppendingState#add|https://github.com/apache/flink/blob/52fdee1d0c7af24d25c51caa073e29f11b07210b/flink-core/src/main/java/org/apache/flink/api/common/state/AppendingState.java#L63]}}
> {code:java}
> If null is passed in, the state value will remain unchanged.{code}
> but currently, the implementation did not respect this, take
> {{HeapReducingState}} as an example, we'll clear the state if the passed
> parameter is null
> {code:java}
> @Override
> public void add(V value) throws IOException {
> if (value == null) {
> clear();
> return;
> }
> try {
> stateTable.transform(currentNamespace, value, reduceTransformation);
> } catch (Exception e) {
> throw new IOException("Exception while applying ReduceFunction in
> reducing state", e);
> }
> }
> {code}
> But in {{RocksDBReducingState}} we would not clear the state, and put the
> null value into state if serializer can serialize null.
> {code:java}
> @Override
> public void add(V value) throws Exception {
>byte[] key = getKeyBytes();
>V oldValue = getInternal(key);
>V newValue = oldValue == null ? value : reduceFunction.reduce(oldValue,
> value);
>updateInternal(key, newValue);
> }
> {code}
> this issue wants to make all {{Appending}}State respect the javadoc of
> {{AppendingState}}, and return directly if the passed in parameter is null.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14748) Migrate duration and memory size related ConfigOptions to typed options
[
https://issues.apache.org/jira/browse/FLINK-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14748:
---
Labels: auto-deprioritized-major stale-minor (was:
auto-deprioritized-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Migrate duration and memory size related ConfigOptions to typed options
> ---
>
> Key: FLINK-14748
> URL: https://issues.apache.org/jira/browse/FLINK-14748
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Configuration
>Reporter: Dawid Wysakowicz
>Priority: Minor
> Labels: auto-deprioritized-major, stale-minor
>
> The goal of this task is to migrate ConfigOptions that describe intervals or
> memory sizes to the typed ConfigOptions. We should also regenerate the
> documentation after migrating.
> Example:
> {code}
> public static final ConfigOption RESTART_STRATEGY_FAILURE_RATE_DELAY
> = ConfigOptions
> .key("restart-strategy.failure-rate.delay")
> .defaultValue("1 s")
> .withDescription(...);
> {code}
> should become
> {code}
> public static final ConfigOption
> RESTART_STRATEGY_FAILURE_RATE_DELAY = ConfigOptions
> .key("restart-strategy.failure-rate.delay")
> .durationType()
> .defaultValue(Duration.ofSeconds(1))
> .withDescription(...);
> {code}
> We should also migrate other types whenever possible.
> For example we should migrate:
> {code}
> public static final ConfigOption CONNECT_TIMEOUT =
> key("blob.client.connect.timeout")
> .defaultValue(0)
> .withDescription("The connection timeout in
> milliseconds for the blob client.");
> {code}
> but we should not migrate, because it is expressed in seconds which is not
> the default unit:
> {code}
> public static final ConfigOption CLEANUP_INTERVAL =
> key("blob.service.cleanup.interval")
> .defaultValue(3_600L) // once per hour
>
> .withDeprecatedKeys("library-cache-manager.cleanup.interval")
> .withDescription("Cleanup interval of the blob caches
> at the task managers (in seconds).");
> {code}
> *NOTE:* We should retain and deprecate the existing fields and add new
> options with the intended type as they are all annotated with
> {{@PublicEvolving}}.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14848) BaseRowSerializer.toBinaryRow wrongly process null for non-compact decimal
[
https://issues.apache.org/jira/browse/FLINK-14848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-14848:
---
Labels: auto-deprioritized-major pull-request-available stale-minor (was:
auto-deprioritized-major pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> BaseRowSerializer.toBinaryRow wrongly process null for non-compact decimal
> ---
>
> Key: FLINK-14848
> URL: https://issues.apache.org/jira/browse/FLINK-14848
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.9.1
>Reporter: Zhenghua Gao
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available,
> stale-minor
>
> Take non-compact decimal(precision > 18 null as an example, the writer not
> only need to set null bits, but also need to assign bytes of variable-length
> part for future update. The following test case can reproduce the problem.
>
>
> {code:java}
> @Test
> public void test() {
>Decimal dec = Decimal.fromBigDecimal(new BigDecimal(11), 38, 0);
>BinaryRow row1 = new BinaryRow(2);
>BinaryRowWriter writer = new BinaryRowWriter(row1);
>writer.writeDecimal(0, dec, 38);
>writer.writeDecimal(1, null, 38);
>writer.complete();
>BaseRowSerializer serializer = new BaseRowSerializer(null, RowType.of(new
> DecimalType(38, 0), new DecimalType(38, 0)));
>GenericRow row2 = new GenericRow(2);
>row2.setField(0, dec);
>row2.setField(1, null);
>BinaryRow row3 = serializer.toBinaryRow(row2);
>assertTrue(row1.equalsWithoutHeader(row3));
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-14729) Multi-topics consuming from KafkaTableSource
[ https://issues.apache.org/jira/browse/FLINK-14729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14729: --- Labels: auto-deprioritized-major features pull-request-available stale-minor (was: auto-deprioritized-major features pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Multi-topics consuming from KafkaTableSource > > > Key: FLINK-14729 > URL: https://issues.apache.org/jira/browse/FLINK-14729 > Project: Flink > Issue Type: New Feature > Components: Connectors / Kafka >Reporter: Leo Zhang >Priority: Minor > Labels: auto-deprioritized-major, features, > pull-request-available, stale-minor > Time Spent: 0.5h > Remaining Estimate: 0h > > Hi, all. I propose a new functionality of KafkaTableSource which can consume > multiple topics at the same time. > *Design plan* > * Add a new constructor in KafkaTableSource which accepts topics with List > type as one parameter. > * Modify the existed one which only accepts one topic as string type to call > the proposed one to finish the instantiation. That is to say, wrap this topic > in a list and pass it to the multi-topics-consuming constructor. > * Modify the overridden method createKafkaConsumer in KafkaTableSource to > pass topics as List instead of String. > * Replace the field topic with topics as List type in KafkaTableSourceBase > and modify every place using topic with topics. So we just need to modify the > constructor KafkaTableSourceBase, method getDataStream, and equals and > hashCode. > *Test plan* > There is less to do as KafkaTableSource is based on FlinkKafkaConsumer which > already supports consuming multiple topics and is tested well. Of course, we > can easily add further more tests if needed. > > So what's your opinion? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15004:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.9.1, 1.10.0
>Reporter: godfrey he
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-15343) Add Watermarks options for TwoInputStreamOperator
[ https://issues.apache.org/jira/browse/FLINK-15343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-15343: --- Labels: auto-deprioritized-major pull-request-available stale-minor (was: auto-deprioritized-major pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add Watermarks options for TwoInputStreamOperator > - > > Key: FLINK-15343 > URL: https://issues.apache.org/jira/browse/FLINK-15343 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Reporter: jocean.shi >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-minor > Time Spent: 10m > Remaining Estimate: 0h > > Currently, The "TwoInputStreamOperator" such as > "CoBroadcastWithKeyedOperator" "KeyedCoProcessOperator" and the > (Co)stream such as "ConnectedStreams" "BroadcastConnectedStream" only > support compute watermark by two stream. > but user just need one stream to compute watermark in some case. > For example: one stream is only rule data or control command, it > don't hava event time. > I think user has right to choose which stream to compute watermark. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-14990) Add bit functions to Blink planner
[ https://issues.apache.org/jira/browse/FLINK-14990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-14990: --- Labels: auto-deprioritized-major pull-request-available stale-minor (was: auto-deprioritized-major pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add bit functions to Blink planner > -- > > Key: FLINK-14990 > URL: https://issues.apache.org/jira/browse/FLINK-14990 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.9.1 >Reporter: hailong wang >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-minor > Time Spent: 10m > Remaining Estimate: 0h > > Add bit to functions Blink planner which be useful. > Bit functions include：bitAnd、bitNot、bitOr 、bitXor -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24342) Filesystem sink does not escape right bracket in partition name
[
https://issues.apache.org/jira/browse/FLINK-24342?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-24342:
---
Labels: pull-request-available stale-minor (was: pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Filesystem sink does not escape right bracket in partition name
> ---
>
> Key: FLINK-24342
> URL: https://issues.apache.org/jira/browse/FLINK-24342
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Reporter: Alexander Trushev
>Priority: Minor
> Labels: pull-request-available, stale-minor
>
> h3. How to reproduce the problem
> In the following code snippet filesystem sink creates a partition named
> "\{date\}" and writes content "1" to file.
> {code:scala}
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val tEnv = StreamTableEnvironment.create(env)
> val source = env.fromElements(("{date}", 1))
> tEnv.createTemporaryView("source", source)
> val sinkSql =
> """
> |create table sink (
> | part string,
> | content int
> |) partitioned by (part) with (
> | 'connector' = 'filesystem',
> | 'path' = '/tmp/sink',
> | 'format' = 'csv'
> |)
> |""".stripMargin
> tEnv.executeSql(sinkSql).await()
> tEnv.executeSql("insert into sink select * from source").await()
> {code}
> h3. Expected behavior
> Escaped "\{" and "\}" in partition name
> {code}
> $ ls /tmp/sink/
> part=%7Bdate%7D
> {code}
> h3. Actual behavior
> Escaped only "\{" in partition name
> {code}
> $ ls /tmp/sink/
> part=%7Bdate}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25835) The task initialization duration is recorded in logs
[ https://issues.apache.org/jira/browse/FLINK-25835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-25835: --- Labels: auto-deprioritized-major pull-request-available (was: pull-request-available stale-major) Priority: Minor (was: Major) This issue was labeled "stale-major" 7 days ago and has not received any updates so it is being deprioritized. If this ticket is actually Major, please raise the priority and ask a committer to assign you the issue or revive the public discussion. > The task initialization duration is recorded in logs > > > Key: FLINK-25835 > URL: https://issues.apache.org/jira/browse/FLINK-25835 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Affects Versions: 1.12.2, 1.15.0 >Reporter: Bo Cui >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > > [https://github.com/apache/flink/blob/a543e658acfbc22c1579df0d043654037b9ec4b0/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L644] > We are testing the time of state backend initialization for different data > levels.However, the task initialization time cannot be obtained from the log > file and the time taken to restore the status at the backend cannot be > obtained. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-24586) SQL functions should return STRING instead of VARCHAR(2000)
[ https://issues.apache.org/jira/browse/FLINK-24586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-24586: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > SQL functions should return STRING instead of VARCHAR(2000) > --- > > Key: FLINK-24586 > URL: https://issues.apache.org/jira/browse/FLINK-24586 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Ingo Bürk >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.15.0 > > > There are some SQL functions which currently return VARCHAR(2000). With more > strict CAST behavior from FLINK-24413, this could become an issue. > The following functions return VARCHAR(2000) and should be changed to return > STRING instead: > * JSON_VALUE > * JSON_QUERY > * JSON_OBJECT > * JSON_ARRAY > There are also some more functions which should be evaluated: > * CHR > * REVERSE > * SPLIT_INDEX > * PARSE_URL > * FROM_UNIXTIME > * DECODE -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-15674) Let Java and Scala Type Extraction go through the same stack
[
https://issues.apache.org/jira/browse/FLINK-15674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15674:
---
Labels: auto-deprioritized-major auto-unassigned stale-minor usability
(was: auto-deprioritized-major auto-unassigned usability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Let Java and Scala Type Extraction go through the same stack
>
>
> Key: FLINK-15674
> URL: https://issues.apache.org/jira/browse/FLINK-15674
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Stephan Ewen
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned, stale-minor,
> usability
>
> Currently, the Java and Scala Type Extraction stacks are completely different.
> * Java uses the {{TypeExtractor}}
> * Scala uses the type extraction macros.
> As a result, the same class can be extracted as different types in the
> different stacks, which can lead to very confusing results. In particular,
> when you use the TypeExtractor on Scala Classes, you always get a
> {{GenericType}}.
> *Suggestion for New Design*
> There should be one type extraction stack, based on the TypeExtractor.
> * The TypeExtractor should be extensible and load additions through service
> loaders, similar as it currently loads Avro as an extension.
> * The Scala Type Extraction logic should be such an extension.
> * The Scala Marcos would only capture the {{Type}} (as in Java type), meaning
> {{Class}}, or {{ParameterizedType}}, or {{Array}} (etc.) and delegate this to
> the TypeExtractor.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-15964) Getting previous stage in notFollowedBy may throw exception
[
https://issues.apache.org/jira/browse/FLINK-15964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15964:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Getting previous stage in notFollowedBy may throw exception
> ---
>
> Key: FLINK-15964
> URL: https://issues.apache.org/jira/browse/FLINK-15964
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.9.0
>Reporter: shuai.xu
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In a notFollowedBy() condition, it may throw exception if trying to get value
> from previous stage for comparison.
> For example:
> Pattern pattern = Pattern.begin("start",
> AfterMatchSkipStrategy.skipPastLastEvent())
> .notFollowedBy("not").where(new IterativeCondition() {
> private static final long serialVersionUID = -4702359359303151881L;
> @Override
> public boolean filter(Event value, Context ctx) throws Exception {
> return
> value.getName().equals(ctx.getEventsForPattern("start").iterator().next().getName());
> }
> })
> .followedBy("middle").where(new IterativeCondition() {
> @Override
> public boolean filter(Event value, Context ctx) throws Exception {
> return value.getName().equals("b");
> }
> });
> with inputs:
> Event a = new Event(40, "a", 1.0);
> Event b1 = new Event(41, "a", 2.0);
> Event b2 = new Event(43, "b", 3.0);
> It will throw
> org.apache.flink.util.FlinkRuntimeException: Failure happened in filter
> function.org.apache.flink.util.FlinkRuntimeException: Failure happened in
> filter function.
> at org.apache.flink.cep.nfa.NFA.findFinalStateAfterProceed(NFA.java:698) at
> org.apache.flink.cep.nfa.NFA.computeNextStates(NFA.java:628) at
> org.apache.flink.cep.nfa.NFA.doProcess(NFA.java:292) at
> org.apache.flink.cep.nfa.NFA.process(NFA.java:228) at
> org.apache.flink.cep.utils.NFATestHarness.consumeRecord(NFATestHarness.java:107)
> at
> org.apache.flink.cep.utils.NFATestHarness.feedRecord(NFATestHarness.java:84)
> at
> org.apache.flink.cep.utils.NFATestHarness.feedRecords(NFATestHarness.java:77)
> at
> org.apache.flink.cep.nfa.NFAITCase.testEndWithNotFollow(NFAITCase.java:2914)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at
> org.junit.rules.RunRules.evaluate(RunRules.java:20) at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
> at
> com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:47)
> at
>
[jira] [Updated] (FLINK-26590) Triggered checkpoints can be delayed by discarding shared state
[ https://issues.apache.org/jira/browse/FLINK-26590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-26590: --- Labels: pull-request-available stale-assigned (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issue is assigned but has not received an update in 30 days, so it has been labeled "stale-assigned". If you are still working on the issue, please remove the label and add a comment updating the community on your progress. If this issue is waiting on feedback, please consider this a reminder to the committer/reviewer. Flink is a very active project, and so we appreciate your patience. If you are no longer working on the issue, please unassign yourself so someone else may work on it. > Triggered checkpoints can be delayed by discarding shared state > --- > > Key: FLINK-26590 > URL: https://issues.apache.org/jira/browse/FLINK-26590 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.15.0, 1.14.3 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.16.0 > > > Quick note: CheckpointCleaner is not involved here. > When a checkpoint is subsumed, SharedStateRegistry schedules its unused > shared state for async deletion. It uses common IO pool for this and adds a > Runnable per state handle. ( see SharedStateRegistryImpl.scheduleAsyncDelete) > When a checkpoint is started, CheckpointCoordinator uses the same thread pool > to initialize the location for it. (see > CheckpointCoordinator.initializeCheckpoint) > The thread pool is of fixed size > [jobmanager.io-pool.size|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/config/#jobmanager-io-pool-size]; > by default it's the number of CPU cores) and uses FIFO queue for tasks. > When there is a spike in state deletion, the next checkpoint is delayed > waiting for an available IO thread. > Back-pressure seems reasonable here (similar to CheckpointCleaner); however, > this shared state deletion could be spread across multiple subsequent > checkpoints, not neccesarily the next one. > > I believe the issue is an pre-existing one; but it particularly affects > changelog state backend, because 1) such spikes are likely there; 2) > workloads are latency sensitive. > In the tests, checkpoint duration grows from seconds to minutes immediately > after the materialization. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-27156) [FLIP-171] MongoDB implementation of Async Sink
Almog Tavor created FLINK-27156: --- Summary: [FLIP-171] MongoDB implementation of Async Sink Key: FLINK-27156 URL: https://issues.apache.org/jira/browse/FLINK-27156 Project: Flink Issue Type: New Feature Reporter: Almog Tavor *User stories:* I’d like to use MongoDB as a sink for my data pipeline and I think it'll be appropriate if it would inherit AsyncSinkBase. *Scope:* * Implement an asynchronous sink for MongoDb by inheriting the AsyncSinkBase class. The implementation can for now reside in its own module in flink-connectors, or maybe we can open a dedicated repository. * Implement an asynchronous sink writer for MongoDb by extending the AsyncSinkWriter. The implementation must deal with failed requests and retry them. The implementation should batch multiple requests The implemented Sink Writer will be used by the Sink class that will be created as part of this story. I'm currently looking for the right object in the Async MongoDb client that will represent the request. h2. References More details to be found [https://cwiki.apache.org/confluence/display/FLINK/FLIP-171%3A+Async+Sink] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-27155) Reduce multiple reads to the same Changelog file in the same taskmanager during restore
Feifan Wang created FLINK-27155: --- Summary: Reduce multiple reads to the same Changelog file in the same taskmanager during restore Key: FLINK-27155 URL: https://issues.apache.org/jira/browse/FLINK-27155 Project: Flink Issue Type: Improvement Components: Runtime / Checkpointing, Runtime / State Backends Reporter: Feifan Wang h3. Background In the current implementation, State changes of different operators in the same taskmanager may be written to the same changelog file, which effectively reduces the number of files and requests to DFS. But on the other hand, the current implementation also reads the same changelog file multiple times on recovery. More specifically, the number of times the same changelog file is accessed is related to the number of ChangeSets contained in it. And since each read needs to skip the preceding bytes, this network traffic is also wasted. The result is a lot of unnecessary request to DFS when there are multiple slots and keyed state in the same taskmanager. h3. Proposal We can reduce multiple reads to the same changelog file in the same taskmanager during restore. One possible approach is to read the changelog file all at once and cache it in memory or local file for a period of time when reading the changelog file. I think this could be a subtask of [v2 FLIP-158: Generalized incremental checkpoints|https://issues.apache.org/jira/browse/FLINK-25842] . Hi [~ym] , [~roman] how do you think about ? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-27153) Allow optional last-state fallback for savepoint upgrade mode
[ https://issues.apache.org/jira/browse/FLINK-27153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519993#comment-17519993 ] Nicholas Jiang commented on FLINK-27153: [~gyfora], is the `kubernetes.operator.job.upgrade.last-state-fallback` configuration flag used to allow checkpoint based (last-state) recovery? > Allow optional last-state fallback for savepoint upgrade mode > - > > Key: FLINK-27153 > URL: https://issues.apache.org/jira/browse/FLINK-27153 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.0.0 > > > In many cases users would prefer to take a savepoint if the job is healthy > before performing an upgrade but still allow checkpoint based (last-state) > recovery in case the savepoint fails or the job is generally in a bad state. > We should add a configuration flag for this that the user can set in the > flinkConfiguration: > `kubernetes.operator.job.upgrade.last-state-fallback` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] zoltar9264 commented on a diff in pull request #19403: [FLINK-27120][docs-zh] Translate "Gradle" tab of development > conf > overview
zoltar9264 commented on code in PR #19403:
URL: https://github.com/apache/flink/pull/19403#discussion_r846649572
##
docs/content.zh/docs/dev/configuration/overview.md:
##
@@ -85,25 +86,31 @@ plugins {
// artifact properties
group = 'org.quickstart'
version = '0.1-SNAPSHOT'
-mainClassName = 'org.quickstart.StreamingJob'
-mainClassName = 'org.quickstart.StreamingJob'
+mainClassName = 'org.quickstart.DataStreamJob'
description = """Flink Quickstart Job"""
ext {
javaVersion = '1.8'
flinkVersion = '{{< version >}}'
+scalaBinaryVersion = '{{< scala_version >}}'
slf4jVersion = '1.7.32'
log4jVersion = '2.17.1'
}
sourceCompatibility = javaVersion
targetCompatibility = javaVersion
tasks.withType(JavaCompile) {
- options.encoding = 'UTF-8'
+options.encoding = 'UTF-8'
}
applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"]
// declare where to find the dependencies of your project
repositories {
mavenCentral()
+maven {
Review Comment:
Yes, all code modification is copy from English edition page.
And as previous reply mentioned , the other part of this page also use “您”
and missing link tag , should I add hotfix commit to fix that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on a diff in pull request #19403: [FLINK-27120][docs-zh] Translate "Gradle" tab of development > conf > overview
zoltar9264 commented on code in PR #19403:
URL: https://github.com/apache/flink/pull/19403#discussion_r846649572
##
docs/content.zh/docs/dev/configuration/overview.md:
##
@@ -85,25 +86,31 @@ plugins {
// artifact properties
group = 'org.quickstart'
version = '0.1-SNAPSHOT'
-mainClassName = 'org.quickstart.StreamingJob'
-mainClassName = 'org.quickstart.StreamingJob'
+mainClassName = 'org.quickstart.DataStreamJob'
description = """Flink Quickstart Job"""
ext {
javaVersion = '1.8'
flinkVersion = '{{< version >}}'
+scalaBinaryVersion = '{{< scala_version >}}'
slf4jVersion = '1.7.32'
log4jVersion = '2.17.1'
}
sourceCompatibility = javaVersion
targetCompatibility = javaVersion
tasks.withType(JavaCompile) {
- options.encoding = 'UTF-8'
+options.encoding = 'UTF-8'
}
applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"]
// declare where to find the dependencies of your project
repositories {
mavenCentral()
+maven {
Review Comment:
Yes, most code modification is copy from English edition page.
And as previous reply mentioned , the other part of this page also use “您”
and missing link tag , should I add hotfix commit to fix that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-27154) Disable web.cancel.enable for application clusters
[ https://issues.apache.org/jira/browse/FLINK-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27154: --- Labels: Starter pull-request-available (was: Starter) > Disable web.cancel.enable for application clusters > -- > > Key: FLINK-27154 > URL: https://issues.apache.org/jira/browse/FLINK-27154 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Labels: Starter, pull-request-available > Fix For: kubernetes-operator-1.0.0 > > > We should probably always set `web.cancel.enable` to false for Application > Deployments to avoid users accidentally cancelling jobs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-kubernetes-operator] SteNicholas opened a new pull request, #163: [FLINK-27154] Disable web.cancel.enable for application clusters
SteNicholas opened a new pull request, #163: URL: https://github.com/apache/flink-kubernetes-operator/pull/163 `web.cancel.enable` should be set to false for application deployments to avoid users accidentally cancelling jobs. **The brief change log** - `FlinkConfigBuilder` set `web.cancel.enable` to false for application deployments. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27154) Disable web.cancel.enable for application clusters
[ https://issues.apache.org/jira/browse/FLINK-27154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519990#comment-17519990 ] Nicholas Jiang commented on FLINK-27154: [~gyfora], I have worked this configuration. Please assign this ticket to me. > Disable web.cancel.enable for application clusters > -- > > Key: FLINK-27154 > URL: https://issues.apache.org/jira/browse/FLINK-27154 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Labels: Starter > Fix For: kubernetes-operator-1.0.0 > > > We should probably always set `web.cancel.enable` to false for Application > Deployments to avoid users accidentally cancelling jobs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] RocMarshal commented on a diff in pull request #19401: [FLINK-25716][docs-zh] Translate "Streaming Concepts" page of "Applic…
RocMarshal commented on code in PR #19401:
URL: https://github.com/apache/flink/pull/19401#discussion_r846646275
##
docs/content.zh/docs/dev/table/concepts/overview.md:
##
@@ -34,106 +34,79 @@ Flink 的 [Table API]({{< ref "docs/dev/table/tableApi" >}})
和 [SQL]({{< ref "
下面这些页面包含了概念、实际的限制，以及流式数据处理中的一些特定的配置。
-State Management
+状态管理
+流模式下运行的表程序利用了Flink作为有状态流处理器的所有能力。
-Table programs that run in streaming mode leverage all capabilities of Flink
as a stateful stream
-processor.
+事实上，一个表程序（Table program）可以配置一个 [state backend]({{< ref
"docs/ops/state/state_backends" >}})
+和多个不同的 [checkpoint 选项]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
+以处理对不同状态大小和容错需求。这可以对正在运行的 Table API & SQL 管道（pipeline）生成
savepoint，并在这之后用其恢复应用程序的状态。
-In particular, a table program can be configured with a [state backend]({{<
ref "docs/ops/state/state_backends" >}})
-and various [checkpointing options]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
-for handling different requirements regarding state size and fault tolerance.
It is possible to take
-a savepoint of a running Table API & SQL pipeline and to restore the
application's state at a later
-point in time.
+### 状态使用
-### State Usage
-
-Due to the declarative nature of Table API & SQL programs, it is not always
obvious where and how much
-state is used within a pipeline. The planner decides whether state is
necessary to compute a correct
-result. A pipeline is optimized to claim as little state as possible given the
current set of optimizer
-rules.
+由于 Table API & SQL 程序是声明式的，管道内的状态会在哪以及如何被使用并不显然。 Planner 会确认是否需要状态来得到正确的计算结果，
+管道会被现有优化规则集优化成尽可能少地索要状态。
Review Comment:
```suggestion
管道会被现有优化规则集优化成尽可能少地使用状态。
```
##
docs/content.zh/docs/dev/table/concepts/overview.md:
##
@@ -34,106 +34,79 @@ Flink 的 [Table API]({{< ref "docs/dev/table/tableApi" >}})
和 [SQL]({{< ref "
下面这些页面包含了概念、实际的限制，以及流式数据处理中的一些特定的配置。
-State Management
+状态管理
+流模式下运行的表程序利用了Flink作为有状态流处理器的所有能力。
-Table programs that run in streaming mode leverage all capabilities of Flink
as a stateful stream
-processor.
+事实上，一个表程序（Table program）可以配置一个 [state backend]({{< ref
"docs/ops/state/state_backends" >}})
+和多个不同的 [checkpoint 选项]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
+以处理对不同状态大小和容错需求。这可以对正在运行的 Table API & SQL 管道（pipeline）生成
savepoint，并在这之后用其恢复应用程序的状态。
-In particular, a table program can be configured with a [state backend]({{<
ref "docs/ops/state/state_backends" >}})
-and various [checkpointing options]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
-for handling different requirements regarding state size and fault tolerance.
It is possible to take
-a savepoint of a running Table API & SQL pipeline and to restore the
application's state at a later
-point in time.
+### 状态使用
-### State Usage
-
-Due to the declarative nature of Table API & SQL programs, it is not always
obvious where and how much
-state is used within a pipeline. The planner decides whether state is
necessary to compute a correct
-result. A pipeline is optimized to claim as little state as possible given the
current set of optimizer
-rules.
+由于 Table API & SQL 程序是声明式的，管道内的状态会在哪以及如何被使用并不显然。 Planner 会确认是否需要状态来得到正确的计算结果，
+管道会被现有优化规则集优化成尽可能少地索要状态。
{{< hint info >}}
-Conceptually, source tables are never kept entirely in state. An implementer
deals with logical tables
-(i.e. [dynamic tables]({{< ref "docs/dev/table/concepts/dynamic_tables" >}})).
Their state requirements
-depend on the used operations.
+从概念上讲， 源表从来不会在状态中被完全保存。 实现者在处理逻辑表（即[动态表]({{< ref
"docs/dev/table/concepts/dynamic_tables" >}})）时，
Review Comment:
从概念上讲， 源表从来不会在状态中被完全保存。 实现者处理的是逻辑表（即[动态表]({{< ref
"docs/dev/table/concepts/dynamic_tables" >}})）。
##
docs/content.zh/docs/dev/table/concepts/overview.md:
##
@@ -34,106 +34,79 @@ Flink 的 [Table API]({{< ref "docs/dev/table/tableApi" >}})
和 [SQL]({{< ref "
下面这些页面包含了概念、实际的限制，以及流式数据处理中的一些特定的配置。
-State Management
+状态管理
+流模式下运行的表程序利用了Flink作为有状态流处理器的所有能力。
Review Comment:
```suggestion
流模式下运行的表程序利用了 Flink 作为有状态流处理器的所有能力。
```
##
docs/content.zh/docs/dev/table/concepts/overview.md:
##
@@ -34,106 +34,79 @@ Flink 的 [Table API]({{< ref "docs/dev/table/tableApi" >}})
和 [SQL]({{< ref "
下面这些页面包含了概念、实际的限制，以及流式数据处理中的一些特定的配置。
-State Management
+状态管理
+流模式下运行的表程序利用了Flink作为有状态流处理器的所有能力。
-Table programs that run in streaming mode leverage all capabilities of Flink
as a stateful stream
-processor.
+事实上，一个表程序（Table program）可以配置一个 [state backend]({{< ref
"docs/ops/state/state_backends" >}})
+和多个不同的 [checkpoint 选项]({{< ref
"docs/dev/datastream/fault-tolerance/checkpointing" >}})
+以处理对不同状态大小和容错需求。这可以对正在运行的 Table API & SQL 管道（pipeline）生成
savepoint，并在这之后用其恢复应用程序的状态。
-In particular, a table program can be configured with a [state backend]({{<
ref
[jira] [Created] (FLINK-27154) Disable web.cancel.enable for application clusters
Gyula Fora created FLINK-27154: -- Summary: Disable web.cancel.enable for application clusters Key: FLINK-27154 URL: https://issues.apache.org/jira/browse/FLINK-27154 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Gyula Fora Fix For: kubernetes-operator-1.0.0 We should probably always set `web.cancel.enable` to false for Application Deployments to avoid users accidentally cancelling jobs. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-27153) Allow optional last-state fallback for savepoint upgrade mode
[ https://issues.apache.org/jira/browse/FLINK-27153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora updated FLINK-27153: --- Description: In many cases users would prefer to take a savepoint if the job is healthy before performing an upgrade but still allow checkpoint based (last-state) recovery in case the savepoint fails or the job is generally in a bad state. We should add a configuration flag for this that the user can set in the flinkConfiguration: `kubernetes.operator.job.upgrade.last-state-fallback` was: In many cases users would prefer to take a savepoint if the job is healthy before performing an upgrade but still allow checkpoint based (last-state) recovery in case the savepoint fails or the job is generally in a bad state. We should add a configuration flag for this that the user can set in the flinkConfiguration: `kubernetes.operator.upgrade.last-state-fallback` > Allow optional last-state fallback for savepoint upgrade mode > - > > Key: FLINK-27153 > URL: https://issues.apache.org/jira/browse/FLINK-27153 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.0.0 > > > In many cases users would prefer to take a savepoint if the job is healthy > before performing an upgrade but still allow checkpoint based (last-state) > recovery in case the savepoint fails or the job is generally in a bad state. > We should add a configuration flag for this that the user can set in the > flinkConfiguration: > `kubernetes.operator.job.upgrade.last-state-fallback` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-26926) Allow users to force upgrade even if savepoint is in progress
[ https://issues.apache.org/jira/browse/FLINK-26926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519983#comment-17519983 ] Gyula Fora edited comment on FLINK-26926 at 4/9/22 2:07 PM: `kubernetes.operator.job.upgrade.ignore-pending-savepoint` was (Author: gyfora): `kubernetes.operator.upgrade.ignore-pending-savepoint` > Allow users to force upgrade even if savepoint is in progress > - > > Key: FLINK-26926 > URL: https://issues.apache.org/jira/browse/FLINK-26926 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.0.0 > > > Currently all upgrades (regardless of upgrade mode) are delayed as long as > there is a pending savepoint operation. > We should allow users to override this and execute the upgrade (thus > potentially cancelling the savepoint) regardless of the savepoint status. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26926) Allow users to force upgrade even if savepoint is in progress
[ https://issues.apache.org/jira/browse/FLINK-26926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519983#comment-17519983 ] Gyula Fora commented on FLINK-26926: `kubernetes.operator.upgrade.ignore-pending-savepoint` > Allow users to force upgrade even if savepoint is in progress > - > > Key: FLINK-26926 > URL: https://issues.apache.org/jira/browse/FLINK-26926 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.0.0 > > > Currently all upgrades (regardless of upgrade mode) are delayed as long as > there is a pending savepoint operation. > We should allow users to override this and execute the upgrade (thus > potentially cancelling the savepoint) regardless of the savepoint status. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-27153) Allow optional last-state fallback for savepoint upgrade mode
Gyula Fora created FLINK-27153: -- Summary: Allow optional last-state fallback for savepoint upgrade mode Key: FLINK-27153 URL: https://issues.apache.org/jira/browse/FLINK-27153 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Gyula Fora Fix For: kubernetes-operator-1.0.0 In many cases users would prefer to take a savepoint if the job is healthy before performing an upgrade but still allow checkpoint based (last-state) recovery in case the savepoint fails or the job is generally in a bad state. We should add a configuration flag for this that the user can set in the flinkConfiguration: `kubernetes.operator.upgrade.last-state-fallback` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-27053) IncrementalRemoteKeyedStateHandle.discardState swallows errors
[ https://issues.apache.org/jira/browse/FLINK-27053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519981#comment-17519981 ] zhouenning commented on FLINK-27053: hi Matthias Pohl I see the source code which to use try/catch error. Would it be better if I changed the log warn to error level? > IncrementalRemoteKeyedStateHandle.discardState swallows errors > -- > > Key: FLINK-27053 > URL: https://issues.apache.org/jira/browse/FLINK-27053 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Matthias Pohl >Priority: Major > > IncrementalRemoteKeyedStateHandle.discardState swallows errors instead of > propagating them which would make the discard failure go un-noticed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] RocMarshal commented on a diff in pull request #19403: [FLINK-27120][docs-zh] Translate "Gradle" tab of development > conf > overview
RocMarshal commented on code in PR #19403:
URL: https://github.com/apache/flink/pull/19403#discussion_r846629724
##
docs/content.zh/docs/dev/configuration/overview.md:
##
@@ -85,25 +86,31 @@ plugins {
// artifact properties
group = 'org.quickstart'
version = '0.1-SNAPSHOT'
-mainClassName = 'org.quickstart.StreamingJob'
-mainClassName = 'org.quickstart.StreamingJob'
+mainClassName = 'org.quickstart.DataStreamJob'
description = """Flink Quickstart Job"""
ext {
javaVersion = '1.8'
flinkVersion = '{{< version >}}'
+scalaBinaryVersion = '{{< scala_version >}}'
slf4jVersion = '1.7.32'
log4jVersion = '2.17.1'
}
sourceCompatibility = javaVersion
targetCompatibility = javaVersion
tasks.withType(JavaCompile) {
- options.encoding = 'UTF-8'
+options.encoding = 'UTF-8'
}
applicationDefaultJvmArgs = ["-Dlog4j.configurationFile=log4j2.properties"]
// declare where to find the dependencies of your project
repositories {
mavenCentral()
+maven {
Review Comment:
Minor confusion from me, Was the new lines caused by the English edition
page ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] smattheis commented on pull request #19398: [FLINK-26864][metrics] Fix performance regression from mailbox latency measuement
smattheis commented on PR #19398: URL: https://github.com/apache/flink/pull/19398#issuecomment-1094008704 @pnowojski There was a merge conflict. Therefore, I pushed an update and reran the ci check which succeeded. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on pull request #19401: [FLINK-25716][docs-zh] Translate "Streaming Concepts" page of "Applic…
RocMarshal commented on PR #19401: URL: https://github.com/apache/flink/pull/19401#issuecomment-1094004365 Thanks @snailHumming for the translation and @MartijnVisser ping. I check it ASAP. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-27152) Examples of state usage of various data structures
zhangjingcun created FLINK-27152: Summary: Examples of state usage of various data structures Key: FLINK-27152 URL: https://issues.apache.org/jira/browse/FLINK-27152 Project: Flink Issue Type: Improvement Components: API / Python, Examples Reporter: zhangjingcun -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-27029) DeploymentValidator should take default flink config into account during validation
[ https://issues.apache.org/jira/browse/FLINK-27029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-27029: --- Labels: pull-request-available starter (was: starter) > DeploymentValidator should take default flink config into account during > validation > --- > > Key: FLINK-27029 > URL: https://issues.apache.org/jira/browse/FLINK-27029 > Project: Flink > Issue Type: Bug > Components: Kubernetes Operator >Reporter: Gyula Fora >Assignee: Nicholas Jiang >Priority: Major > Labels: pull-request-available, starter > Fix For: kubernetes-operator-1.0.0 > > > Currently the DefaultDeploymentValidator only takes the FlinkDeployment > object into account. > However in places where we validate the presence of config keys we should > also consider the default flink config which might already provide default > values for the required configs even if the deployment itself doesnt. > We should make sure this works correctly both in the operator and the webhook -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink-kubernetes-operator] SteNicholas opened a new pull request, #162: [FLINK-27029] DeploymentValidator should take default flink config into account during validation
SteNicholas opened a new pull request, #162: URL: https://github.com/apache/flink-kubernetes-operator/pull/162 `DefaultDeploymentValidator` only takes the `FlinkDeployment` object into account and should validate the effective Flink configs. However in places where we validate the presence of config keys we should also consider the default Flink config which might already provide default values for the required configs even if the deployment itself doesnt. **The brief change log** - `DefaultDeploymentValidator` adds the validation for the effective Flink configs which merge the default config and user conf. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] smattheis commented on pull request #19398: [FLINK-26864][metrics] Fix performance regression from mailbox latency measuement
smattheis commented on PR #19398: URL: https://github.com/apache/flink/pull/19398#issuecomment-1093891085 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on a diff in pull request #19403: [FLINK-27120][docs-zh] Translate "Gradle" tab of development > conf > overview
zoltar9264 commented on code in PR #19403:
URL: https://github.com/apache/flink/pull/19403#discussion_r846616288
##
docs/content.zh/docs/dev/configuration/overview.md:
##
@@ -67,7 +67,7 @@ $ curl https://flink.apache.org/q/quickstart.sh | bash -s {{<
version >}}
{{< /tab >}}
{{< tab "Gradle" >}}
-您可以使用如下的 Gradle 构建脚本或快速启动脚本创建一个项目。
+您可以创建一个空项目，您需要在其中手动创建 `src/main/java` 和 `src/main/resources`
目录并开始在其中编写一些类，使用如下 Gradle 构建脚本或下面提供的快速启动脚本以获得功能齐全的启动项目。
Review Comment:
Thanks for your comment, I had add link tag and , but it
seems not work inner {{ tab }}. Is that a another problem.
By the way, the other part of this page also use “您” and missing link tag ,
should I add hotfix commit to fix that ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-15980) The notFollowedBy in the end of GroupPattern may be ignored
[
https://issues.apache.org/jira/browse/FLINK-15980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-15980:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> The notFollowedBy in the end of GroupPattern may be ignored
> ---
>
> Key: FLINK-15980
> URL: https://issues.apache.org/jira/browse/FLINK-15980
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.9.0
>Reporter: shuai.xu
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If we write a Pattern like this:
> Pattern group = Pattern.begin('A').notFollowedBy("B");
> Pattern pattern = Pattern.begin(group).followedBy("C");
> Let notFollowedBy as the last part of a GroupPattern.
> This pattern can be compile normally, but the notFollowedBy("B") doesn't work
> in fact.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16043) Support non-BMP Unicode for JsonRowSerializationSchema
[
https://issues.apache.org/jira/browse/FLINK-16043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16043:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Support non-BMP Unicode for JsonRowSerializationSchema
> --
>
> Key: FLINK-16043
> URL: https://issues.apache.org/jira/browse/FLINK-16043
> Project: Flink
> Issue Type: Improvement
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.10.0
>Reporter: Benchao Li
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> This is a known issue for jackson:
> [https://github.com/FasterXML/jackson-core/issues/223]
> You can see more details:
> [https://github.com/FasterXML/jackson-core/blob/master/src/main/java/com/fasterxml/jackson/core/json/UTF8JsonGenerator.java#L2105]
>
> And I also encountered this issue in my production environment. I've figured
> out a solution to solve this issue. Java's String.getBytes() can deal with
> UTF-8 encoding well. So we can do it like this:
> {{mapper.writeValueAsString(node).getBytes()}} instead of
> {{mapper.writeValueAsBytes(node)}}
> cc [~jark] [~twalthr]
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16491) Add compression support for ParquetAvroWriters
[ https://issues.apache.org/jira/browse/FLINK-16491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16491: --- Labels: pull-request-available stale-minor (was: pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add compression support for ParquetAvroWriters > -- > > Key: FLINK-16491 > URL: https://issues.apache.org/jira/browse/FLINK-16491 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.10.0 >Reporter: Yao Zhang >Priority: Minor > Labels: pull-request-available, stale-minor > Time Spent: 10m > Remaining Estimate: 0h > > Add compression support for ParquetAvroWriters -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-16708) When a JDBC connection has been closed, the retry policy of the JDBCUpsertOutputFormat cannot take effect
[
https://issues.apache.org/jira/browse/FLINK-16708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-16708:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> When a JDBC connection has been closed, the retry policy of the
> JDBCUpsertOutputFormat cannot take effect
> --
>
> Key: FLINK-16708
> URL: https://issues.apache.org/jira/browse/FLINK-16708
> Project: Flink
> Issue Type: Bug
> Components: Connectors / JDBC
>Affects Versions: 1.10.0
>Reporter: Wally Tang
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In our test environment, I used the tcpkill command to simulate a scenario
> where the postgresql connection was closed. I found that the retry strategy
> of the flush method did not take effect
> {code:java}
> 2020-03-20 21:16:18.246 [jdbc-upsert-output-format-thread-1] ERROR
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat - JDBC executeBatch
> error, retry times = 1
> org.postgresql.util.PSQLException: This connection has been closed.
> at
> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:857)
> at
> org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:817)
> at
> org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:813)
> at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569)
> at
> org.apache.flink.api.java.io.jdbc.writer.AppendOnlyWriter.executeBatch(AppendOnlyWriter.java:62)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.flush(JDBCUpsertOutputFormat.java:159)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.lambda$open$0(JDBCUpsertOutputFormat.java:124)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> 2020-03-20 21:16:21.247 [jdbc-upsert-output-format-thread-1] ERROR
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat - JDBC executeBatch
> error, retry times = 1
> org.postgresql.util.PSQLException: This connection has been closed.
> at
> org.postgresql.jdbc.PgConnection.checkClosed(PgConnection.java:857)
> at
> org.postgresql.jdbc.PgConnection.getAutoCommit(PgConnection.java:817)
> at
> org.postgresql.jdbc.PgStatement.internalExecuteBatch(PgStatement.java:813)
> at org.postgresql.jdbc.PgStatement.executeBatch(PgStatement.java:873)
> at
> org.postgresql.jdbc.PgPreparedStatement.executeBatch(PgPreparedStatement.java:1569)
> at
> org.apache.flink.api.java.io.jdbc.writer.AppendOnlyWriter.executeBatch(AppendOnlyWriter.java:62)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.flush(JDBCUpsertOutputFormat.java:159)
> at
> org.apache.flink.api.java.io.jdbc.JDBCUpsertOutputFormat.lambda$open$0(JDBCUpsertOutputFormat.java:124)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
>
[jira] [Updated] (FLINK-16356) Some dependencies contain CVEs
[ https://issues.apache.org/jira/browse/FLINK-16356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16356: --- Labels: auto-deprioritized-major stale-minor (was: auto-deprioritized-major) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Some dependencies contain CVEs > -- > > Key: FLINK-16356 > URL: https://issues.apache.org/jira/browse/FLINK-16356 > Project: Flink > Issue Type: Bug > Components: Build System >Reporter: XuCongying >Priority: Minor > Labels: auto-deprioritized-major, stale-minor > Attachments: apache-flink_CVE-report.md > > > I found your project used some dependencies that contain CVEs. To prevent > potential risk it may cause, I suggest a library update. The following is a > detailed content. > > Vulnerable Library Version: com.squareup.okhttp3 : okhttp : 3.7.0 > CVE ID: > [CVE-2018-20200](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-20200) > Import Path: flink-metrics/flink-metrics-datadog/pom.xml, > flink-end-to-end-tests/flink-end-to-end-tests-common/pom.xml, > flink-end-to-end-tests/flink-metrics-reporter-prometheus-test/pom.xml, > flink-runtime/pom.xml > Suggested Safe Versions: 3.12.1, 3.12.2, 3.12.3, 3.12.4, 3.12.5, 3.12.6, > 3.12.7, 3.12.8, 3.13.0, 3.13.1, 3.14.0, 3.14.1, 3.14.2, 3.14.3, 3.14.4, > 3.14.5, 3.14.6, 4.0.0, 4.0.0-RC1, 4.0.0-RC2, 4.0.0-RC3, 4.0.0-alpha01, > 4.0.0-alpha02, 4.0.1, 4.1.0, 4.1.1, 4.2.0, 4.2.1, 4.2.2, 4.3.0, 4.3.1, 4.4.0 > Vulnerable Library Version: com.google.guava : guava : 18.0 > CVE ID: > [CVE-2018-10237](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237) > Import Path: flink-connectors/flink-connector-kinesis/pom.xml, > flink-connectors/flink-connector-cassandra/pom.xml > Suggested Safe Versions: 24.1.1-android, 24.1.1-jre, 25.0-android, > 25.0-jre, 25.1-android, 25.1-jre, 26.0-android, 26.0-jre, 27.0-android, > 27.0-jre, 27.0.1-android, 27.0.1-jre, 27.1-android, 27.1-jre, 28.0-android, > 28.0-jre, 28.1-android, 28.1-jre, 28.2-android, 28.2-jre > > Vulnerable Library Version: org.apache.hive : hive-exec : 1.2.1 > CVE ID: > [CVE-2018-11777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11777), > > [CVE-2015-7521](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521), > [CVE-2018-1314](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1314) > Import Path: flink-connectors/flink-connector-hive/pom.xml > Suggested Safe Versions: 2.3.4, 2.3.5, 2.3.6, 3.1.1, 3.1.2 > > Vulnerable Library Version: org.apache.hive : hive-exec : 2.0.0 > CVE ID: > [CVE-2018-11777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11777), > [CVE-2018-1314](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1314) > Import Path: flink-connectors/flink-connector-hive/pom.xml > Suggested Safe Versions: 2.3.4, 2.3.5, 2.3.6, 3.1.1, 3.1.2 > > Vulnerable Library Version: org.apache.hive : hive-exec : 1.1.0 > CVE ID: > [CVE-2018-11777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11777), > > [CVE-2015-7521](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521), > [CVE-2018-1314](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1314) > Import Path: flink-connectors/flink-connector-hive/pom.xml > Suggested Safe Versions: 2.3.4, 2.3.5, 2.3.6, 3.1.1, 3.1.2 > > Vulnerable Library Version: org.apache.hive : hive-exec : 2.1.1 > CVE ID: > [CVE-2017-12625](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-12625), > > [CVE-2018-11777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11777), > [CVE-2018-1314](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1314) > Import Path: flink-connectors/flink-connector-hive/pom.xml > Suggested Safe Versions: 2.3.4, 2.3.5, 2.3.6, 3.1.1, 3.1.2 > > Vulnerable Library Version: org.apache.hive : hive-exec : 1.0.1 > CVE ID: > [CVE-2018-11777](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11777), > > [CVE-2015-7521](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521), > [CVE-2018-1314](https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1314) > Import Path: flink-connectors/flink-connector-hive/pom.xml > Suggested Safe Versions: 2.3.4, 2.3.5, 2.3.6, 3.1.1, 3.1.2 > Vulnerable Library Version: org.apache.hive : hive-exec : 2.2.0 > CVE ID: >
[jira] [Updated] (FLINK-16294) Support to create non-existed table in database automatically when writing data to JDBC connector
[ https://issues.apache.org/jira/browse/FLINK-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16294: --- Labels: auto-deprioritized-major auto-unassigned pull-request-available stale-minor usability (was: auto-deprioritized-major auto-unassigned pull-request-available usability) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support to create non-existed table in database automatically when writing > data to JDBC connector > - > > Key: FLINK-16294 > URL: https://issues.apache.org/jira/browse/FLINK-16294 > Project: Flink > Issue Type: Improvement > Components: Connectors / JDBC, Table SQL / Ecosystem >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available, stale-minor, usability > Time Spent: 10m > Remaining Estimate: 0h > > Kafka connector/Elasticsearch connector support create topic/index > automatically when topic/index not exists in kafka/Elasticsearch from now. > This issue aims to support JDBC connector can create database table > automatically which will be more friendly to user. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17117) There are an useless cast operation for sql on blink when generate code
[
https://issues.apache.org/jira/browse/FLINK-17117?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17117:
---
Labels: auto-unassigned pull-request-available stale-minor (was:
auto-unassigned pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> There are an useless cast operation for sql on blink when generate code
>
>
> Key: FLINK-17117
> URL: https://issues.apache.org/jira/browse/FLINK-17117
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: hehuiyuan
>Priority: Minor
> Labels: auto-unassigned, pull-request-available, stale-minor
> Attachments: image-2020-04-13-19-44-19-174.png
>
>
> !image-2020-04-13-19-44-19-174.png|width=641,height=305!
>
> This mehthod `generateOneInputStreamOperator` when OperatorCodeGenerator
> generates SourceConversion:
> {code:java}
> @Override
> public void processElement($STREAM_RECORD $ELEMENT) throws Exception {
> $inputTypeTerm $inputTerm = ($inputTypeTerm)
> ${converter(s"$ELEMENT.getValue()")};
> ${ctx.reusePerRecordCode()}
> ${ctx.reuseLocalVariableCode()}
> ${if (lazyInputUnboxingCode) "" else ctx.reuseInputUnboxingCode()}
> $processCode
> }
> {code}
>
> {code:java}
> $inputTypeTerm $inputTerm = ($inputTypeTerm)
> ${converter(s"$ELEMENT.getValue()")};
> {code}
> ScanUtil calls generateOneInputStreamOperator
> {code:java}
> val generatedOperator =
> OperatorCodeGenerator.generateOneInputStreamOperator[Any, BaseRow](
> ctx,
> convertName,
> processCode,
> outputRowType,
> converter = inputTermConverter)
> //inputTermConverter
> val (inputTermConverter, inputRowType) = {
> val convertFunc = CodeGenUtils.genToInternal(ctx, inputType)
> internalInType match {
> case rt: RowType => (convertFunc, rt)
> case _ => ((record: String) => s"$GENERIC_ROW.of(${convertFunc(record)})",
> RowType.of(internalInType))
> }
> }
> {code}
> CodeGenUtils.scala : genToInternal
> {code:java}
> def genToInternal(ctx: CodeGeneratorContext, t: DataType): String => String =
> {
> val iTerm = boxedTypeTermForType(fromDataTypeToLogicalType(t))
> if (isConverterIdentity(t)) {
> term => s"($iTerm) $term"
> } else {
> val eTerm = boxedTypeTermForExternalType(t)
> val converter = ctx.addReusableObject(
> DataFormatConverters.getConverterForDataType(t),
> "converter")
> term => s"($iTerm) $converter.toInternal(($eTerm) $term)"
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-17133) Support CsvParser Features for CSV format via SQL DDL
[ https://issues.apache.org/jira/browse/FLINK-17133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-17133: --- Labels: auto-deprioritized-major pull-request-available stale-minor (was: auto-deprioritized-major pull-request-available) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Support CsvParser Features for CSV format via SQL DDL > - > > Key: FLINK-17133 > URL: https://issues.apache.org/jira/browse/FLINK-17133 > Project: Flink > Issue Type: New Feature > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Reporter: Kai Chen >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available, > stale-minor > Time Spent: 20m > Remaining Estimate: 0h > > Should we support CsvParser Features for RFC-compliant CSV format via SQL > DDL? > All CsvParser Features list in > [https://fasterxml.github.io/jackson-dataformats-text/javadoc/csv/2.10/com/fasterxml/jackson/dataformat/csv/CsvParser.Feature.html] > Such as : > IGNORE_TRAILING_UNMAPPABLE > SKIP_EMPTY_LINES > FAIL_ON_MISSING_COLUMNS > ... > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-17445) Allow OperatorTransformation to bootstrapWith a Scala DataSet
[
https://issues.apache.org/jira/browse/FLINK-17445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17445:
---
Labels: auto-deprioritized-major auto-unassigned pull-request-available
stale-minor (was: auto-deprioritized-major auto-unassigned
pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Allow OperatorTransformation to bootstrapWith a Scala DataSet
> -
>
> Key: FLINK-17445
> URL: https://issues.apache.org/jira/browse/FLINK-17445
> Project: Flink
> Issue Type: Improvement
> Components: API / State Processor
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Brandon Bevans
>Priority: Minor
> Labels: auto-deprioritized-major, auto-unassigned,
> pull-request-available, stale-minor
>
> Currently there is no way to use OperatorTransformation.bootstrapWith using a
> Scala DataSet. The functionality does seem to be there since a Scala DataSet
> appears to just be a wrapped Java DataSet. This functionality could be
> achieved with minimal invasiveness by passing through the underlying Java
> DataSet to the call.
>
> Also, the docs suggest that this function should work:
> From
> [https://ci.apache.org/projects/flink/flink-docs-stable/dev/libs/state_processor_api.html]:
> {code:java}
> val currencyDataSet = bEnv.fromCollection(CurrencyRate("USD", 1.0),
> CurrencyRate("EUR", 1.3)) val broadcastTransformation =
> OperatorTransformation .bootstrapWith(currencyDataSet) .transform(new
> CurrencyBootstrapFunction){code}
> I would love to take this issue on.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-17488) JdbcSink has to support setting autoCommit mode of DB
[
https://issues.apache.org/jira/browse/FLINK-17488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-17488:
---
Labels: auto-deprioritized-major pull-request-available stale-minor (was:
auto-deprioritized-major pull-request-available)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> JdbcSink has to support setting autoCommit mode of DB
> -
>
> Key: FLINK-17488
> URL: https://issues.apache.org/jira/browse/FLINK-17488
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / JDBC
>Affects Versions: 1.10.0
>Reporter: Khokhlov Pavel
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available,
> stale-minor
>
> Just played with new
> {noformat}
> org.apache.flink.api.java.io.jdbc.JdbcSink{noformat}
> ({{1.11-SNAPSHOT)}}
> [(https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/jdbc.html|https://ci.apache.org/projects/flink/flink-docs-master/dev/connectors/jdbc.html])
> And batch mode with mysql driver (8.0.19).
> Noticed that *JdbcSink* supports only *autoCommit true* and developer cannot
> change that behaviour. But it's very important from Transactional and
> Performance point of view to support autoCommit {color:#00875a}*false*
> {color:#172b4d}and call commit explicitly. {color}{color}
> When a connection is created, it is in auto-commit mode. This means that
> each individual SQL statement is treated as a transaction and is
> automatically committed right after it is executed.
> For example Confluent connector disable it by default.
> [https://github.com/confluentinc/kafka-connect-jdbc/blob/da9619af1d7442dd91793dbc4dc65b8e7414e7b5/src/main/java/io/confluent/connect/jdbc/sink/JdbcDbWriter.java#L50]
>
> As I see you added it only for JDBCInputFormat in: FLINK-12198
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-16175) Add config option to switch case sensitive for column names in SQL
[ https://issues.apache.org/jira/browse/FLINK-16175?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flink Jira Bot updated FLINK-16175: --- Labels: auto-deprioritized-major auto-unassigned pull-request-available stale-minor usability (was: auto-deprioritized-major auto-unassigned pull-request-available usability) I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help the community manage its development. I see this issues has been marked as Minor but is unassigned and neither itself nor its Sub-Tasks have been updated for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is still Minor, please either assign yourself or give an update. Afterwards, please remove the label or in 7 days the issue will be deprioritized. > Add config option to switch case sensitive for column names in SQL > -- > > Key: FLINK-16175 > URL: https://issues.apache.org/jira/browse/FLINK-16175 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Priority: Minor > Labels: auto-deprioritized-major, auto-unassigned, > pull-request-available, stale-minor, usability > Time Spent: 10m > Remaining Estimate: 0h > > Flink SQL is default CaseSensitive and have no option to config. This issue > aims to support > a configOption so that user can set CaseSensitive for their SQL. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26389) Many operators of ”flink-end-to-end-tests“ have been deprecated and need to be updated
[ https://issues.apache.org/jira/browse/FLINK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daisy Tsang updated FLINK-26389: External issue URL: https://github.com/apache/flink/pull/19414 (was: https://github.com/infoverload/flink/tree/dt/FLINK-26389) > Many operators of ”flink-end-to-end-tests“ have been deprecated and need to > be updated > > > Key: FLINK-26389 > URL: https://issues.apache.org/jira/browse/FLINK-26389 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.14.3 >Reporter: ChengyunZhang >Assignee: Daisy Tsang >Priority: Major > > Many functions of ”flink-end-to-end-tests“ have been deprecated .the > specific contents are as follows: > 1.org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSourceTest：the > "collect" on line 147 has been deprecated； > 2.org.apache.flink.streaming.tests.PeriodicStreamingJob：both the "keyby" on > line 68 and the "writeAsText" on line 70 have been deprecated; > 3.FileSinkProgram：the "keyBy" . line 89 has been deprecated ; > 4.org.apache.flink.streaming.tests.DistributedCacheViaBlobTestProgram: the > "writeAsText" on line 68 has been deprecated > 5.org.apache.flink.streaming.tests.RocksDBStateMemoryControlTestProgram:the > "assignTimestampsAndWatermarks" on line 64 has been deprecated; > 6.org.apache.flink.streaming.tests.StatefulStreamJobUpgradeTestProgram: the > "assignTimestampsAndWatermarks" on line 96 has been deprecated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #19414: [FLINK 26389][tests] update deprecated operators in e2e tests
flinkbot commented on PR #19414: URL: https://github.com/apache/flink/pull/19414#issuecomment-1093869397 ## CI report: * 11e4ba949253a3894a0d264afdf929c4a35fca47 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] infoverload opened a new pull request, #19414: [FLINK 26389][tests] update deprecated operators in e2e tests
infoverload opened a new pull request, #19414: URL: https://github.com/apache/flink/pull/19414 ## What is the purpose of the change (https://issues.apache.org/jira/browse/FLINK-26389)](https://issues.apache.org/jira/browse/FLINK-26389) ## Brief change log Many operators of ”flink-end-to-end-tests“ have been deprecated and need to be updated. More details are listed in the JIRA issue. ## Verifying this change This change is a code cleanup on existing e2e test. This change is already covered by existing tests, such as *(please describe tests)*. CI tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16078) Translate "Tuning Checkpoints and Large State" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519944#comment-17519944 ] Zhuang Liu commented on FLINK-16078: - [~yunta]，I submitted a pr for translation of this document, please help me with document review, thanks. > Translate "Tuning Checkpoints and Large State" page into Chinese > > > Key: FLINK-16078 > URL: https://issues.apache.org/jira/browse/FLINK-16078 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.11.0 >Reporter: Yu Li >Assignee: Zhuang Liu >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0 > > > Complete the translation in `docs/ops/state/large_state_tuning.zh.md` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #19413: [FLINK-16078] [docs-zh] Translate "Tuning Checkpoints and Large State…
flinkbot commented on PR #19413: URL: https://github.com/apache/flink/pull/19413#issuecomment-1093840338 ## CI report: * 5313a08b9e63d286641dc58f95101cf407278623 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-16078) Translate "Tuning Checkpoints and Large State" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-16078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-16078: --- Labels: auto-unassigned pull-request-available (was: auto-unassigned) > Translate "Tuning Checkpoints and Large State" page into Chinese > > > Key: FLINK-16078 > URL: https://issues.apache.org/jira/browse/FLINK-16078 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.11.0 >Reporter: Yu Li >Assignee: Zhuang Liu >Priority: Major > Labels: auto-unassigned, pull-request-available > Fix For: 1.15.0 > > > Complete the translation in `docs/ops/state/large_state_tuning.zh.md` -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] liuzhuang2017 opened a new pull request, #19413: [FLINK-16078] [docs-zh] Translate "Tuning Checkpoints and Large State…
liuzhuang2017 opened a new pull request, #19413: URL: https://github.com/apache/flink/pull/19413 …" page into Chinese ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-27151) Group By Aggregation With Sorted Data
Karthik created FLINK-27151: --- Summary: Group By Aggregation With Sorted Data Key: FLINK-27151 URL: https://issues.apache.org/jira/browse/FLINK-27151 Project: Flink Issue Type: Bug Components: Table SQL / API, Table SQL / Client Affects Versions: 1.13.6 Reporter: Karthik Hi, I want To aggregate a table by grouping some of the columns. But after grouping the columns when I send the data to UDF aggregate function it is not going in sorted order. Is there any way to do that? Thank You. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26389) Many operators of ”flink-end-to-end-tests“ have been deprecated and need to be updated
[ https://issues.apache.org/jira/browse/FLINK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519927#comment-17519927 ] Daisy Tsang commented on FLINK-26389: - I have made some changes, but it may relate to https://issues.apache.org/jira/browse/FLINK-26692 and the test here ([https://github.com/apache/flink/pull/19152/files)] might need to be changed. > Many operators of ”flink-end-to-end-tests“ have been deprecated and need to > be updated > > > Key: FLINK-26389 > URL: https://issues.apache.org/jira/browse/FLINK-26389 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.14.3 >Reporter: ChengyunZhang >Assignee: Daisy Tsang >Priority: Major > > Many functions of ”flink-end-to-end-tests“ have been deprecated .the > specific contents are as follows: > 1.org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSourceTest：the > "collect" on line 147 has been deprecated； > 2.org.apache.flink.streaming.tests.PeriodicStreamingJob：both the "keyby" on > line 68 and the "writeAsText" on line 70 have been deprecated; > 3.FileSinkProgram：the "keyBy" . line 89 has been deprecated ; > 4.org.apache.flink.streaming.tests.DistributedCacheViaBlobTestProgram: the > "writeAsText" on line 68 has been deprecated > 5.org.apache.flink.streaming.tests.RocksDBStateMemoryControlTestProgram:the > "assignTimestampsAndWatermarks" on line 64 has been deprecated; > 6.org.apache.flink.streaming.tests.StatefulStreamJobUpgradeTestProgram: the > "assignTimestampsAndWatermarks" on line 96 has been deprecated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26389) Many operators of ”flink-end-to-end-tests“ have been deprecated and need to be updated
[ https://issues.apache.org/jira/browse/FLINK-26389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daisy Tsang updated FLINK-26389: External issue URL: https://github.com/infoverload/flink/tree/dt/FLINK-26389 > Many operators of ”flink-end-to-end-tests“ have been deprecated and need to > be updated > > > Key: FLINK-26389 > URL: https://issues.apache.org/jira/browse/FLINK-26389 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.14.3 >Reporter: ChengyunZhang >Assignee: Daisy Tsang >Priority: Major > > Many functions of ”flink-end-to-end-tests“ have been deprecated .the > specific contents are as follows: > 1.org.apache.flink.streaming.connectors.gcp.pubsub.EmulatedPubSubSourceTest：the > "collect" on line 147 has been deprecated； > 2.org.apache.flink.streaming.tests.PeriodicStreamingJob：both the "keyby" on > line 68 and the "writeAsText" on line 70 have been deprecated; > 3.FileSinkProgram：the "keyBy" . line 89 has been deprecated ; > 4.org.apache.flink.streaming.tests.DistributedCacheViaBlobTestProgram: the > "writeAsText" on line 68 has been deprecated > 5.org.apache.flink.streaming.tests.RocksDBStateMemoryControlTestProgram:the > "assignTimestampsAndWatermarks" on line 64 has been deprecated; > 6.org.apache.flink.streaming.tests.StatefulStreamJobUpgradeTestProgram: the > "assignTimestampsAndWatermarks" on line 96 has been deprecated. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (FLINK-26985) With legacy restore mode, incremental checkpoints would be deleted by mistake
[
https://issues.apache.org/jira/browse/FLINK-26985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Roman Khachatryan resolved FLINK-26985.
---
Resolution: Fixed
Merged into master as c0936deaf99390fc727acc8633e3be22e62f4bf5,
into release-1.15 as 5afff68c89c5e51a55f135109089c22d073b507f.
> With legacy restore mode, incremental checkpoints would be deleted by mistake
> -
>
> Key: FLINK-26985
> URL: https://issues.apache.org/jira/browse/FLINK-26985
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.15.0, 1.16.0
>Reporter: Yun Tang
>Assignee: Roman Khachatryan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.15.0, 1.16.0
>
>
> Before flink-1.15, the restored checkpoint would be regsiterd and not discard
> on subsume, which means the restored incremental checkpoint would have one
> more reference counting to avoid discard.
> However, after state registry refactored, we could delete artificats in the
> restored incremental checkpoint with legacy restore mode.
> The error could be reproduced via
> {{ResumeCheckpointManuallyITCase#testExternalizedIncrementalRocksDBCheckpointsStandalone}}
> in my [local branch|https://github.com/Myasuka/flink/tree/legacy-error].
> Thanks for [~masteryhx] who found this problem in the manual test.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] rkhachatryan merged pull request #19412: [BP-1.15][FLINK-26985][runtime] Don't discard shared state of restored checkpoints
rkhachatryan merged PR #19412: URL: https://github.com/apache/flink/pull/19412 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #19412: [BP-1.15][FLINK-26985][runtime] Don't discard shared state of restored checkpoints
rkhachatryan commented on PR #19412: URL: https://github.com/apache/flink/pull/19412#issuecomment-1093791544 Build failure unrelated, the failed phase [succeeded](https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=1551=logs=ab6e269b-88b2-5ded-2544-4aa5b1124530=0a15d512-44ac-5ba5-97ab-13a5d066c22c) for the private branch, merging. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-27137) Remove usage of AdminClient from KafkaSource logic
[ https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519919#comment-17519919 ] Gabor Somogyi commented on FLINK-27137: --- Yep, Kafka community thinks that all offset management MUST be done w/ AdminClient. Majority of the Kafka community agrees that this is a strategic direction. Consumers can be used for offset fetching but if any issue arise then they suggest to switch to AdminClient and not willing to fix Consumer. We've spent several months on SPARK-32032 to discover all the issues and possibilities... > Remove usage of AdminClient from KafkaSource logic > -- > > Key: FLINK-27137 > URL: https://issues.apache.org/jira/browse/FLINK-27137 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.15.0, 1.14.4 >Reporter: Gyula Fora >Priority: Major > > Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses > the Kafka AdminClient instead of the KafkaConsumer. > It seems that the KafkaConsumer already provides all the necessary > information that the enumerator needs so there is no reason for introducing > the AdminClient. > In some environments using the AdminClient can be problematic even if we are > not using certaing features. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-26926) Allow users to force upgrade even if savepoint is in progress
[ https://issues.apache.org/jira/browse/FLINK-26926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519913#comment-17519913 ] Gyula Fora commented on FLINK-26926: The conclusion was to include these custom configuration options directly in the `flinkConfiguration` > Allow users to force upgrade even if savepoint is in progress > - > > Key: FLINK-26926 > URL: https://issues.apache.org/jira/browse/FLINK-26926 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Gyula Fora >Priority: Major > Fix For: kubernetes-operator-1.0.0 > > > Currently all upgrades (regardless of upgrade mode) are delayed as long as > there is a pending savepoint operation. > We should allow users to override this and execute the upgrade (thus > potentially cancelling the savepoint) regardless of the savepoint status. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-27137) Remove usage of AdminClient from KafkaSource logic
[ https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gyula Fora closed FLINK-27137. -- Resolution: Won't Do > Remove usage of AdminClient from KafkaSource logic > -- > > Key: FLINK-27137 > URL: https://issues.apache.org/jira/browse/FLINK-27137 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.15.0, 1.14.4 >Reporter: Gyula Fora >Priority: Major > > Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses > the Kafka AdminClient instead of the KafkaConsumer. > It seems that the KafkaConsumer already provides all the necessary > information that the enumerator needs so there is no reason for introducing > the AdminClient. > In some environments using the AdminClient can be problematic even if we are > not using certaing features. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-27137) Remove usage of AdminClient from KafkaSource logic
[ https://issues.apache.org/jira/browse/FLINK-27137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17519912#comment-17519912 ] Gyula Fora commented on FLINK-27137: Thanks [~dengziming] , I can see that AdminClient is more targeted for offset management (even though the consumer seems to cover all our current use-cases) talked to [~gaborgsomogyi] and he actually pointed out some practical limitations of fetching the offsets through the consumer. That sounds like a more convincing argument. We will try to deal with our current issues on our end. Thanks all! > Remove usage of AdminClient from KafkaSource logic > -- > > Key: FLINK-27137 > URL: https://issues.apache.org/jira/browse/FLINK-27137 > Project: Flink > Issue Type: Improvement > Components: Connectors / Kafka >Affects Versions: 1.15.0, 1.14.4 >Reporter: Gyula Fora >Priority: Major > > Parts of the KafkaSource logic (specifically the KafkaSourceEnumerator) uses > the Kafka AdminClient instead of the KafkaConsumer. > It seems that the KafkaConsumer already provides all the necessary > information that the enumerator needs so there is no reason for introducing > the AdminClient. > In some environments using the AdminClient can be problematic even if we are > not using certaing features. -- This message was sent by Atlassian Jira (v8.20.1#820001)
