[GitHub] [flink] chucheng92 commented on pull request #21442: [FLINK-30274] Upgrade commons-collections 3.x to commons-collections4
chucheng92 commented on PR #21442: URL: https://github.com/apache/flink/pull/21442#issuecomment-1425353481 @MartijnVisser thanks. i rebased it. ci passed. PTAL -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] zhipeng93 commented on pull request #206: [FLINK-30933] Fix missing max watermark when executing join in iteration body
zhipeng93 commented on PR #206: URL: https://github.com/apache/flink-ml/pull/206#issuecomment-1425353355 > As a whole LGTM, and @zhipeng93 could you modify the test to also check the max watermark is indeed emitted before termination? Thanks for the reply. The unit test here [1] already verifies that the max watermark is emitted. If there is no max watermark, there should be no output and the unit test should fail. Do you mean we directly check that max watermark is emitted? [1] https://github.com/apache/flink-ml/pull/206/files#diff-0d6c01f07abbec3796dfa449243437549c19615dd6439a27af08b6b68005e8a4R127 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a diff in pull request #21889: [FLINK-29801][runtime] FLIP-274: Introduce metric group for OperatorCoordinator
ruanhang1993 commented on code in PR #21889:
URL: https://github.com/apache/flink/pull/21889#discussion_r1102391367
##
flink-core/src/main/java/org/apache/flink/configuration/MetricOptions.java:
##
@@ -272,6 +272,17 @@ public static Configuration forReporter(Configuration
configuration, String repo
.withDescription(
"Defines the scope format string that is applied
to all metrics scoped to a job on a JobManager. Only effective when a
identifier-based reporter is configured");
+/**
+ * The scope format string that is applied to all metrics scoped to the
components running on a
+ * JobManager of an operator.
+ */
+public static final ConfigOption SCOPE_NAMING_JM_OPERATOR =
+key("metrics.scope.jm-operator")
+.stringType()
+
.defaultValue(".jobmanager..")
Review Comment:
I will add the docs about it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on pull request #21889: [FLINK-29801][runtime] FLIP-274: Introduce metric group for OperatorCoordinator
ruanhang1993 commented on PR #21889: URL: https://github.com/apache/flink/pull/21889#issuecomment-142564 Hi, @zentol, Thanks for your reviewing. This PR may contain too much code. I plan to split it into 2 PRs. This one is for the JobManagerOperatorMetricGroup, the other is for the InternalOperatorCoordinatorMetricGroup. I will add the missing part and reorganize the commits later. Thanks~ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order
[
https://issues.apache.org/jira/browse/FLINK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686921#comment-17686921
]
ming li commented on FLINK-31008:
-
[~lzljs3620320] Yes, thanks a lot, I've been doing some related work recently
that I think will help fix this issue.
In addition, if you have time, please help to take a look at this
issue([FLINK-30985|https://issues.apache.org/jira/browse/FLINK-30985]), I think
it is also helpful for split allocation.
> [Flink][Table Store] The Split allocation of the same bucket in
> ContinuousFileSplitEnumerator may be out of order
> -
>
> Key: FLINK-31008
> URL: https://issues.apache.org/jira/browse/FLINK-31008
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: ming li
>Priority: Major
>
> There are two places in {{ContinuousFileSplitEnumerator}} that add
> {{FileStoreSourceSplit}} to {{{}bucketSplits{}}}: {{addSplitsBack}} and
> {{{}processDiscoveredSplits{}}}. {{processDiscoveredSplits}} will
> continuously check for new splits and add them to the queue. At this time,
> the order of the splits is in order.
> {code:java}
> private void addSplits(Collection splits) {
> splits.forEach(this::addSplit);
> }
> private void addSplit(FileStoreSourceSplit split) {
> bucketSplits
> .computeIfAbsent(((DataSplit) split.split()).bucket(), i -> new
> LinkedList<>())
> .add(split);
> }{code}
> However, when the task failover, the splits that have been allocated before
> will be returned. At this time, these returned splits are also added to the
> end of the queue, which leads to disorder in the allocation of splits.
>
> I think these returned splits should be added to the head of the queue to
> ensure the order of allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30870) Performance regressions notifications in Slack are cut off
[
https://issues.apache.org/jira/browse/FLINK-30870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martijn Visser closed FLINK-30870.
--
Resolution: Fixed
Fixed in flink-benchmarks master: 1a6da18e0933b6e43297c1fc44a556c0c162b3e0
> Performance regressions notifications in Slack are cut off
> --
>
> Key: FLINK-30870
> URL: https://issues.apache.org/jira/browse/FLINK-30870
> Project: Flink
> Issue Type: Bug
> Components: Benchmarks
>Reporter: Martijn Visser
>Assignee: Yanfei Lei
>Priority: Major
> Labels: pull-request-available
> Attachments: image-2023-02-06-12-26-57-268.png
>
>
> Example from today at
> https://apache-flink.slack.com/archives/C0471S0DFJ9/p1675253720571659
> {code}
> Performance regression
> mapRebalanceMapSink.F27_UNBOUNDED median=17231.7398765
> recent_median=16165.0549395
> multiInputOneIdleMapSink median=11254.5329375 recent_median=10727.7280915
> calculateRegionToRestart.BATCH median=12.881527 recent_median=12.096391
> partitionRelease.BATCH median=23.2130145 recent_median=21.4858475
> checkpointMultiInput median=2.6094395 recent_median=2.477736
> checkpointSingleInput.UNALIGNED median=339.229515 recent_median=67.2695295
> checkpointSingleInput.UNALIGNED_1 median=215.2789775 recent_median=40.1294965
> fireProcessingTimers median=50.9977185 recent_median=44.0925955
> globalWindow median=5459.689767 recent_median=5045.436655
> {code}
> As you can see, the last part is cut off with {{ sure if this is because there are more regressions, but they aren't posted to
> Slack (could be) or if this is just a mistake in the output and the
> {{globalWindow}} was the last regression to be reported. It would be great if
> this could be validated.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] reswqa commented on a diff in pull request #21890: [FLINK-30860][doc] Add document for hybrid shuffle with adaptive batch scheduler
reswqa commented on code in PR #21890:
URL: https://github.com/apache/flink/pull/21890#discussion_r1102386579
##
docs/content.zh/docs/ops/batch/batch_shuffle.md:
##
@@ -114,12 +114,34 @@ Hybrid shuffle provides two spilling strategies:
To use hybrid shuffle mode, you need to configure the
[execution.batch-shuffle-mode]({{< ref "docs/deployment/config"
>}}#execution-batch-shuffle-mode) to `ALL_EXCHANGES_HYBRID_FULL` (full spilling
strategy) or `ALL_EXCHANGES_HYBRID_SELECTIVE` (selective spilling strategy).
+ Supports AdaptiveBatchScheduler and SpeculativeExecution
+
+Hybrid shuffle currently supports `AdaptiveBatchScheduler` by default. If you
want to use `DefaultScheduler`, please configure the [jobmanager.scheduler]({{<
ref "docs/deployment/config" >}}#jobmanager-scheduler) to `DefaultScheduler`.
See [elastic_scaling]({{< ref "docs/deployment/elastic_scaling"
>}}#adaptive-batch-scheduler) for details.
+
+If you want to enable `SpeculativeExecution` in the same time, see
[speculative_execution]({{< ref "docs/deployment/speculative_execution" >}})
for details.
+
+Hybrid shuffle divides the partition data consumption constraints between
producer and consumer into the following three cases:
+
+- **ALL_PRODUCERS_FINISHED** : hybrid partition data can be consumed only when
all producers are finished.
+- **ONLY_FINISHED_PRODUCERS** : hybrid partition data can be consumed when its
producer is finished.
+- **UNFINISHED_PRODUCERS** : hybrid partition data can be consumed even if its
producer is un-finished.
+
+If `SpeculativeExecution` is enabled, the default constraint is
`ONLY_FINISHED_PRODUCERS` to bring some performance optimization compared with
blocking shuffle. Otherwise, the default constraint is `UNFINISHED_PRODUCERS`
to perform pipelined-like shuffle. These could be configured via
[jobmanager.partition.hybrid.partition-data-consume-constraint]({{< ref
"docs/deployment/config"
>}}#jobmanager-partition-hybrid-partition-data-consume-constraint).
Review Comment:
Added some descriptions of this part.
##
docs/content.zh/docs/ops/batch/batch_shuffle.md:
##
@@ -114,12 +114,34 @@ Hybrid shuffle provides two spilling strategies:
To use hybrid shuffle mode, you need to configure the
[execution.batch-shuffle-mode]({{< ref "docs/deployment/config"
>}}#execution-batch-shuffle-mode) to `ALL_EXCHANGES_HYBRID_FULL` (full spilling
strategy) or `ALL_EXCHANGES_HYBRID_SELECTIVE` (selective spilling strategy).
+ Supports AdaptiveBatchScheduler and SpeculativeExecution
+
+Hybrid shuffle currently supports `AdaptiveBatchScheduler` by default. If you
want to use `DefaultScheduler`, please configure the [jobmanager.scheduler]({{<
ref "docs/deployment/config" >}}#jobmanager-scheduler) to `DefaultScheduler`.
See [elastic_scaling]({{< ref "docs/deployment/elastic_scaling"
>}}#adaptive-batch-scheduler) for details.
+
+If you want to enable `SpeculativeExecution` in the same time, see
[speculative_execution]({{< ref "docs/deployment/speculative_execution" >}})
for details.
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] reswqa commented on a diff in pull request #21890: [FLINK-30860][doc] Add document for hybrid shuffle with adaptive batch scheduler
reswqa commented on code in PR #21890:
URL: https://github.com/apache/flink/pull/21890#discussion_r1102385798
##
docs/content.zh/docs/ops/batch/batch_shuffle.md:
##
@@ -114,12 +114,34 @@ Hybrid shuffle provides two spilling strategies:
To use hybrid shuffle mode, you need to configure the
[execution.batch-shuffle-mode]({{< ref "docs/deployment/config"
>}}#execution-batch-shuffle-mode) to `ALL_EXCHANGES_HYBRID_FULL` (full spilling
strategy) or `ALL_EXCHANGES_HYBRID_SELECTIVE` (selective spilling strategy).
+ Supports AdaptiveBatchScheduler and SpeculativeExecution
+
+Hybrid shuffle currently supports `AdaptiveBatchScheduler` by default. If you
want to use `DefaultScheduler`, please configure the [jobmanager.scheduler]({{<
ref "docs/deployment/config" >}}#jobmanager-scheduler) to `DefaultScheduler`.
See [elastic_scaling]({{< ref "docs/deployment/elastic_scaling"
>}}#adaptive-batch-scheduler) for details.
+
+If you want to enable `SpeculativeExecution` in the same time, see
[speculative_execution]({{< ref "docs/deployment/speculative_execution" >}})
for details.
+
+Hybrid shuffle divides the partition data consumption constraints between
producer and consumer into the following three cases:
+
+- **ALL_PRODUCERS_FINISHED** : hybrid partition data can be consumed only when
all producers are finished.
+- **ONLY_FINISHED_PRODUCERS** : hybrid partition data can be consumed when its
producer is finished.
+- **UNFINISHED_PRODUCERS** : hybrid partition data can be consumed even if its
producer is un-finished.
+
+If `SpeculativeExecution` is enabled, the default constraint is
`ONLY_FINISHED_PRODUCERS` to bring some performance optimization compared with
blocking shuffle. Otherwise, the default constraint is `UNFINISHED_PRODUCERS`
to perform pipelined-like shuffle. These could be configured via
[jobmanager.partition.hybrid.partition-data-consume-constraint]({{< ref
"docs/deployment/config"
>}}#jobmanager-partition-hybrid-partition-data-consume-constraint).
+
+ Index Spilling
+
+Hybrid shuffle indexes the shuffle data in memory and disk. Generally
speaking, all index can be cached in memory to speed up index retrieval.
However, for large batch jobs, this part of memory may bring OOM risks.
+Therefore, hybrid shuffle supports spilling index data to disk. The following
configuration options can control this behavior:
+
+-
**[taskmanager.network.hybrid-shuffle.num-retained-in-memory-regions-max]({{<
ref "docs/deployment/config"
>}}#taskmanager-network-hybrid-shuffle-num-retained-in-memory-regions-max)** :
Controls the max number of hybrid retained regions in memory. Increasing this
value will allow more index entries to be cached in memory.
+- **[taskmanager.network.hybrid-shuffle.spill-index-segment-size]({{< ref
"docs/deployment/config"
>}}#taskmanager-network-hybrid-shuffle-spill-index-segment-size)** : Controls
the segment size(in bytes) of hybrid spilled file data index.
Review Comment:
Agree with you, removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-30924) Conversion issues between timestamp and bingint
[
https://issues.apache.org/jira/browse/FLINK-30924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu closed FLINK-30924.
--
Resolution: Not A Problem
> Conversion issues between timestamp and bingint
> ---
>
> Key: FLINK-30924
> URL: https://issues.apache.org/jira/browse/FLINK-30924
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.16.1
>Reporter: Feng Jin
>Priority: Major
>
> When casting to convert timestamp and bigint, the following exception is
> thrown:
> {code:java}
> //代码占位符
> org.apache.flink.table.api.ValidationException: The cast from NUMERIC type to
> TIMESTAMP type is not allowed. It's recommended to use
> TO_TIMESTAMP(FROM_UNIXTIME(numeric_col)) instead, note the numeric is in
> seconds.
> {code}
> However, the FROM_UNIXTIME function will use the local time zone for
> conversion, but the TO_TIMESTAMP function will not use the local time zone
> but will use the UTC time zone conversion, so that the actual result in the
> wrong result.
>
> The following is an example of the results of the test
> {code:java}
> //代码占位符
> Flink SQL> SET 'table.local-time-zone' = 'Asia/Shanghai';
> Flink SQL> select TO_TIMESTAMP(FROM_UNIXTIME(0));
> // result
> EXPR$0
> 1970-01-01 08:00:00.000
> {code}
>
>
> UNIX_TIMESTAMP(CAST(timestamp_col AS STRING)) has the same problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-benchmarks] MartijnVisser merged pull request #65: [FLINK-30870][benchmark] Replace sending slack messages with attachments
MartijnVisser merged PR #65: URL: https://github.com/apache/flink-benchmarks/pull/65 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a diff in pull request #21889: [FLINK-29801][runtime] FLIP-274: Introduce metric group for OperatorCoordinator
ruanhang1993 commented on code in PR #21889:
URL: https://github.com/apache/flink/pull/21889#discussion_r1102384957
##
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/groups/InternalOperatorCoordinatorMetricGroup.java:
##
@@ -0,0 +1,33 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.metrics.groups;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.metrics.MetricGroup;
+import org.apache.flink.metrics.groups.OperatorCoordinatorMetricGroup;
+
+/** Special {@link org.apache.flink.metrics.MetricGroup} representing an
OperatorCoordinator. */
+@Internal
+public class InternalOperatorCoordinatorMetricGroup extends
ProxyMetricGroup
+implements OperatorCoordinatorMetricGroup {
+
+public
InternalOperatorCoordinatorMetricGroup(JobManagerOperatorMetricGroup parent) {
Review Comment:
I think the InternalOperatorCoordinatorMetricGroup should be only registered
under the JobManagerOperatorMetricGroup. So I add this limit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-30945) FTS does not support multiple writers into the same table and topic
[
https://issues.apache.org/jira/browse/FLINK-30945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686919#comment-17686919
]
Jingsong Lee commented on FLINK-30945:
--
[~vicky_papavas] Thanks for reporting! I think this needs to be improved!
> FTS does not support multiple writers into the same table and topic
> ---
>
> Key: FLINK-30945
> URL: https://issues.apache.org/jira/browse/FLINK-30945
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Vicky Papavasileiou
>Priority: Major
>
> When creating two different streaming jobs that INSERT INTO the same table
> and kafka topic, the second job is never able to make progress as the
> transaction gets constantly aborted due to the producer getting fenced.
> FTS should set the transactionalIdPrefix to avoid transactions of different
> jobs clashing.
> {code:java}
> 2023-02-06 17:13:36,088 WARN org.apache.flink.runtime.taskmanager.Task [] -
> Writer -> Global Committer -> Sink: end (1/1)#0
> (8cf4197af9716623c3c19e7fa3d7c071_b5c8d46f3e7b141acf271f12622e752b_0_0)
> switched from RUNNING to FAILED with failure cause:
> org.apache.flink.util.FlinkRuntimeException: Committing one of transactions
> failed, logging first encountered failure at
> org.apache.flink.streaming.api.functions.sink.TwoPhaseCommitSinkFunction.notifyCheckpointComplete(TwoPhaseCommitSinkFunction.java:323)
> at
> org.apache.flink.table.store.connector.sink.StoreWriteOperator.notifyCheckpointComplete(StoreWriteOperator.java:175)
> at
> org.apache.flink.streaming.runtime.tasks.StreamOperatorWrapper.notifyCheckpointComplete(StreamOperatorWrapper.java:104)
> at
> org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.notifyCheckpointComplete(RegularOperatorChain.java:145)
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpoint(SubtaskCheckpointCoordinatorImpl.java:479)
> at
> org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.notifyCheckpointComplete(SubtaskCheckpointCoordinatorImpl.java:413)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.notifyCheckpointComplete(StreamTask.java:1412)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointCompleteAsync$15(StreamTask.java:1353)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$18(StreamTask.java:1392)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50)
> at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsNonBlocking(MailboxProcessor.java:383)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:345)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:831)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:780)
> at
> org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:935)
> at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:914)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:728) at
> org.apache.flink.runtime.taskmanager.Task.run(Task.java:550) at
> java.lang.Thread.run(Thread.java:750) Caused by:
> org.apache.flink.table.store.shaded.org.apache.kafka.common.errors.ProducerFencedException:
> There is a newer producer with the same transactionalId which fences the
> current one. {code}
> Sample queries:
>
>
> {code:java}
> CREATE CATALOG table_store_catalog WITH (
> 'type'='table-store',
> 'warehouse'='s3://my-bucket/table-store'
> );
> USE CATALOG table_store_catalog;
> SET 'execution.checkpointing.interval' = '10 s';
> CREATE TABLE word_count_kafka (
> word STRING PRIMARY KEY NOT ENFORCED,
> cnt BIGINT
> ) WITH (
> 'log.system' = 'kafka',
> 'kafka.bootstrap.servers' = 'broker:9092',
> 'kafka.topic' = 'word_count_log'
> );
> CREATE TEMPORARY TABLE word_table (
> word STRING
> ) WITH (
> 'connector' = 'datagen',
> 'fields.word.length' = '1'
> );
> {code}
>
> And the two INSERT jobs:
> {code:java}
> INSERT INTO word_count_kafka SELECT word, COUNT(*) FROM word_table GROUP BY
> word;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order
[
https://issues.apache.org/jira/browse/FLINK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686917#comment-17686917
]
Jingsong Lee commented on FLINK-31008:
--
[~Ming Li] Thanks for reporting!
Wow, you're right. This should be a blocker issue.
Do you want to contribute this jira?
> [Flink][Table Store] The Split allocation of the same bucket in
> ContinuousFileSplitEnumerator may be out of order
> -
>
> Key: FLINK-31008
> URL: https://issues.apache.org/jira/browse/FLINK-31008
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: ming li
>Priority: Major
>
> There are two places in {{ContinuousFileSplitEnumerator}} that add
> {{FileStoreSourceSplit}} to {{{}bucketSplits{}}}: {{addSplitsBack}} and
> {{{}processDiscoveredSplits{}}}. {{processDiscoveredSplits}} will
> continuously check for new splits and add them to the queue. At this time,
> the order of the splits is in order.
> {code:java}
> private void addSplits(Collection splits) {
> splits.forEach(this::addSplit);
> }
> private void addSplit(FileStoreSourceSplit split) {
> bucketSplits
> .computeIfAbsent(((DataSplit) split.split()).bucket(), i -> new
> LinkedList<>())
> .add(split);
> }{code}
> However, when the task failover, the splits that have been allocated before
> will be returned. At this time, these returned splits are also added to the
> end of the queue, which leads to disorder in the allocation of splits.
>
> I think these returned splits should be added to the head of the queue to
> ensure the order of allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30977) flink tumbling window stream converting to pandas dataframe not work
[
https://issues.apache.org/jira/browse/FLINK-30977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joekwal updated FLINK-30977:
Description:
I want to know if tumbling window supported to convert to pandas?
{code:java}
code... #create env
kafka_src = """
CREATE TABLE if not exists `kafka_src` (
...
`event_time` as CAST(`end_time` as TIMESTAMP(3)),
WATERMARK FOR event_time as event_time - INTERVAL '5' SECOND
)
with (
'connector' = 'kafka',
'topic' = 'topic',
'properties.bootstrap.servers' = '***',
'properties.group.id' = '***',
'scan.startup.mode' = 'earliest-offset',
'value.format' = 'debezium-json'
);
"""
t_env.execute_sql(kafka_src)
table = st_env.sql_query("SELECT columns,`event_time` \
FROM TABLE(TUMBLE(TABLE table_name, DESCRIPTOR(event_time), INTERVAL '1'
MINUTES))")
table.execute().print() #could print the result
df = table.to_pandas()
#schema is correct!
schema = DataTypes.ROW([DataTypes.FIELD("column1", DataTypes.STRING()),
...
])
table = st_env.from_pandas(df,schema=schema)
st_env.create_temporary_view("view_table",table)
st_env.sql_query("select * from view_table").execute().print() # Not work!Can't
print the result {code}
Tumbling window stream from kafka source convert to pandas dataframe and it
can't print the result.The schema is right.I have tested in another job with
using batch stream from jdbc source.It can print the result.The only different
thing is the input stream.Is tumbling windows supported to convert to Pandas?
was:
I want to know if tumbling window supported to convert to pandas?
{code:java}
code... #create env
kafka_src = """
CREATE TABLE if not exists `kafka_src` (
...
`event_time` as CAST(`end_time` as TIMESTAMP(3)),
WATERMARK FOR event_time as event_time - INTERVAL '5' SECOND
)
with (
'connector' = 'kafka',
'topic' = 'topic',
'properties.bootstrap.servers' = '***',
'properties.group.id' = '***',
'scan.startup.mode' = 'earliest-offset',
'value.format' = 'debezium-json'
);
"""
t_env.execute_sql(kafka_src)
table = st_env.sql_query("SELECT columns,`event_time` \
FROM TABLE(TUMBLE(TABLE table_name, DESCRIPTOR(event_time), INTERVAL '1'
MINUTES))")
table.execute().print() #could print the result
df = table.to_pandas()
#schema is correct!
schema = DataTypes.ROW([DataTypes.FIELD("column1", DataTypes.STRING()),
...
])
table = st_env.from_pandas(df,schema=schema)
st_env.create_temporary_view("view_table",table)
st_env.sql_query("select * from view_table").execute().print() # Not work!Can't
print the result {code}
Tumbling window stream from kafka source convert to pandas dataframe and it
can't print the result.The schema is right.I have tested in another job with
using batch stream from jdbc source.It can print the result.The only different
thing is the input stream.As doc mentioned, the bounded stream is supported to
convert to pandas.So what could have gone wrong?
> flink tumbling window stream converting to pandas dataframe not work
>
>
> Key: FLINK-30977
> URL: https://issues.apache.org/jira/browse/FLINK-30977
> Project: Flink
> Issue Type: Bug
> Components: API / Python
> Environment: pyflink1.15.2
>Reporter: Joekwal
>Priority: Major
>
> I want to know if tumbling window supported to convert to pandas?
> {code:java}
> code... #create env
> kafka_src = """
> CREATE TABLE if not exists `kafka_src` (
> ...
> `event_time` as CAST(`end_time` as TIMESTAMP(3)),
> WATERMARK FOR event_time as event_time - INTERVAL '5' SECOND
> )
> with (
> 'connector' = 'kafka',
> 'topic' = 'topic',
> 'properties.bootstrap.servers' = '***',
> 'properties.group.id' = '***',
> 'scan.startup.mode' = 'earliest-offset',
> 'value.format' = 'debezium-json'
> );
> """
>
> t_env.execute_sql(kafka_src)
> table = st_env.sql_query("SELECT columns,`event_time` \
> FROM TABLE(TUMBLE(TABLE table_name, DESCRIPTOR(event_time), INTERVAL '1'
> MINUTES))")
> table.execute().print() #could print the result
> df = table.to_pandas()
> #schema is correct!
> schema = DataTypes.ROW([DataTypes.FIELD("column1", DataTypes.STRING()),
> ...
> ])
> table = st_env.from_pandas(df,schema=schema)
> st_env.create_temporary_view("view_table",table)
> st_env.sql_query("select * from view_table").execute().print() # Not
> work!Can't print the result {code}
> Tumbling window stream from kafka source convert to pandas dataframe and it
> can't print the result.The schema is right.I have tested in another job with
> using batch stream from jdbc source.It can print the result.The only
> different thing is the input stream.Is tumbling windows supported to convert
> to Pandas?
--
This message was
[jira] [Commented] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order
[
https://issues.apache.org/jira/browse/FLINK-31008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686912#comment-17686912
]
ming li commented on FLINK-31008:
-
hi, [~lzljs3620320], if you have time, please help to take a look at this
issue, thank you.
> [Flink][Table Store] The Split allocation of the same bucket in
> ContinuousFileSplitEnumerator may be out of order
> -
>
> Key: FLINK-31008
> URL: https://issues.apache.org/jira/browse/FLINK-31008
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: ming li
>Priority: Major
>
> There are two places in {{ContinuousFileSplitEnumerator}} that add
> {{FileStoreSourceSplit}} to {{{}bucketSplits{}}}: {{addSplitsBack}} and
> {{{}processDiscoveredSplits{}}}. {{processDiscoveredSplits}} will
> continuously check for new splits and add them to the queue. At this time,
> the order of the splits is in order.
> {code:java}
> private void addSplits(Collection splits) {
> splits.forEach(this::addSplit);
> }
> private void addSplit(FileStoreSourceSplit split) {
> bucketSplits
> .computeIfAbsent(((DataSplit) split.split()).bucket(), i -> new
> LinkedList<>())
> .add(split);
> }{code}
> However, when the task failover, the splits that have been allocated before
> will be returned. At this time, these returned splits are also added to the
> end of the queue, which leads to disorder in the allocation of splits.
>
> I think these returned splits should be added to the head of the queue to
> ensure the order of allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-benchmarks] fredia commented on pull request #65: [FLINK-30870][benchmark] Replace sending slack messages with attachments
fredia commented on PR #65: URL: https://github.com/apache/flink-benchmarks/pull/65#issuecomment-1425316266 Hi @MartijnVisser, could you please help merge this PR? Thanks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31008) [Flink][Table Store] The Split allocation of the same bucket in ContinuousFileSplitEnumerator may be out of order
ming li created FLINK-31008:
---
Summary: [Flink][Table Store] The Split allocation of the same
bucket in ContinuousFileSplitEnumerator may be out of order
Key: FLINK-31008
URL: https://issues.apache.org/jira/browse/FLINK-31008
Project: Flink
Issue Type: Bug
Components: Table Store
Reporter: ming li
There are two places in {{ContinuousFileSplitEnumerator}} that add
{{FileStoreSourceSplit}} to {{{}bucketSplits{}}}: {{addSplitsBack}} and
{{{}processDiscoveredSplits{}}}. {{processDiscoveredSplits}} will continuously
check for new splits and add them to the queue. At this time, the order of the
splits is in order.
{code:java}
private void addSplits(Collection splits) {
splits.forEach(this::addSplit);
}
private void addSplit(FileStoreSourceSplit split) {
bucketSplits
.computeIfAbsent(((DataSplit) split.split()).bucket(), i -> new
LinkedList<>())
.add(split);
}{code}
However, when the task failover, the splits that have been allocated before
will be returned. At this time, these returned splits are also added to the end
of the queue, which leads to disorder in the allocation of splits.
I think these returned splits should be added to the head of the queue to
ensure the order of allocation.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30109) Checked exceptions are sneakingly transformed into unchecked exceptions in the Pulsar
[
https://issues.apache.org/jira/browse/FLINK-30109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-30109:
---
Labels: pull-request-available (was: )
> Checked exceptions are sneakingly transformed into unchecked exceptions in
> the Pulsar
> -
>
> Key: FLINK-30109
> URL: https://issues.apache.org/jira/browse/FLINK-30109
> Project: Flink
> Issue Type: Technical Debt
> Components: Connectors / Pulsar, Documentation
>Affects Versions: 1.16.0, 1.17.0, 1.15.2
>Reporter: Matthias Pohl
>Assignee: Yufan Sheng
>Priority: Major
> Labels: pull-request-available
>
> [PulsarExceptionUtils|https://github.com/apache/flink/blob/c675f786c51038801161e861826d1c54654f0dde/flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarExceptionUtils.java#L33]
> provides {{sneaky*}} utility methods for hiding checked exceptions. This is
> rather unusual coding. Based on what's provided in the code I would have
> concerns as a reader that we're not handling errors properly in calling code.
> Either, we remove these methods and add proper exception handling or we add
> proper documentation on why this workaround is necessary.
> [~syhily] already hinted in his [FLINK-29830 PR
> comment|https://github.com/apache/flink/pull/21252#discussion_r1019822514]
> that this is related to flaws of the Pulsar API.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-connector-pulsar] syhily opened a new pull request, #24: [FLINK-30109][Connector/Pulsar] Drop the use of sneaky exception.
syhily opened a new pull request, #24: URL: https://github.com/apache/flink-connector-pulsar/pull/24 ## What is the purpose of the change Remove `PulsarExceptionUtils` and handle all the checked exceptions more properly. ## Brief change log - `PulsarClientFactory` - `PulsarExceptionUtils` - `PulsarTransactionUtils` - `PulsarCommitter` - `SinkConfiguration` - `MetadataListener` - `ProducerRegister` - `PulsarWriter` - `PulsarSink` - `LatestMessageStopCursor` - `PulsarSourceEnumerator` - `PulsarPartitionSplitReader` - `PulsarSourceFetcherManager` - `PulsarSource` - `PulsarSourceBuilder` - `PulsarSchemaTest` - `MetadataListenerTest` - `ProducerRegisterTest` - `StopCursorTest` - `PulsarSubscriberTest` - `PulsarSourceEnumeratorTest` - `PulsarDeserializationSchemaTest` - `PulsarPartitionSplitReaderTest` - `PulsarSourceReaderTest` - `ControlSource` - `PulsarContainerRuntime` - `PulsarRuntime` - `PulsarRuntimeOperator` - `PulsarPartitionDataReader` - `PulsarSinkTestContext` - `MultipleTopicsConsumingContext` - `SingleTopicConsumingContext` - `KeyedPulsarPartitionDataWriter` - `PulsarEncryptDataWriter` - `PulsarPartitionDataWriter` - `PulsarTestCommonUtils` - `PulsarTestEnvironment` ## Verifying this change This change is a minor change and don't have any tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18356) flink-table-planner Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686902#comment-17686902
]
Matthias Pohl edited comment on FLINK-18356 at 2/10/23 7:21 AM:
2 independent 137 exit code failures (one in an Azure pipeline and one on
Alibaba007) making multiple jobs crash:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45978=results
It also caused a test failure in the concurrently running build because it was
operating on Alibaba007as well:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45979=logs=a549b384-c55a-52c0-c451-00e0477ab6db=eef5922c-08d9-5ba3-7299-8393476594e7
was (Author: mapohl):
2 independent 137 exit code failures (one in an Azure pipeline and one on
Alibaba005) making multiple jobs crash:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45978=results
> flink-table-planner Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0
>Reporter: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available, test-stability
> Attachments: 1234.jpg, app-profiling_4.gif,
> image-2023-01-11-22-21-57-784.png, image-2023-01-11-22-22-32-124.png
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30924) Conversion issues between timestamp and bingint
[
https://issues.apache.org/jira/browse/FLINK-30924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686908#comment-17686908
]
Feng Jin commented on FLINK-30924:
--
[~leonard] Thank you very much for the answer. In my past knowledge, I
thought that TIMESTAMP is the UTC-0 time zone by default, but in fact it should
not be bound to any time zone meaning, it can only represent the year, month,
day, hour, minute, and second.
> Conversion issues between timestamp and bingint
> ---
>
> Key: FLINK-30924
> URL: https://issues.apache.org/jira/browse/FLINK-30924
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.16.1
>Reporter: Feng Jin
>Priority: Major
>
> When casting to convert timestamp and bigint, the following exception is
> thrown:
> {code:java}
> //代码占位符
> org.apache.flink.table.api.ValidationException: The cast from NUMERIC type to
> TIMESTAMP type is not allowed. It's recommended to use
> TO_TIMESTAMP(FROM_UNIXTIME(numeric_col)) instead, note the numeric is in
> seconds.
> {code}
> However, the FROM_UNIXTIME function will use the local time zone for
> conversion, but the TO_TIMESTAMP function will not use the local time zone
> but will use the UTC time zone conversion, so that the actual result in the
> wrong result.
>
> The following is an example of the results of the test
> {code:java}
> //代码占位符
> Flink SQL> SET 'table.local-time-zone' = 'Asia/Shanghai';
> Flink SQL> select TO_TIMESTAMP(FROM_UNIXTIME(0));
> // result
> EXPR$0
> 1970-01-01 08:00:00.000
> {code}
>
>
> UNIX_TIMESTAMP(CAST(timestamp_col AS STRING)) has the same problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi commented on pull request #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
JingsongLi commented on PR #515: URL: https://github.com/apache/flink-table-store/pull/515#issuecomment-1425298328 Where to use `test-utils`? Maybe we can modify the usage too? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31007) The code generated by the IF function throws NullPointerException
[
https://issues.apache.org/jira/browse/FLINK-31007?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
tivanli updated FLINK-31007:
Attachment: StreamExecCalc$19.java
> The code generated by the IF function throws NullPointerException
> -
>
> Key: FLINK-31007
> URL: https://issues.apache.org/jira/browse/FLINK-31007
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.15.2, 1.15.3
> Environment: {code:java}
> // code placeholder
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> final StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
> final DataStream tab =
> env.fromCollection(Arrays.asList(
> new Tuple2<>(1L, "a_b_c"),
> new Tuple2<>(-1L, "a_b_c")));
> final Table tableA = tableEnv.fromDataStream(tab);
> tableEnv.executeSql("SELECT if(f0 = -1, '', split_index(f1, '_', 0)) as id
> FROM " + tableA)
> .print(); {code}
>Reporter: tivanli
>Priority: Major
> Attachments: StreamExecCalc$19.java
>
>
> Caused by: java.lang.NullPointerException
> at StreamExecCalc$19.processElement_split1(Unknown Source)
> at StreamExecCalc$19.processElement(Unknown Source)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at
> org.apache.flink.table.runtime.operators.source.InputConversionOperator.processElement(InputConversionOperator.java:128)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
> at
> org.apache.flink.streaming.api.functions.source.FromElementsFunction.run(FromElementsFunction.java:231)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:332)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30975) Upgrade aws sdk to v2
[ https://issues.apache.org/jira/browse/FLINK-30975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Samrat Deb updated FLINK-30975: --- Description: [changelog-details|https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md] > Upgrade aws sdk to v2 > - > > Key: FLINK-30975 > URL: https://issues.apache.org/jira/browse/FLINK-30975 > Project: Flink > Issue Type: Improvement >Reporter: Samrat Deb >Priority: Major > > [changelog-details|https://github.com/aws/aws-sdk-java-v2/blob/master/docs/LaunchChangelog.md] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] leeoo commented on pull request #13081: [FLINK-18590][json] Support json array explode to multi messages
leeoo commented on PR #13081: URL: https://github.com/apache/flink/pull/13081#issuecomment-1425296224 Looks like this is a good feature, I also have this kind issue when handling Kafka message data, currently I deserialize json array to multi records using Python UDTF. When can finalize this feature PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25813) TableITCase.testCollectWithClose failed on azure
[
https://issues.apache.org/jira/browse/FLINK-25813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686904#comment-17686904
]
Matthias Pohl commented on FLINK-25813:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45979=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=10822
> TableITCase.testCollectWithClose failed on azure
>
>
> Key: FLINK-25813
> URL: https://issues.apache.org/jira/browse/FLINK-25813
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.15.0, 1.16.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Caizhi Weng
>Priority: Critical
> Labels: pull-request-available, stale-assigned, test-stability
>
> {code:java}
> 2022-01-25T08:35:25.3735884Z Jan 25 08:35:25 [ERROR]
> TableITCase.testCollectWithClose Time elapsed: 0.377 s <<< FAILURE!
> 2022-01-25T08:35:25.3737127Z Jan 25 08:35:25 java.lang.AssertionError: Values
> should be different. Actual: RUNNING
> 2022-01-25T08:35:25.3738167Z Jan 25 08:35:25 at
> org.junit.Assert.fail(Assert.java:89)
> 2022-01-25T08:35:25.3739085Z Jan 25 08:35:25 at
> org.junit.Assert.failEquals(Assert.java:187)
> 2022-01-25T08:35:25.3739922Z Jan 25 08:35:25 at
> org.junit.Assert.assertNotEquals(Assert.java:163)
> 2022-01-25T08:35:25.3740846Z Jan 25 08:35:25 at
> org.junit.Assert.assertNotEquals(Assert.java:177)
> 2022-01-25T08:35:25.3742302Z Jan 25 08:35:25 at
> org.apache.flink.table.api.TableITCase.testCollectWithClose(TableITCase.scala:135)
> 2022-01-25T08:35:25.3743327Z Jan 25 08:35:25 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-01-25T08:35:25.3744343Z Jan 25 08:35:25 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-01-25T08:35:25.3745575Z Jan 25 08:35:25 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-01-25T08:35:25.3746840Z Jan 25 08:35:25 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-01-25T08:35:25.3747922Z Jan 25 08:35:25 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-01-25T08:35:25.3749151Z Jan 25 08:35:25 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-01-25T08:35:25.3750422Z Jan 25 08:35:25 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-01-25T08:35:25.3751820Z Jan 25 08:35:25 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-01-25T08:35:25.3753196Z Jan 25 08:35:25 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-01-25T08:35:25.3754253Z Jan 25 08:35:25 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-01-25T08:35:25.3755441Z Jan 25 08:35:25 at
> org.junit.rules.ExpectedException$ExpectedExceptionStatement.evaluate(ExpectedException.java:258)
> 2022-01-25T08:35:25.3756656Z Jan 25 08:35:25 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-01-25T08:35:25.3757778Z Jan 25 08:35:25 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-01-25T08:35:25.3758821Z Jan 25 08:35:25 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-01-25T08:35:25.3759840Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-01-25T08:35:25.3760919Z Jan 25 08:35:25 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-01-25T08:35:25.3762249Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-01-25T08:35:25.3763322Z Jan 25 08:35:25 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-01-25T08:35:25.3764436Z Jan 25 08:35:25 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-01-25T08:35:25.3765907Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-01-25T08:35:25.3766957Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-01-25T08:35:25.3768104Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-01-25T08:35:25.3769128Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-01-25T08:35:25.3770125Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-01-25T08:35:25.3771118Z Jan 25 08:35:25 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:413)
> 2022-01-25T08:35:25.3772264Z Jan 25 08:35:25 at
>
[jira] [Commented] (FLINK-30863) Do not delete the local changelog file of aborted checkpoint
[ https://issues.apache.org/jira/browse/FLINK-30863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686903#comment-17686903 ] Yanfei Lei commented on FLINK-30863: [~roman] Thanks for your reply. # Yes, this issue might make local recovery fail after checkpoint abortion, and then the job would recovery from remote DFS. This issue doesn't cause data loss. # In case of many subsequent aborted checkpoints, all aborted local state will not be deleted until the next completed checkpoint. Right, this is a degradation in some case. As [~xiarui] [suggested|https://github.com/apache/flink/pull/21822#issuecomment-1418605498] in PR, I'm going to use reference counting to decide when to delete a file. > Do not delete the local changelog file of aborted checkpoint > > > Key: FLINK-30863 > URL: https://issues.apache.org/jira/browse/FLINK-30863 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing, Runtime / State Backends >Affects Versions: 1.17.0 >Reporter: Yanfei Lei >Assignee: Yanfei Lei >Priority: Major > Labels: pull-request-available > > Do not delete the local changelog file of aborted checkpoint, because this > checkpoint may contain the files of the previous checkpoint's file which > would be used by local recovery. The local files of the aborted checkpoint > would be deleted at next checkpoint completed or deleted when deleting the > entire allocation folder when exiting the TM process. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-18356) flink-table-planner Exit code 137 returned from process
[
https://issues.apache.org/jira/browse/FLINK-18356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686902#comment-17686902
]
Matthias Pohl commented on FLINK-18356:
---
2 independent 137 exit code failures (one in an Azure pipeline and one on
Alibaba005) making multiple jobs crash:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45978=results
> flink-table-planner Exit code 137 returned from process
> ---
>
> Key: FLINK-18356
> URL: https://issues.apache.org/jira/browse/FLINK-18356
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Tests
>Affects Versions: 1.12.0, 1.13.0, 1.14.0, 1.15.0, 1.16.0, 1.17.0
>Reporter: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available, test-stability
> Attachments: 1234.jpg, app-profiling_4.gif,
> image-2023-01-11-22-21-57-784.png, image-2023-01-11-22-22-32-124.png
>
>
> {noformat}
> = test session starts
> ==
> platform linux -- Python 3.7.3, pytest-5.4.3, py-1.8.2, pluggy-0.13.1
> cachedir: .tox/py37-cython/.pytest_cache
> rootdir: /__w/3/s/flink-python
> collected 568 items
> pyflink/common/tests/test_configuration.py ..[
> 1%]
> pyflink/common/tests/test_execution_config.py ...[
> 5%]
> pyflink/dataset/tests/test_execution_environment.py .
> ##[error]Exit code 137 returned from process: file name '/bin/docker',
> arguments 'exec -i -u 1002
> 97fc4e22522d2ced1f4d23096b8929045d083dd0a99a4233a8b20d0489e9bddb
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> Finishing: Test - python
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3729=logs=9cada3cb-c1d3-5621-16da-0f718fb86602=8d78fe4f-d658-5c70-12f8-4921589024c3
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30793) PyFlink YARN per-job on Docker test fails on Azure due to permission issues
[
https://issues.apache.org/jira/browse/FLINK-30793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686901#comment-17686901
]
Matthias Pohl commented on FLINK-30793:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45970=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=11207
> PyFlink YARN per-job on Docker test fails on Azure due to permission issues
> ---
>
> Key: FLINK-30793
> URL: https://issues.apache.org/jira/browse/FLINK-30793
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hadoop Compatibility, Deployment / YARN
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> The following build failed due to some hdfs/yarn permission issues in
> PyFlink YARN per-job on Docker e2e test:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45202=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=10587
> {code}
> [...]
> Jan 26 02:17:31 23/01/26 02:12:20 FATAL hs.JobHistoryServer: Error starting
> JobHistoryServer
> Jan 26 02:17:31 org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Error
> creating done directory:
> [hdfs://master.docker-hadoop-cluster-network:9000/tmp/hadoop-yarn/staging/history/done]
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.tryCreatingHistoryDirs(HistoryFileManager.java:698)
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.createHistoryDirs(HistoryFileManager.java:634)
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.HistoryFileManager.serviceInit(HistoryFileManager.java:595)
> Jan 26 02:17:31 at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.JobHistory.serviceInit(JobHistory.java:96)
> Jan 26 02:17:31 at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Jan 26 02:17:31 at
> org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:108)
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.serviceInit(JobHistoryServer.java:152)
> Jan 26 02:17:31 at
> org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.launchJobHistoryServer(JobHistoryServer.java:228)
> Jan 26 02:17:31 at
> org.apache.hadoop.mapreduce.v2.hs.JobHistoryServer.main(JobHistoryServer.java:238)
> Jan 26 02:17:31 Caused by: org.apache.hadoop.security.AccessControlException:
> Permission denied: user=mapred, access=WRITE, inode="/":hdfs:hadoop:drwxr-xr-x
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.check(FSPermissionChecker.java:350)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:251)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSPermissionChecker.checkPermission(FSPermissionChecker.java:189)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1756)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkPermission(FSDirectory.java:1740)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSDirectory.checkAncestorAccess(FSDirectory.java:1699)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSDirMkdirOp.mkdirs(FSDirMkdirOp.java:60)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.mkdirs(FSNamesystem.java:3007)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.mkdirs(NameNodeRpcServer.java:1141)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.mkdirs(ClientNamenodeProtocolServerSideTranslatorPB.java:659)
> Jan 26 02:17:31 at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30924) Conversion issues between timestamp and bingint
[
https://issues.apache.org/jira/browse/FLINK-30924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686899#comment-17686899
]
Leonard Xu edited comment on FLINK-30924 at 2/10/23 7:03 AM:
-
{code:java}
Flink SQL> SET 'table.local-time-zone' = 'Asia/Shanghai';
Flink SQL> select TO_TIMESTAMP(FROM_UNIXTIME(0));
// result
1970-01-01 08:00:00.000
{code}
Hi, [~hackergin]
The result is also reasonable and correct from my understanding. Time is a
complex topic, and I'll try to explain some part of it and make your case
simpler to understand.
When we use a long value with BIGINT(or numeric SQL type) to represent a time,
it usually comes from the upstream system's unix time (e.g.
System.currentMills() in java, unix time in Linux OS). We may have some
question about the long value.
*Q1: What's the meaning of the long value?*
* It means the *past period* since an fixed point in the absolute timeline,
the fixed point people called it epoch,
the wall-clock is 1970-01-01 00:00:00 in region whose time zone is UTC+0 when
epoch point arrived，
the wall-clock is 1970-01-01 01:00:00 in region whose time zone is UTC+1 when
epoch point arrived, and so on.
*Q2:Thus, a epoch 4 seconds means the time past 4 seconds since the epoch
point, what the wall-clock is in UTC+0 region and UTC+1 region at this time
point？*
I guess you could give the right wall-clocks:
1970-01-01 00:00:04 in UTC+0 regions,
1970-01-01 01:00:04 in UTC+1 regions.
*Q3: Is the epoch seconds related to timezone?*
No, it means the past time period , the past time period is same and equal in
any region，it usually represented in a long value in various systems as well as
Flink SQL.
*Q4:How we interpret the epoch seconds to a human readable timestamp expression
like a wall-clock ?*
Yes, we need timezone here.
We interpret epoch 4 seconds to wall-clock 1970-01-01 01:00:04 in these regions
with UTC+1 timezone,
We interpret epoch 4 seconds to wall-clock 1970-01-01 08:00:04 in these regions
with UTC+8 timezone.
Here you should understand why the conversion between timestamp and bingint in
Flink SQL is correct.
I didn't explain the SQL type TIMESTAMP and TIMESTAMP_LTZ, nor the concept
Instant, time zone. If you want to dig more about the type and concept , I can
give some conclusion is :
* `long epoch second` matches `Instant concept` matches SQL type
`TIMESTAMP_LTZ`
* `timestamp string` matches `wall-clock concept` matches SQL type `TIMESTAMP`
was (Author: leonard xu):
{code:java}
Flink SQL> SET 'table.local-time-zone' = 'Asia/Shanghai';
Flink SQL> select TO_TIMESTAMP(FROM_UNIXTIME(0));
// result
1970-01-01 08:00:00.000
{code}
[~hackergin] The result is also reasonable and correct from my understanding.
Time is an complex topic, and I'll try to explain some some part of it and make
your case simpler to understand.
When we use a long value with BIGINT(or numeric SQL type) to represent a time,
it usually comes from the upstream system's unix time (e.g.
System.currentMills() in java, unix time in Linux OS). We may have some
question about the long value.
Q1: What's the meaning of the long value?
It means the *past period* since an fixed point in the absolute timeline,
the fixed point people called it epoch,
the wall-clock is 1970-01-01 00:00:00 in region whose time zone is UTC+0 when
epoch point arrived，
the wall-clock is 1970-01-01 01:00:00 in region whose time zone is UTC+1 when
epoch point arrived, and so on.
Q2:Thus, a epoch 4 seconds means the time past 4 seconds since the epoch point,
what the wall-clock is in UTC+0 region and UTC+1 region at this time point？I
guess you could give the right wall-clocks:
1970-01-01 00:00:04 in UTC+0 regions,
1970-01-01 01:00:04 in UTC+1 regions.
Q3: Is the epoch seconds related to timezone?
No, it means the past time period , the past time period is same and equal in
any region，it usually represented in a long value in various systems as well as
Flink SQL.
Q4:How we interpret the epoch seconds to a human readable timestamp expression
like a wall-clock ?
Yes, we need timezone here.
We interpret epoch 4 seconds to wall-clock 1970-01-01 01:00:04 in these regions
with UTC+1 timezone,
We interpret epoch 4 seconds to wall-clock 1970-01-01 08:00:04 in these regions
with UTC+8 timezone.
Here you should understand why the conversion between timestamp and bingint in
Flink SQL is correct.
I didn't explain the SQL type TIMESTAMP and TIMESTAMP_LTZ, nor the concept
Instant, time zone. If you want to dig more about the type and concept , I can
give some conclusion is :
* `long epoch second` matches `Instant concept` matches SQL type `TIMESTAMP_LTZ`
* `timestamp string` matches `wall-clock concept` matches SQL type `TIMESTAMP`
> Conversion issues between timestamp and bingint
> ---
>
> Key: FLINK-30924
>
[jira] [Commented] (FLINK-30629) ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat is unstable
[
https://issues.apache.org/jira/browse/FLINK-30629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686900#comment-17686900
]
Matthias Pohl commented on FLINK-30629:
---
I'm posting it here as well because [~renqs] already opened this reopened this
one:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45934=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=9716
{code}
Feb 09 07:53:07 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time
elapsed: 9.939 s <<< FAILURE! - in org.apache.flink.client.ClientHeartbeatTest
Feb 09 07:53:07 [ERROR]
org.apache.flink.client.ClientHeartbeatTest.testJobCancelledIfClientHeartbeatTimeout
Time elapsed: 2.708 s <<< FAILURE!
Feb 09 07:53:07 java.lang.AssertionError:
Feb 09 07:53:07
Feb 09 07:53:07 Expecting actual throwable to be an instance of:
Feb 09 07:53:07 java.util.concurrent.ExecutionException
Feb 09 07:53:07 but was:
Feb 09 07:53:07 java.util.concurrent.TimeoutException
Feb 09 07:53:07 at
java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1784)
Feb 09 07:53:07 at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
Feb 09 07:53:07 at
org.assertj.core.internal.Futures.assertFailedWithin(Futures.java:118)
Feb 09 07:53:07 ...(61 remaining lines not displayed - this can be
changed with Assertions.setMaxStackTraceElementsDisplayed)
Feb 09 07:53:07 at
org.apache.flink.client.ClientHeartbeatTest.testJobCancelledIfClientHeartbeatTimeout(ClientHeartbeatTest.java:64)
[...]
{code}
> ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat is unstable
> -
>
> Key: FLINK-30629
> URL: https://issues.apache.org/jira/browse/FLINK-30629
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission
>Affects Versions: 1.17.0
>Reporter: Xintong Song
>Assignee: Liu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
> Attachments: ClientHeartbeatTestLog.txt
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44690=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef=10819
> {code:java}
> Jan 11 04:32:39 [ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 21.02 s <<< FAILURE! - in
> org.apache.flink.client.ClientHeartbeatTest
> Jan 11 04:32:39 [ERROR]
> org.apache.flink.client.ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat
> Time elapsed: 9.157 s <<< ERROR!
> Jan 11 04:32:39 java.lang.IllegalStateException: MiniCluster is not yet
> running or has already been shut down.
> Jan 11 04:32:39 at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:193)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniCluster.getDispatcherGatewayFuture(MiniCluster.java:1044)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniCluster.runDispatcherCommand(MiniCluster.java:917)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniCluster.getJobStatus(MiniCluster.java:841)
> Jan 11 04:32:39 at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.getJobStatus(MiniClusterJobClient.java:91)
> Jan 11 04:32:39 at
> org.apache.flink.client.ClientHeartbeatTest.testJobRunningIfClientReportHeartbeat(ClientHeartbeatTest.java:79)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31007) The code generated by the IF function throws NullPointerException
tivanli created FLINK-31007:
---
Summary: The code generated by the IF function throws
NullPointerException
Key: FLINK-31007
URL: https://issues.apache.org/jira/browse/FLINK-31007
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.15.3, 1.15.2
Environment: {code:java}
// code placeholder
final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
env.setParallelism(1);
final StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env);
final DataStream tab =
env.fromCollection(Arrays.asList(
new Tuple2<>(1L, "a_b_c"),
new Tuple2<>(-1L, "a_b_c")));
final Table tableA = tableEnv.fromDataStream(tab);
tableEnv.executeSql("SELECT if(f0 = -1, '', split_index(f1, '_', 0)) as id FROM
" + tableA)
.print(); {code}
Reporter: tivanli
Caused by: java.lang.NullPointerException
at StreamExecCalc$19.processElement_split1(Unknown Source)
at StreamExecCalc$19.processElement(Unknown Source)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at
org.apache.flink.table.runtime.operators.source.InputConversionOperator.processElement(InputConversionOperator.java:128)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:82)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:57)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:29)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:56)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:29)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:418)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:513)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$SwitchingOnClose.collect(StreamSourceContexts.java:103)
at
org.apache.flink.streaming.api.functions.source.FromElementsFunction.run(FromElementsFunction.java:231)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:67)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:332)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30924) Conversion issues between timestamp and bingint
[
https://issues.apache.org/jira/browse/FLINK-30924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686899#comment-17686899
]
Leonard Xu commented on FLINK-30924:
{code:java}
Flink SQL> SET 'table.local-time-zone' = 'Asia/Shanghai';
Flink SQL> select TO_TIMESTAMP(FROM_UNIXTIME(0));
// result
1970-01-01 08:00:00.000
{code}
[~hackergin] The result is also reasonable and correct from my understanding.
Time is an complex topic, and I'll try to explain some some part of it and make
your case simpler to understand.
When we use a long value with BIGINT(or numeric SQL type) to represent a time,
it usually comes from the upstream system's unix time (e.g.
System.currentMills() in java, unix time in Linux OS). We may have some
question about the long value.
Q1: What's the meaning of the long value?
It means the *past period* since an fixed point in the absolute timeline,
the fixed point people called it epoch,
the wall-clock is 1970-01-01 00:00:00 in region whose time zone is UTC+0 when
epoch point arrived，
the wall-clock is 1970-01-01 01:00:00 in region whose time zone is UTC+1 when
epoch point arrived, and so on.
Q2:Thus, a epoch 4 seconds means the time past 4 seconds since the epoch point,
what the wall-clock is in UTC+0 region and UTC+1 region at this time point？I
guess you could give the right wall-clocks:
1970-01-01 00:00:04 in UTC+0 regions,
1970-01-01 01:00:04 in UTC+1 regions.
Q3: Is the epoch seconds related to timezone?
No, it means the past time period , the past time period is same and equal in
any region，it usually represented in a long value in various systems as well as
Flink SQL.
Q4:How we interpret the epoch seconds to a human readable timestamp expression
like a wall-clock ?
Yes, we need timezone here.
We interpret epoch 4 seconds to wall-clock 1970-01-01 01:00:04 in these regions
with UTC+1 timezone,
We interpret epoch 4 seconds to wall-clock 1970-01-01 08:00:04 in these regions
with UTC+8 timezone.
Here you should understand why the conversion between timestamp and bingint in
Flink SQL is correct.
I didn't explain the SQL type TIMESTAMP and TIMESTAMP_LTZ, nor the concept
Instant, time zone. If you want to dig more about the type and concept , I can
give some conclusion is :
* `long epoch second` matches `Instant concept` matches SQL type `TIMESTAMP_LTZ`
* `timestamp string` matches `wall-clock concept` matches SQL type `TIMESTAMP`
> Conversion issues between timestamp and bingint
> ---
>
> Key: FLINK-30924
> URL: https://issues.apache.org/jira/browse/FLINK-30924
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.16.1
>Reporter: Feng Jin
>Priority: Major
>
> When casting to convert timestamp and bigint, the following exception is
> thrown:
> {code:java}
> //代码占位符
> org.apache.flink.table.api.ValidationException: The cast from NUMERIC type to
> TIMESTAMP type is not allowed. It's recommended to use
> TO_TIMESTAMP(FROM_UNIXTIME(numeric_col)) instead, note the numeric is in
> seconds.
> {code}
> However, the FROM_UNIXTIME function will use the local time zone for
> conversion, but the TO_TIMESTAMP function will not use the local time zone
> but will use the UTC time zone conversion, so that the actual result in the
> wrong result.
>
> The following is an example of the results of the test
> {code:java}
> //代码占位符
> Flink SQL> SET 'table.local-time-zone' = 'Asia/Shanghai';
> Flink SQL> select TO_TIMESTAMP(FROM_UNIXTIME(0));
> // result
> EXPR$0
> 1970-01-01 08:00:00.000
> {code}
>
>
> UNIX_TIMESTAMP(CAST(timestamp_col AS STRING)) has the same problem.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-30871) add bloom filter for orc
[ https://issues.apache.org/jira/browse/FLINK-30871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30871. Assignee: Jun Zhang Resolution: Fixed master: bd8b6c3d9f71f0fdc3b86df860be8634d62c55cc > add bloom filter for orc > - > > Key: FLINK-30871 > URL: https://issues.apache.org/jira/browse/FLINK-30871 > Project: Flink > Issue Type: Improvement > Components: Table Store >Reporter: Jun Zhang >Assignee: Jun Zhang >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30972) E2e tests always fail in phase "Prepare E2E run"
[
https://issues.apache.org/jira/browse/FLINK-30972?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686895#comment-17686895
]
Matthias Pohl commented on FLINK-30972:
---
The following builds didn't contain the fixes mentioned above, yet:
*
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45934=results
*
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45919=results
*
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45922=results
*
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=45979=results
> E2e tests always fail in phase "Prepare E2E run"
>
>
> Key: FLINK-30972
> URL: https://issues.apache.org/jira/browse/FLINK-30972
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI, Tests
>Affects Versions: 1.17.0, 1.15.4, 1.16.2, 1.18.0
>Reporter: Lijie Wang
>Assignee: Qingsheng Ren
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.17.0, 1.15.4, 1.16.2, 1.18.0
>
>
> {code:java}
> Installing required software
> Reading package lists...
> Building dependency tree...
> Reading state information...
> bc is already the newest version (1.07.1-2build1).
> bc set to manually installed.
> libapr1 is already the newest version (1.6.5-1ubuntu1).
> libapr1 set to manually installed.
> 0 upgraded, 0 newly installed, 0 to remove and 13 not upgraded.
> --2023-02-09 04:38:47--
> http://security.ubuntu.com/ubuntu/pool/main/o/openssl1.0/libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb
> Resolving security.ubuntu.com (security.ubuntu.com)... 91.189.91.39,
> 185.125.190.36, 185.125.190.39, ...
> Connecting to security.ubuntu.com (security.ubuntu.com)|91.189.91.39|:80...
> connected.
> HTTP request sent, awaiting response... 404 Not Found
> 2023-02-09 04:38:47 ERROR 404: Not Found.
> WARNING: apt does not have a stable CLI interface. Use with caution in
> scripts.
> Reading package lists...
> E: Unsupported file ./libssl1.0.0_1.0.2n-1ubuntu5.10_amd64.deb given on
> commandline
> ##[error]Bash exited with code '100'.
> Finishing: Prepare E2E run
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #494: [FLINK-30871] add bloom filter for orc
JingsongLi merged PR #494: URL: https://github.com/apache/flink-table-store/pull/494 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] boring-cyborg[bot] commented on pull request #5: [FLINK-25916][connector-kafka] Using upsert-kafka with a flush buffer…
boring-cyborg[bot] commented on PR #5: URL: https://github.com/apache/flink-connector-kafka/pull/5#issuecomment-1425273961 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-kafka] paul8263 opened a new pull request, #5: [FLINK-25916][connector-kafka] Using upsert-kafka with a flush buffer…
paul8263 opened a new pull request, #5: URL: https://github.com/apache/flink-connector-kafka/pull/5 … results in Null Pointer Exception This PR is the migration from [FLINK-25916](https://github.com/apache/flink/pull/20343) # What is the purpose of the change Allow flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/ReducingUpsertWriter.WrappedContext to set timestamp with NULL value in order to avoid NPE. # Brief change log - flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/streaming/connectors/kafka/table/ReducingUpsertWriter.java - flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/ReducingUpsertWriterTest.java # Verifying this change This change is already covered by existing tests, such as flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/streaming/connectors/kafka/table/ReducingUpsertWriterTest::testWriteDataWithNullTimestamp(). # Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with @Public(Evolving): no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no # Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on pull request #471: [FLINK-30608] support rename table
JingsongLi commented on PR #471: URL: https://github.com/apache/flink-table-store/pull/471#issuecomment-1425273350 > It seems that spark tests have some leak tables, I will fix this. fixed in e4f0feb08845a41f477938c08830cd17daa29a68 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] liming30 commented on a diff in pull request #21503: [FLINK-30251] Move the IO with DFS during abort checkpoint to an asynchronous thread pool.
liming30 commented on code in PR #21503:
URL: https://github.com/apache/flink/pull/21503#discussion_r1102335317
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SubtaskCheckpointCoordinatorImpl.java:
##
@@ -177,6 +182,14 @@ class SubtaskCheckpointCoordinatorImpl implements
SubtaskCheckpointCoordinator {
this.checkpoints = new HashMap<>();
this.lock = new Object();
this.asyncOperationsThreadPool =
checkNotNull(asyncOperationsThreadPool);
+this.asyncDisposeThreadPool =
+new ThreadPoolExecutor(
+0,
+4,
+60L,
+TimeUnit.SECONDS,
+new LinkedBlockingQueue<>(),
+new ExecutorThreadFactory("AsyncDispose"));
Review Comment:
In @gaoyunhaii's comment in
[FLINK-30251](https://issues.apache.org/jira/browse/FLINK-30251):
> There is also one concern:
>
> 1. The current `asyncOperationsThreadPool` is a cached thread pool, which
do not have an upper limit of the number of threads, and it will create a new
thread whenever there is not free thread when submitting tasks. Then if we have
a large number of file to close, we might end up with a lot of threads, which
might further cause a large number of memory consumption (1MB for each thread
RSS region).
>
> 2. Thus we might change it to a thread pool with a limited maximum number
of thread and one unbounded Blocking Queue. Also since the thread in this pool
might be blocked, we might need to use a separate thread pool.
The main concern is that too many threads will be generated when
asynchronous cleanup may block. From your point of view, which is better to
reuse `asyncOperationsThreadPool` or create a new pool?
##
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/SubtaskCheckpointCoordinatorImpl.java:
##
@@ -529,15 +542,19 @@ public void close() throws IOException {
}
public void cancel() throws IOException {
-List asyncCheckpointRunnables = null;
synchronized (lock) {
if (!closed) {
closed = true;
-asyncCheckpointRunnables = new
ArrayList<>(checkpoints.values());
+final List asyncCheckpointRunnables =
+new ArrayList<>(checkpoints.values());
checkpoints.clear();
+if (!asyncCheckpointRunnables.isEmpty()) {
+asyncDisposeThreadPool.execute(
+() ->
IOUtils.closeAllQuietly(asyncCheckpointRunnables));
+}
+asyncDisposeThreadPool.shutdown();
Review Comment:
Yes, you are right, I originally just hoped that all async operations should
not block the process, but from the perspective of preventing resource leaks,
cancel should not be asynchronous. I'll change this part of the logic back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] zhangjun0x01 commented on pull request #494: [FLINK-30871] add bloom filter for orc
zhangjun0x01 commented on PR #494: URL: https://github.com/apache/flink-table-store/pull/494#issuecomment-1425263065 > UTCase failed I update it -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-30508) CliClientITCase.testSqlStatements failed with output not matched with expected
[ https://issues.apache.org/jira/browse/FLINK-30508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686893#comment-17686893 ] Matthias Pohl commented on FLINK-30508: --- Then, the Jira issue management is a bit confusing here: Either, we open this one again to do the 1.16 backport here or (the one I would suggest doing) open FLINK-30640 again, add 1.16.1 as an affected version and provide the backport for it > CliClientITCase.testSqlStatements failed with output not matched with expected > -- > > Key: FLINK-30508 > URL: https://issues.apache.org/jira/browse/FLINK-30508 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.16.0, 1.17.0 >Reporter: Qingsheng Ren >Assignee: Shengkai Fang >Priority: Critical > Labels: test-stability > Fix For: 1.17.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44246=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14992 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zoltar9264 commented on a diff in pull request #21812: [FLINK-28440][state/changelog] record reference count of StreamStateHandle in TaskChangelogRegistry
zoltar9264 commented on code in PR #21812:
URL: https://github.com/apache/flink/pull/21812#discussion_r1102327361
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/TaskChangelogRegistry.java:
##
@@ -46,29 +46,29 @@
@Internal
public interface TaskChangelogRegistry {
-/** Start tracking the state uploaded for the given backends. */
-void startTracking(StreamStateHandle handle, Set backendIDs);
+/** Start tracking the state uploaded. */
+void startTracking(StreamStateHandle handle, long refCount);
Review Comment:
Thanks for your suggestion @rkhachatryan , I will do that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-30792) clean up not uploaded state changes after materialization complete
[ https://issues.apache.org/jira/browse/FLINK-30792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686891#comment-17686891 ] Feifan Wang commented on FLINK-30792: - Thanks for your reply [~roman] , you are right, ref counting state changes per state handle can indeed solve the problem of changelog file not found mentioned above. The changes in this PR are only intended to reduce useless data uploads. As for the performance regression problem you mentioned, I really didn't think about it carefully before. Now I'm also not sure if this is causing a performance regression. On the one hand, this will indeed reduce the amount of data uploaded; on the other hand, it will indeed require more data to be uploaded when the checkpoint is triggered. > clean up not uploaded state changes after materialization complete > -- > > Key: FLINK-30792 > URL: https://issues.apache.org/jira/browse/FLINK-30792 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.16.0 >Reporter: Feifan Wang >Priority: Major > Labels: pull-request-available > Attachments: image-2023-02-03-11-30-40-198.png > > > We should clean up not uploaded state changes after materialization > completed, otherwise it may cause FileNotFoundException. > Since state changes before completed materialization in > FsStateChangelogWriter#notUploaded will not be used in any subsequent > checkpoint, I suggest clean up it while handle materialization result. > How do you think about this ? [~ym] , [~roman] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] FangYongs commented on pull request #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
FangYongs commented on PR #515: URL: https://github.com/apache/flink-table-store/pull/515#issuecomment-1425256487 Thanks @SteNicholas DONE -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] paul8263 commented on pull request #20343: [FLINK-25916][connector-kafka] Using upsert-kafka with a flush buffer…
paul8263 commented on PR #20343: URL: https://github.com/apache/flink/pull/20343#issuecomment-1425254629 Hi @jaumebecks, Thanks for reviewing the PR. I'd like to do the migration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31004) Introduce data input and output stream for table store
[ https://issues.apache.org/jira/browse/FLINK-31004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31004: --- Labels: pull-request-available (was: ) > Introduce data input and output stream for table store > -- > > Key: FLINK-31004 > URL: https://issues.apache.org/jira/browse/FLINK-31004 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Shammon >Priority: Major > Labels: pull-request-available > > Introduce data input/output stream for table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] FangYongs opened a new pull request, #516: [FLINK-31004] Introduce input and output stream for table store
FangYongs opened a new pull request, #516: URL: https://github.com/apache/flink-table-store/pull/516 Introduce input and output stream from `flink-core` for table store -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31006) job is not finished when using pipeline mode to run bounded source like kafka/pulsar
[ https://issues.apache.org/jira/browse/FLINK-31006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Qingsheng Ren reassigned FLINK-31006: - Assignee: jackylau > job is not finished when using pipeline mode to run bounded source like > kafka/pulsar > > > Key: FLINK-31006 > URL: https://issues.apache.org/jira/browse/FLINK-31006 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: jackylau >Assignee: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > Attachments: image-2023-02-10-13-20-52-890.png, > image-2023-02-10-13-23-38-430.png, image-2023-02-10-13-24-46-929.png > > > when i do failover works like kill jm/tm when using pipeline mode to run > bounded source like kafka, i found job is not finished, when every partition > data has consumed. > > After dig into code, i found this logical not run when JM recover. the > partition infos are not changed. so noMoreNewPartitionSplits is not set to > true. then this will not run > > !image-2023-02-10-13-23-38-430.png! > > !image-2023-02-10-13-24-46-929.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30654) Add option to force consumption from StartCursor every time the application starts
[ https://issues.apache.org/jira/browse/FLINK-30654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30654. - Fix Version/s: pulsar-4.0.0 Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/23 > Add option to force consumption from StartCursor every time the application > starts > -- > > Key: FLINK-30654 > URL: https://issues.apache.org/jira/browse/FLINK-30654 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: likang >Assignee: Yufan Sheng >Priority: Minor > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > Attachments: WechatIMG250.png > > > Pulsar does not take effect when the subscription is set to start position > consumption. It is recommended to add an option to be determined by the user > or to add a lastAck strategy to adapt to the scenario of whether to start > consumption from the last consumption submission position -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-pulsar] tisonkun merged pull request #23: [FLINK-30654][Connector/Pulsar] Add an option to force consumption from StartCursor every time the application starts.
tisonkun merged PR #23: URL: https://github.com/apache/flink-connector-pulsar/pull/23 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-pulsar] tisonkun commented on pull request #23: [FLINK-30654][Connector/Pulsar] Add an option to force consumption from StartCursor every time the application starts.
tisonkun commented on PR #23: URL: https://github.com/apache/flink-connector-pulsar/pull/23#issuecomment-1425228305 No new tests or related changes. Merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21909: [FLINK-31006][connector/kafka] Fix noMoreNewPartitionSplits is not se…
flinkbot commented on PR #21909: URL: https://github.com/apache/flink/pull/21909#issuecomment-1425218840 ## CI report: * 40bb921d8a5f3290c9ddb5afa8d834fbf5b51ae4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31006) job is not finished when using pipeline mode to run bounded source like kafka/pulsar
[ https://issues.apache.org/jira/browse/FLINK-31006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31006: --- Labels: pull-request-available (was: ) > job is not finished when using pipeline mode to run bounded source like > kafka/pulsar > > > Key: FLINK-31006 > URL: https://issues.apache.org/jira/browse/FLINK-31006 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Connectors / Pulsar >Affects Versions: 1.17.0 >Reporter: jackylau >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > Attachments: image-2023-02-10-13-20-52-890.png, > image-2023-02-10-13-23-38-430.png, image-2023-02-10-13-24-46-929.png > > > when i do failover works like kill jm/tm when using pipeline mode to run > bounded source like kafka, i found job is not finished, when every partition > data has consumed. > > After dig into code, i found this logical not run when JM recover. the > partition infos are not changed. so noMoreNewPartitionSplits is not set to > true. then this will not run > > !image-2023-02-10-13-23-38-430.png! > > !image-2023-02-10-13-24-46-929.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] liuyongvs opened a new pull request, #21909: [FLINK-31006][connector/kafka] Fix noMoreNewPartitionSplits is not se…
liuyongvs opened a new pull request, #21909: URL: https://github.com/apache/flink/pull/21909 …t correct when using pipeline mode runs bounded source. ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
SteNicholas commented on code in PR #515:
URL: https://github.com/apache/flink-table-store/pull/515#discussion_r1102290035
##
flink-table-store-test-utils/src/main/java/org/apache/flink/table/store/testutils/junit/parameterized/Parameter.java:
##
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.testutils.junit.parameterized;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * The annotation is used to replace {@code Parameterized.Parameter} for Junit
5 parameterized
+ * tests.
+ */
+@Target(ElementType.FIELD)
+@Retention(RetentionPolicy.RUNTIME)
+public @interface Parameter {
+int value() default 0;
Review Comment:
Adds the java doc of this method:
```
/**
* Method that returns the index of the parameter in the array
* returned by the method annotated by Parameters.
* Index range must start at 0.
* Default value is 0.
*
* @return the index of the parameter.
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-30654) Add option to force consumption from StartCursor every time the application starts
[ https://issues.apache.org/jira/browse/FLINK-30654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30654: --- Labels: pull-request-available (was: ) > Add option to force consumption from StartCursor every time the application > starts > -- > > Key: FLINK-30654 > URL: https://issues.apache.org/jira/browse/FLINK-30654 > Project: Flink > Issue Type: Improvement > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: likang >Assignee: Yufan Sheng >Priority: Minor > Labels: pull-request-available > Attachments: WechatIMG250.png > > > Pulsar does not take effect when the subscription is set to start position > consumption. It is recommended to add an option to be determined by the user > or to add a lastAck strategy to adapt to the scenario of whether to start > consumption from the last consumption submission position -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
SteNicholas commented on code in PR #515:
URL: https://github.com/apache/flink-table-store/pull/515#discussion_r1102289414
##
flink-table-store-test-utils/src/main/java/org/apache/flink/table/store/testutils/junit/parameterized/Parameters.java:
##
@@ -0,0 +1,34 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.testutils.junit.parameterized;
+
+import java.lang.annotation.ElementType;
+import java.lang.annotation.Retention;
+import java.lang.annotation.RetentionPolicy;
+import java.lang.annotation.Target;
+
+/**
+ * The annotation is used to replace {@code Parameterized.Parameters} (Junit4)
for Junit 5
+ * parameterized tests.
+ */
+@Retention(RetentionPolicy.RUNTIME)
+@Target(ElementType.METHOD)
+public @interface Parameters {
+String name() default "{index}";
Review Comment:
Adds the java doc of this method:
```
/**
* Optional pattern to derive the test's name from the parameters. Use
* numbers in braces to refer to the parameters or the additional data
* as follows:
*
* {index} - the current parameter index
* {0} - the first parameter value
* {1} - the second parameter value
* etc...
*
*
* Default value is "{index}" for compatibility with previous JUnit
* versions.
*
* @return {@link MessageFormat} pattern string, except the index
* placeholder.
* @see MessageFormat
*/
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-connector-pulsar] syhily opened a new pull request, #23: [FLINK-30654][Connector/Pulsar] Add an option to force consumption from StartCursor every time the application starts.
syhily opened a new pull request, #23: URL: https://github.com/apache/flink-connector-pulsar/pull/23 ## What is the purpose of the change This PR supports to set the `StartCursor` as the default start position for consuming. ## Brief change log - Add a new `PULSAR_RESET_SUBSCRIPTION_CURSOR` option in `PulsarSourceOptions` ## Verifying this change This change is a minor change and don't have any tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs, JavaDocs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
SteNicholas commented on code in PR #515:
URL: https://github.com/apache/flink-table-store/pull/515#discussion_r1102286204
##
flink-table-store-test-utils/src/main/java/org/apache/flink/table/store/testutils/junit/TestLoggerExtension.java:
##
@@ -0,0 +1,83 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.store.testutils.junit;
+
+import org.junit.jupiter.api.extension.BeforeEachCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.TestWatcher;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.io.PrintWriter;
+import java.io.StringWriter;
+
+/** A JUnit-5-style test logger. */
+public class TestLoggerExtension implements TestWatcher, BeforeEachCallback {
+private static final Logger LOG =
LoggerFactory.getLogger(TestLoggerExtension.class);
+
+@Override
+public void beforeEach(ExtensionContext context) {
+LOG.info(
+
"\n"
++ "\nTest {}.{}[{}] is running."
++
"\n",
+context.getRequiredTestClass().getCanonicalName(),
+context.getRequiredTestMethod().getName(),
+context.getDisplayName());
+}
+
+@Override
+public void testSuccessful(ExtensionContext context) {
+LOG.info(
+
"\n"
++ "\nTest {}.{}[{}] successfully run."
++
"\n",
+context.getRequiredTestClass().getCanonicalName(),
+context.getRequiredTestMethod().getName(),
+context.getDisplayName());
+}
+
+@Override
+public void testFailed(ExtensionContext context, Throwable cause) {
+LOG.error(
+
"\n"
++ "\nTest {}.{}[{}] failed with:\n{}"
Review Comment:
```suggestion
+ "\nTest {}.{}[{}] failed with:\n{}."
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-31006) job is not finished when using pipeline mode to run bounded source like kafka/pulsar
jackylau created FLINK-31006: Summary: job is not finished when using pipeline mode to run bounded source like kafka/pulsar Key: FLINK-31006 URL: https://issues.apache.org/jira/browse/FLINK-31006 Project: Flink Issue Type: Bug Components: Connectors / Kafka, Connectors / Pulsar Affects Versions: 1.17.0 Reporter: jackylau Fix For: 1.17.0 Attachments: image-2023-02-10-13-20-52-890.png, image-2023-02-10-13-23-38-430.png, image-2023-02-10-13-24-46-929.png when i do failover works like kill jm/tm when using pipeline mode to run bounded source like kafka, i found job is not finished, when every partition data has consumed. After dig into code, i found this logical not run when JM recover. the partition infos are not changed. so noMoreNewPartitionSplits is not set to true. then this will not run !image-2023-02-10-13-23-38-430.png! !image-2023-02-10-13-24-46-929.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] SteNicholas commented on a diff in pull request #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
SteNicholas commented on code in PR #515: URL: https://github.com/apache/flink-table-store/pull/515#discussion_r1102284387 ## flink-table-store-test-utils/pom.xml: ## @@ -0,0 +1,57 @@ + + +http://maven.apache.org/POM/4.0.0; + xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance; + xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd;> +4.0.0 + + +flink-table-store-parent +org.apache.flink +0.4-SNAPSHOT + + +flink-table-store-test-utils +Flink Table Store: Test utils : Review Comment: ```suggestion Flink Table Store : Test Utils ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30988) Deprecate TableStoreManagedFactory
[ https://issues.apache.org/jira/browse/FLINK-30988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30988: --- Labels: pull-request-available (was: ) > Deprecate TableStoreManagedFactory > -- > > Key: FLINK-30988 > URL: https://issues.apache.org/jira/browse/FLINK-30988 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30988) Deprecate TableStoreManagedFactory
[ https://issues.apache.org/jira/browse/FLINK-30988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30988. Fix Version/s: table-store-0.4.0 Assignee: yuzelin Resolution: Fixed master: 0cc23940b274945c1b69a6f65fdfc4134acab77b > Deprecate TableStoreManagedFactory > -- > > Key: FLINK-30988 > URL: https://issues.apache.org/jira/browse/FLINK-30988 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Assignee: yuzelin >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #514: [FLINK-30988] Deprecate TableStoreManagedFactory
JingsongLi merged PR #514: URL: https://github.com/apache/flink-table-store/pull/514 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31005) Release Testing: Verify FLIP-281 Supports speculative execution of sinks
[
https://issues.apache.org/jira/browse/FLINK-31005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhu Zhu updated FLINK-31005:
Description:
This task aims to verify [FLIP-281 Supports speculative execution of
sinks|https://issues.apache.org/jira/browse/FLINK-30725].
The documentation can be found
[here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#enable-sinks-for-speculative-execution]
.
Things to verify:
1. If a sink implements the decorative interface
{{SupportsConcurrentExecutionAttempts}, Speculative executions can be
performed, otherwise not. Sinks to verify includes SinkFunction, OutputFormat
and Sink(V2).
2. These built-in sinks supports speculative execution: DiscardingSink,
PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat, HiveTableSink
It's important to verify that the final data produced are correct and the
operator chain is as expected, i.e. no unexpected chain-break.
If it's hard to construct a case that speculative execution would happen,
especially for those built-in sinks, the speculative execution configuration
can be tuned to allow it easier to happen, e.g. set
{{slow-task-detector.execution-time.baseline-lower-bound}} and
{{slow-task-detector.execution-time.baseline-ratio}} to {{0}}.
was:
This task aims to verify [FLIP-281 Supports speculative execution of
sinks|https://issues.apache.org/jira/browse/FLINK-30725].
The documentation can be found
[here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#enable-sinks-for-speculative-execution]
.
Things to verify:
1. If a sink implements the decorative interface
{{SupportsConcurrentExecutionAttempts}, Speculative executions can be
performed, otherwise not. Sinks to verify includes SinkFunction, OutputFormat
and Sink(V2).
2. These built-in sinks supports speculative execution: DiscardingSink,
PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat, HiveTableSink
If it's hard to construct a case that speculative execution would happen,
especially for those built-in sinks, the speculative execution configuration
can be tuned to allow it easier to happen, e.g. set
{{slow-task-detector.execution-time.baseline-lower-bound}} and
{{slow-task-detector.execution-time.baseline-ratio}} to {{0}}.
> Release Testing: Verify FLIP-281 Supports speculative execution of sinks
>
>
> Key: FLINK-31005
> URL: https://issues.apache.org/jira/browse/FLINK-31005
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Reporter: Zhu Zhu
>Priority: Blocker
> Fix For: 1.17.0
>
>
> This task aims to verify [FLIP-281 Supports speculative execution of
> sinks|https://issues.apache.org/jira/browse/FLINK-30725].
> The documentation can be found
> [here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#enable-sinks-for-speculative-execution]
> .
> Things to verify:
> 1. If a sink implements the decorative interface
> {{SupportsConcurrentExecutionAttempts}, Speculative executions can be
> performed, otherwise not. Sinks to verify includes SinkFunction, OutputFormat
> and Sink(V2).
> 2. These built-in sinks supports speculative execution: DiscardingSink,
> PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat, HiveTableSink
> It's important to verify that the final data produced are correct and the
> operator chain is as expected, i.e. no unexpected chain-break.
> If it's hard to construct a case that speculative execution would happen,
> especially for those built-in sinks, the speculative execution configuration
> can be tuned to allow it easier to happen, e.g. set
> {{slow-task-detector.execution-time.baseline-lower-bound}} and
> {{slow-task-detector.execution-time.baseline-ratio}} to {{0}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31005) Release Testing: Verify FLIP-281 Supports speculative execution of sinks
Zhu Zhu created FLINK-31005:
---
Summary: Release Testing: Verify FLIP-281 Supports speculative
execution of sinks
Key: FLINK-31005
URL: https://issues.apache.org/jira/browse/FLINK-31005
Project: Flink
Issue Type: Sub-task
Components: Runtime / Coordination
Reporter: Zhu Zhu
Fix For: 1.17.0
This task aims to verify [FLIP-281 Supports speculative execution of
sinks|https://issues.apache.org/jira/browse/FLINK-30725].
The documentation can be found
[here|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/speculative_execution/#enable-sinks-for-speculative-execution]
.
Things to verify:
1. If a sink implements the decorative interface
{{SupportsConcurrentExecutionAttempts}, Speculative executions can be
performed, otherwise not. Sinks to verify includes SinkFunction, OutputFormat
and Sink(V2).
2. These built-in sinks supports speculative execution: DiscardingSink,
PrintSinkFunction, PrintSink, FileSink, FileSystemOutputFormat, HiveTableSink
If it's hard to construct a case that speculative execution would happen,
especially for those built-in sinks, the speculative execution configuration
can be tuned to allow it easier to happen, e.g. set
{{slow-task-detector.execution-time.baseline-lower-bound}} and
{{slow-task-detector.execution-time.baseline-ratio}} to {{0}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31003) Flink SQL IF / CASE WHEN Funcation incorrect
[
https://issues.apache.org/jira/browse/FLINK-31003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686862#comment-17686862
]
weiqinpan commented on FLINK-31003:
---
Only find this in Flink 1.15.0 +, everything is right before 1.15.0.
> Flink SQL IF / CASE WHEN Funcation incorrect
>
>
> Key: FLINK-31003
> URL: https://issues.apache.org/jira/browse/FLINK-31003
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.15.0, 1.15.1, 1.16.0, 1.15.2, 1.15.3, 1.16.1
>Reporter: weiqinpan
>Priority: Major
>
> When I execute the below sql using sql-client，i found something wrong.
>
> {code:java}
> CREATE TEMPORARY TABLE source (
> mktgmsg_biz_type STRING,
> marketing_flow_id STRING,
> mktgmsg_campaign_id STRING
> )
> WITH
> (
> 'connector' = 'filesystem',
> 'path' = 'file:///Users/xxx/Desktop/demo.json',
> 'format' = 'json'
> );
> -- return correct value('marketing_flow_id')
> SELECT IF(`marketing_flow_id` IS NOT NULL, `marketing_flow_id`, '') FROM
> source;
> -- return incorrect value('')
> SELECT IF(`marketing_flow_id` IS NULL, '', `marketing_flow_id`) FROM
> source;{code}
> The demo.json data is
>
> {code:java}
> {"mktgmsg_biz_type": "marketing_flow", "marketing_flow_id":
> "marketing_flow_id", "mktgmsg_campaign_id": "mktgmsg_campaign_id"} {code}
>
>
> BTW, use case when + if / ifnull also have something wrong.
>
> {code:java}
> -- return wrong value(''), expect return marketing_flow_id
> select CASE
> WHEN `mktgmsg_biz_type` = 'marketing_flow' THEN IF(`marketing_flow_id`
> IS NULL, `marketing_flow_id`, '')
> WHEN `mktgmsg_biz_type` = 'mktgmsg_campaign' THEN
> IF(`mktgmsg_campaign_id` IS NULL, '', `mktgmsg_campaign_id`)
> ELSE ''
> END AS `message_campaign_instance_id` FROM source;
> -- return wrong value('')
> select CASE
> WHEN `mktgmsg_biz_type` = 'marketing_flow' THEN
> IFNULL(`marketing_flow_id`, '')
> WHEN `mktgmsg_biz_type` = 'mktgmsg_campaign' THEN
> IFNULL(`mktgmsg_campaign_id`, '')
> ELSE ''
> END AS `message_campaign_instance_id` FROM source;
> -- return correct value, the difference is [else return ' ']
> select CASE
> WHEN `mktgmsg_biz_type` = 'marketing_flow' THEN
> IFNULL(`marketing_flow_id`, '')
> WHEN `mktgmsg_biz_type` = 'mktgmsg_campaign' THEN
> IFNULL(`mktgmsg_campaign_id`, '')
> ELSE ' '
> END AS `message_campaign_instance_id` FROM source;
> {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31004) Introduce data input and output stream for table store
Shammon created FLINK-31004: --- Summary: Introduce data input and output stream for table store Key: FLINK-31004 URL: https://issues.apache.org/jira/browse/FLINK-31004 Project: Flink Issue Type: Sub-task Components: Table Store Affects Versions: table-store-0.4.0 Reporter: Shammon Introduce data input/output stream for table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31003) Flink SQL IF / CASE WHEN Funcation incorrect
weiqinpan created FLINK-31003:
-
Summary: Flink SQL IF / CASE WHEN Funcation incorrect
Key: FLINK-31003
URL: https://issues.apache.org/jira/browse/FLINK-31003
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.16.1, 1.15.3, 1.15.2, 1.16.0, 1.15.1, 1.15.0
Reporter: weiqinpan
When I execute the below sql using sql-client，i found something wrong.
{code:java}
CREATE TEMPORARY TABLE source (
mktgmsg_biz_type STRING,
marketing_flow_id STRING,
mktgmsg_campaign_id STRING
)
WITH
(
'connector' = 'filesystem',
'path' = 'file:///Users/xxx/Desktop/demo.json',
'format' = 'json'
);
-- return correct value('marketing_flow_id')
SELECT IF(`marketing_flow_id` IS NOT NULL, `marketing_flow_id`, '') FROM source;
-- return incorrect value('')
SELECT IF(`marketing_flow_id` IS NULL, '', `marketing_flow_id`) FROM
source;{code}
The demo.json data is
{code:java}
{"mktgmsg_biz_type": "marketing_flow", "marketing_flow_id":
"marketing_flow_id", "mktgmsg_campaign_id": "mktgmsg_campaign_id"} {code}
BTW, use case when + if / ifnull also have something wrong.
{code:java}
-- return wrong value(''), expect return marketing_flow_id
select CASE
WHEN `mktgmsg_biz_type` = 'marketing_flow' THEN IF(`marketing_flow_id` IS
NULL, `marketing_flow_id`, '')
WHEN `mktgmsg_biz_type` = 'mktgmsg_campaign' THEN IF(`mktgmsg_campaign_id`
IS NULL, '', `mktgmsg_campaign_id`)
ELSE ''
END AS `message_campaign_instance_id` FROM source;
-- return wrong value('')
select CASE
WHEN `mktgmsg_biz_type` = 'marketing_flow' THEN
IFNULL(`marketing_flow_id`, '')
WHEN `mktgmsg_biz_type` = 'mktgmsg_campaign' THEN
IFNULL(`mktgmsg_campaign_id`, '')
ELSE ''
END AS `message_campaign_instance_id` FROM source;
-- return correct value, the difference is [else return ' ']
select CASE
WHEN `mktgmsg_biz_type` = 'marketing_flow' THEN
IFNULL(`marketing_flow_id`, '')
WHEN `mktgmsg_biz_type` = 'mktgmsg_campaign' THEN
IFNULL(`mktgmsg_campaign_id`, '')
ELSE ' '
END AS `message_campaign_instance_id` FROM source;
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi commented on pull request #471: [FLINK-30608] support rename table
JingsongLi commented on PR #471: URL: https://github.com/apache/flink-table-store/pull/471#issuecomment-1425135653 It seems that spark tests have some leak tables, I will fix this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30988) Deprecate TableStoreManagedFactory
[ https://issues.apache.org/jira/browse/FLINK-30988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] yuzelin updated FLINK-30988: Summary: Deprecate TableStoreManagedFactory (was: Refactor E2E tests to get rid of managed table) > Deprecate TableStoreManagedFactory > -- > > Key: FLINK-30988 > URL: https://issues.apache.org/jira/browse/FLINK-30988 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: yuzelin >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30990) Performance of deployDownstreamTasks.BATCH has worsened
[ https://issues.apache.org/jira/browse/FLINK-30990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-30990: Labels: (was: pull-request-available) > Performance of deployDownstreamTasks.BATCH has worsened > --- > > Key: FLINK-30990 > URL: https://issues.apache.org/jira/browse/FLINK-30990 > Project: Flink > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Martijn Visser >Priority: Blocker > > See > http://codespeed.dak8s.net:8000/timeline/#/?exe=8=deployDownstreamTasks.BATCH=on=on=off=2=200 > It appears that the trend has now stabilized upwards, from 60-70 ms/op to > 80-90 ms/op, while less is better. > Not sure if this has meaningful impact in a production setup, but I would > like to get this verified. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zoltar9264 commented on a diff in pull request #21679: [FLINK-30561][state/changelog] fix changelog local cache file not found
zoltar9264 commented on code in PR #21679:
URL: https://github.com/apache/flink/pull/21679#discussion_r1102239105
##
flink-dstl/flink-dstl-dfs/src/test/java/org/apache/flink/changelog/fs/ChangelogStreamHandleReaderWithCacheTest.java:
##
@@ -0,0 +1,78 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog.fs;
+
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.CoreOptions;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.runtime.state.filesystem.FileStateHandle;
+import org.apache.flink.runtime.state.memory.ByteStreamStateHandle;
+
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.io.TempDir;
+
+import java.io.DataInputStream;
+import java.time.Duration;
+import java.util.UUID;
+
+import static
org.apache.flink.changelog.fs.FsStateChangelogOptions.CACHE_IDLE_TIMEOUT;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+/** {@link ChangelogStreamHandleReaderWithCache} test. */
+class ChangelogStreamHandleReaderWithCacheTest {
+
+@TempDir java.nio.file.Path tempFolder;
+
+@Test
+void testCloseStreamTwice() throws Exception {
+String path = tempFolder.toUri().getPath();
+
+byte[] bytes = {0x00}; // not compressed, empty data
+ByteStreamStateHandle byteStreamHandle =
+new ByteStreamStateHandle(UUID.randomUUID().toString(), bytes);
+
+FileStateHandle stateHandler = mock(FileStateHandle.class);
+
when(stateHandler.openInputStream()).thenReturn(byteStreamHandle.openInputStream());
+
+FileSystem mockFileSystem = mock(FileSystem.class);
+when(mockFileSystem.isDistributedFS()).thenReturn(true);
Review Comment:
Thanks @rkhachatryan , I updated the pr and use real objects instead of
mocks. But I'm confused about "tests with mocks are fragile", can you explain
further ?
BTW, I squash all commits to one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] FangYongs opened a new pull request, #515: [FLINK-30999] Introduce flink-table-store-test-utils for table store
FangYongs opened a new pull request, #515: URL: https://github.com/apache/flink-table-store/pull/515 Introduce flink-table-store-test-utils module for table store -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30999) Introduce flink-table-store-test-utils for table store
[ https://issues.apache.org/jira/browse/FLINK-30999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30999: --- Labels: pull-request-available (was: ) > Introduce flink-table-store-test-utils for table store > -- > > Key: FLINK-30999 > URL: https://issues.apache.org/jira/browse/FLINK-30999 > Project: Flink > Issue Type: Sub-task > Components: Table Store >Affects Versions: table-store-0.4.0 >Reporter: Shammon >Priority: Major > Labels: pull-request-available > > Introduce flink-table-store-test-utils module for table store -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] FangYongs closed pull request #513: [FLINK-30999] Introduce flink-table-store-test-utils for table store
FangYongs closed pull request #513: [FLINK-30999] Introduce flink-table-store-test-utils for table store URL: https://github.com/apache/flink-table-store/pull/513 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SinBex commented on pull request #21898: [FLINK-30904][docs] Update the documentation and configuration description of slow task detector.
SinBex commented on PR #21898: URL: https://github.com/apache/flink/pull/21898#issuecomment-1425118977 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] FangYongs commented on pull request #513: [FLINK-30990] Introduce flink-table-store-test-utils for table store
FangYongs commented on PR #513: URL: https://github.com/apache/flink-table-store/pull/513#issuecomment-1425114950 > I guess the JIRA ID should be `FLINK-30999`? Yes, thanks :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] zhuzhurk commented on pull request #513: [FLINK-30990] Introduce flink-table-store-test-utils for table store
zhuzhurk commented on PR #513: URL: https://github.com/apache/flink-table-store/pull/513#issuecomment-1425106008 I guess the JIRA ID should be `FLINK-30999`? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on pull request #21897: [FLINK-30922][table-planner] Apply persisted columns when doing appendPartitionAndNu…
shuiqiangchen commented on PR #21897: URL: https://github.com/apache/flink/pull/21897#issuecomment-1425101680 @PatrickRen could you please help figure out why it didn't trigger the azure run after the `@flinkbot run azure` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fredia commented on pull request #21895: [FLINK-30792][state/changelog] make state changelog pre-emptively upl…
fredia commented on PR #21895: URL: https://github.com/apache/flink/pull/21895#issuecomment-1425101525 Thanks for the update, CI test failed because of https://issues.apache.org/jira/browse/FLINK-30972, maybe we can rebase this into PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-30966) Flink SQL IF FUNCTION logic error
[
https://issues.apache.org/jira/browse/FLINK-30966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686843#comment-17686843
]
Shuiqiang Chen commented on FLINK-30966:
BTW, the expected result of your query might be:
+I[succeed, sent, u, 2023-02-08]
> Flink SQL IF FUNCTION logic error
> -
>
> Key: FLINK-30966
> URL: https://issues.apache.org/jira/browse/FLINK-30966
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.16.1
>Reporter: 谢波
>Priority: Major
>
> my data is
> {code:java}
> //
> { "before": { "status": "sent" }, "after": { "status": "succeed" }, "op":
> "u", "ts_ms": 1671926400225, "transaction": null } {code}
> my sql is
>
> {code:java}
> CREATE TABLE t
> (
> before ROW (
> status varchar (32)
> ),
> after ROW (
> status varchar (32)
> ),
> ts_msbigint,
> op string,
> kafka_timestamp timestamp METADATA FROM 'timestamp',
> -- @formatter:off
> proctime AS PROCTIME()
> -- @formatter:on
> ) WITH (
> 'connector' = 'kafka',
> -- 'topic' = '',
> 'topic' = 'test',
> 'properties.bootstrap.servers' = ' ',
> 'properties.group.id' = '',
> 'format' = 'json',
> 'scan.topic-partition-discovery.interval' = '60s',
> 'scan.startup.mode' = 'earliest-offset',
> 'json.ignore-parse-errors' = 'true'
> );
> create table p
> (
> status STRING ,
> before_status STRING ,
> after_status STRING ,
> metadata_operation STRING COMMENT '源记录操作类型',
> dt STRING
> )WITH (
> 'connector' = 'print'
> );
> INSERT INTO p
> SELECT
>IF(op <> 'd', after.status, before.status),
> before.status,
> after.status,
>op AS metadata_operation,
>DATE_FORMAT(kafka_timestamp, '-MM-dd') AS dt
> FROM t;
> {code}
> my local env output is
>
>
> {code:java}
> +I[null, sent, succeed, u, 2023-02-08] {code}
>
> my produtionc env output is
> {code:java}
> +I[sent, sent, succeed, u, 2023-02-08] {code}
> why?
> This look like a bug.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30966) Flink SQL IF FUNCTION logic error
[
https://issues.apache.org/jira/browse/FLINK-30966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686842#comment-17686842
]
Shuiqiang Chen edited comment on FLINK-30966 at 2/10/23 2:39 AM:
-
Hi [~hiscat], I have reproduced the same error and it seems a bug in IFCallGen.
After investigated the generated code, there are two problems:
1. It perform the result term casting before the calculation logic, and finally
the actual result always refer to a non-initialized field.
2. when normalizing arguments, it always align to the type of ARG1, like IF(1 >
2, 'true', 'false')
the result will be string 'fals' which length is the same as 'true'.
I would like to help fix it.
was (Author: csq):
Hi [~hiscat], I have reproduced the same error and it seems a bug in IFCallGen.
There are two problems.
1. It do the result term casting before the calculation logic, and finally the
actual result always refer to a non-initialized field.
2. when normalizing arguments, it always align to the type of ARG1, like IF(1 >
2, 'true', 'false')
the result will be string 'fals' which length is the same as 'true'.
I would like to help fix it.
> Flink SQL IF FUNCTION logic error
> -
>
> Key: FLINK-30966
> URL: https://issues.apache.org/jira/browse/FLINK-30966
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.16.1
>Reporter: 谢波
>Priority: Major
>
> my data is
> {code:java}
> //
> { "before": { "status": "sent" }, "after": { "status": "succeed" }, "op":
> "u", "ts_ms": 1671926400225, "transaction": null } {code}
> my sql is
>
> {code:java}
> CREATE TABLE t
> (
> before ROW (
> status varchar (32)
> ),
> after ROW (
> status varchar (32)
> ),
> ts_msbigint,
> op string,
> kafka_timestamp timestamp METADATA FROM 'timestamp',
> -- @formatter:off
> proctime AS PROCTIME()
> -- @formatter:on
> ) WITH (
> 'connector' = 'kafka',
> -- 'topic' = '',
> 'topic' = 'test',
> 'properties.bootstrap.servers' = ' ',
> 'properties.group.id' = '',
> 'format' = 'json',
> 'scan.topic-partition-discovery.interval' = '60s',
> 'scan.startup.mode' = 'earliest-offset',
> 'json.ignore-parse-errors' = 'true'
> );
> create table p
> (
> status STRING ,
> before_status STRING ,
> after_status STRING ,
> metadata_operation STRING COMMENT '源记录操作类型',
> dt STRING
> )WITH (
> 'connector' = 'print'
> );
> INSERT INTO p
> SELECT
>IF(op <> 'd', after.status, before.status),
> before.status,
> after.status,
>op AS metadata_operation,
>DATE_FORMAT(kafka_timestamp, '-MM-dd') AS dt
> FROM t;
> {code}
> my local env output is
>
>
> {code:java}
> +I[null, sent, succeed, u, 2023-02-08] {code}
>
> my produtionc env output is
> {code:java}
> +I[sent, sent, succeed, u, 2023-02-08] {code}
> why?
> This look like a bug.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30966) Flink SQL IF FUNCTION logic error
[
https://issues.apache.org/jira/browse/FLINK-30966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686842#comment-17686842
]
Shuiqiang Chen edited comment on FLINK-30966 at 2/10/23 2:38 AM:
-
Hi [~hiscat], I have reproduced the same error and it seems a bug in IFCallGen.
There are two problems.
1. It do the result term casting before the calculation logic, and finally the
actual result always refer to a non-initialized field.
2. when normalizing arguments, it always align to the type of ARG1, like IF(1 >
2, 'true', 'false')
the result will be string 'fals' which length is the same as 'true'.
I would like to help fix it.
was (Author: csq):
Hi [~hiscat], I have reproduced the same error and it seems a bug in IFCodeGen.
There are two problems.
1. It do the result term casting before the calculation logic, and finally the
actual result always refer to a non-initialized field.
2. when normalizing arguments, it always align to the type of ARG1, like IF(1 >
2, 'true', 'false')
the result will be string 'fals' which length is the same as 'true'.
I would like to help fix it.
> Flink SQL IF FUNCTION logic error
> -
>
> Key: FLINK-30966
> URL: https://issues.apache.org/jira/browse/FLINK-30966
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.16.1
>Reporter: 谢波
>Priority: Major
>
> my data is
> {code:java}
> //
> { "before": { "status": "sent" }, "after": { "status": "succeed" }, "op":
> "u", "ts_ms": 1671926400225, "transaction": null } {code}
> my sql is
>
> {code:java}
> CREATE TABLE t
> (
> before ROW (
> status varchar (32)
> ),
> after ROW (
> status varchar (32)
> ),
> ts_msbigint,
> op string,
> kafka_timestamp timestamp METADATA FROM 'timestamp',
> -- @formatter:off
> proctime AS PROCTIME()
> -- @formatter:on
> ) WITH (
> 'connector' = 'kafka',
> -- 'topic' = '',
> 'topic' = 'test',
> 'properties.bootstrap.servers' = ' ',
> 'properties.group.id' = '',
> 'format' = 'json',
> 'scan.topic-partition-discovery.interval' = '60s',
> 'scan.startup.mode' = 'earliest-offset',
> 'json.ignore-parse-errors' = 'true'
> );
> create table p
> (
> status STRING ,
> before_status STRING ,
> after_status STRING ,
> metadata_operation STRING COMMENT '源记录操作类型',
> dt STRING
> )WITH (
> 'connector' = 'print'
> );
> INSERT INTO p
> SELECT
>IF(op <> 'd', after.status, before.status),
> before.status,
> after.status,
>op AS metadata_operation,
>DATE_FORMAT(kafka_timestamp, '-MM-dd') AS dt
> FROM t;
> {code}
> my local env output is
>
>
> {code:java}
> +I[null, sent, succeed, u, 2023-02-08] {code}
>
> my produtionc env output is
> {code:java}
> +I[sent, sent, succeed, u, 2023-02-08] {code}
> why?
> This look like a bug.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30966) Flink SQL IF FUNCTION logic error
[
https://issues.apache.org/jira/browse/FLINK-30966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686842#comment-17686842
]
Shuiqiang Chen commented on FLINK-30966:
Hi [~hiscat], I have reproduced the same error and it seems a bug in IFCodeGen.
There are two problems.
1. It do the result term casting before the calculation logic, and finally the
actual result always refer to a non-initialized field.
2. when normalizing arguments, it always align to the type of ARG1, like IF(1 >
2, 'true', 'false')
the result will be string 'fals' which length is the same as 'true'.
I would like to help fix it.
> Flink SQL IF FUNCTION logic error
> -
>
> Key: FLINK-30966
> URL: https://issues.apache.org/jira/browse/FLINK-30966
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.16.1
>Reporter: 谢波
>Priority: Major
>
> my data is
> {code:java}
> //
> { "before": { "status": "sent" }, "after": { "status": "succeed" }, "op":
> "u", "ts_ms": 1671926400225, "transaction": null } {code}
> my sql is
>
> {code:java}
> CREATE TABLE t
> (
> before ROW (
> status varchar (32)
> ),
> after ROW (
> status varchar (32)
> ),
> ts_msbigint,
> op string,
> kafka_timestamp timestamp METADATA FROM 'timestamp',
> -- @formatter:off
> proctime AS PROCTIME()
> -- @formatter:on
> ) WITH (
> 'connector' = 'kafka',
> -- 'topic' = '',
> 'topic' = 'test',
> 'properties.bootstrap.servers' = ' ',
> 'properties.group.id' = '',
> 'format' = 'json',
> 'scan.topic-partition-discovery.interval' = '60s',
> 'scan.startup.mode' = 'earliest-offset',
> 'json.ignore-parse-errors' = 'true'
> );
> create table p
> (
> status STRING ,
> before_status STRING ,
> after_status STRING ,
> metadata_operation STRING COMMENT '源记录操作类型',
> dt STRING
> )WITH (
> 'connector' = 'print'
> );
> INSERT INTO p
> SELECT
>IF(op <> 'd', after.status, before.status),
> before.status,
> after.status,
>op AS metadata_operation,
>DATE_FORMAT(kafka_timestamp, '-MM-dd') AS dt
> FROM t;
> {code}
> my local env output is
>
>
> {code:java}
> +I[null, sent, succeed, u, 2023-02-08] {code}
>
> my produtionc env output is
> {code:java}
> +I[sent, sent, succeed, u, 2023-02-08] {code}
> why?
> This look like a bug.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] chucheng92 commented on pull request #21902: [FLINK-30974] Add commons-io to flink-sql-connector-hbase
chucheng92 commented on PR #21902: URL: https://github.com/apache/flink/pull/21902#issuecomment-1425094522 @xicm good job ! btw, i think u need to modify your commit and PR name like: [FLINK-30974][connectors/hbase] Add commons-io to flink-sql-connector-hbase or [FLINK-30974][build] Add commons-io to flink-sql-connector-hbase to add module name. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-30508) CliClientITCase.testSqlStatements failed with output not matched with expected
[ https://issues.apache.org/jira/browse/FLINK-30508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686840#comment-17686840 ] Shengkai Fang commented on FLINK-30508: --- [~mapohl] you are right. But we haven't cherry pick the fix to the 1.16 yet. > CliClientITCase.testSqlStatements failed with output not matched with expected > -- > > Key: FLINK-30508 > URL: https://issues.apache.org/jira/browse/FLINK-30508 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.16.0, 1.17.0 >Reporter: Qingsheng Ren >Assignee: Shengkai Fang >Priority: Critical > Labels: test-stability > Fix For: 1.17.0 > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44246=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=14992 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30936) Release Testing: Verify FLINK-29941 Support SQL Client connects to SQL Gateway
[
https://issues.apache.org/jira/browse/FLINK-30936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686838#comment-17686838
]
Shengkai Fang commented on FLINK-30936:
---
Yes, you are right. I have unassigned me.
> Release Testing: Verify FLINK-29941 Support SQL Client connects to SQL Gateway
> --
>
> Key: FLINK-30936
> URL: https://issues.apache.org/jira/browse/FLINK-30936
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Qingsheng Ren
>Priority: Blocker
> Labels: release-testing
> Fix For: 1.17.0
>
>
> The issue aims to verify [FLIP-275 Support SQL Client connects to SQL
> Gateway|https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway]
> Please verify:
> In embedded mode:
> # All CLI parameter still work in the embedded mode including -i, -f, --jar,
> --libary
> # Please submit DDL/DML to verify SQL Client can execute SQL correctly
> # Please submit a query and use CTRL+C to cancel the query to make sure SQL
> Client can cancel the job
> # Please submit a SQL with error to make sure the Client can display
> exception clearly with/without {{sql-client.verbose}} option
> In gateway mode:
> 1. Please verify SQL Client can connect to the gateway with --endpoint
> parameter
> 2. When connecting to release-1.16 Gateway, the SQL Client stop with
> clear exception.
> 3. Leave the SQL Client alone for a long time, the SQL Client is still
> alive and can submit SQL.
> If you meet any problems, it's welcome to ping me directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] fredia commented on a diff in pull request #21812: [FLINK-28440][state/changelog] record reference count of StreamStateHandle in TaskChangelogRegistry
fredia commented on code in PR #21812:
URL: https://github.com/apache/flink/pull/21812#discussion_r1102214645
##
flink-dstl/flink-dstl-dfs/src/test/java/org/apache/flink/changelog/fs/TestingBatchingUploadScheduler.java:
##
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.changelog.fs;
+
+import java.io.IOException;
+import java.util.LinkedList;
+import java.util.Queue;
+
+/** Implementation class for {@link StateChangeUploadScheduler} to test. */
+class TestingBatchingUploadScheduler implements StateChangeUploadScheduler {
Review Comment:
Thanks for the clarification, you're right, `testFileAvailableAfterClose`
needs it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-30936) Release Testing: Verify FLINK-29941 Support SQL Client connects to SQL Gateway
[
https://issues.apache.org/jira/browse/FLINK-30936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shengkai Fang reassigned FLINK-30936:
-
Assignee: (was: Shengkai Fang)
> Release Testing: Verify FLINK-29941 Support SQL Client connects to SQL Gateway
> --
>
> Key: FLINK-30936
> URL: https://issues.apache.org/jira/browse/FLINK-30936
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Qingsheng Ren
>Priority: Blocker
> Labels: release-testing
> Fix For: 1.17.0
>
>
> The issue aims to verify [FLIP-275 Support SQL Client connects to SQL
> Gateway|https://cwiki.apache.org/confluence/display/FLINK/FLIP-275%3A+Support+Remote+SQL+Client+Based+on+SQL+Gateway]
> Please verify:
> In embedded mode:
> # All CLI parameter still work in the embedded mode including -i, -f, --jar,
> --libary
> # Please submit DDL/DML to verify SQL Client can execute SQL correctly
> # Please submit a query and use CTRL+C to cancel the query to make sure SQL
> Client can cancel the job
> # Please submit a SQL with error to make sure the Client can display
> exception clearly with/without {{sql-client.verbose}} option
> In gateway mode:
> 1. Please verify SQL Client can connect to the gateway with --endpoint
> parameter
> 2. When connecting to release-1.16 Gateway, the SQL Client stop with
> clear exception.
> 3. Leave the SQL Client alone for a long time, the SQL Client is still
> alive and can submit SQL.
> If you meet any problems, it's welcome to ping me directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30990) Performance of deployDownstreamTasks.BATCH has worsened
[ https://issues.apache.org/jira/browse/FLINK-30990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30990: --- Labels: pull-request-available (was: ) > Performance of deployDownstreamTasks.BATCH has worsened > --- > > Key: FLINK-30990 > URL: https://issues.apache.org/jira/browse/FLINK-30990 > Project: Flink > Issue Type: Bug >Affects Versions: 1.17.0 >Reporter: Martijn Visser >Priority: Blocker > Labels: pull-request-available > > See > http://codespeed.dak8s.net:8000/timeline/#/?exe=8=deployDownstreamTasks.BATCH=on=on=off=2=200 > It appears that the trend has now stabilized upwards, from 60-70 ms/op to > 80-90 ms/op, while less is better. > Not sure if this has meaningful impact in a production setup, but I would > like to get this verified. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] FangYongs opened a new pull request, #513: [FLINK-30990] Introduce flink-table-store-test-utils for table store
FangYongs opened a new pull request, #513: URL: https://github.com/apache/flink-table-store/pull/513 This pr aims to introduce flink-table-store-test-utils for test units in table store -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-30712) Update network memory configuration docs
[ https://issues.apache.org/jira/browse/FLINK-30712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-30712. Fix Version/s: 1.17.0 Resolution: Done - master (1.18): ae89c997fa7c431a0ff2078af441c938fd66e6b9 - release-1.17: 022ab7680f0dab511b2f498acbb89edcaaad2624 > Update network memory configuration docs > - > > Key: FLINK-30712 > URL: https://issues.apache.org/jira/browse/FLINK-30712 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Affects Versions: 1.17.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > After [FLINK-30469|https://issues.apache.org/jira/browse/FLINK-30469], the > network memory configuration docs for TaskManager should also be updated. > The configuration descriptions to be updated mainly include > `taskmanager.network.memory.buffers-per-channel` > and `taskmanager.network.memory.floating-buffers-per-gate`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30712) Update network memory configuration docs
[ https://issues.apache.org/jira/browse/FLINK-30712?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-30712: - Component/s: Documentation > Update network memory configuration docs > - > > Key: FLINK-30712 > URL: https://issues.apache.org/jira/browse/FLINK-30712 > Project: Flink > Issue Type: Improvement > Components: Documentation, Runtime / Network >Affects Versions: 1.17.0 >Reporter: Yuxin Tan >Assignee: Yuxin Tan >Priority: Major > Labels: pull-request-available > Fix For: 1.17.0 > > > After [FLINK-30469|https://issues.apache.org/jira/browse/FLINK-30469], the > network memory configuration docs for TaskManager should also be updated. > The configuration descriptions to be updated mainly include > `taskmanager.network.memory.buffers-per-channel` > and `taskmanager.network.memory.floating-buffers-per-gate`. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] xintongsong closed pull request #21843: [FLINK-30712][network] Update documents for taskmanager memory configurations and tuning
xintongsong closed pull request #21843: [FLINK-30712][network] Update documents for taskmanager memory configurations and tuning URL: https://github.com/apache/flink/pull/21843 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-30622) Support auto consume schema in Pulsar source
[ https://issues.apache.org/jira/browse/FLINK-30622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zili Chen closed FLINK-30622. - Resolution: Fixed master via https://github.com/apache/flink-connector-pulsar/pull/19 > Support auto consume schema in Pulsar source > > > Key: FLINK-30622 > URL: https://issues.apache.org/jira/browse/FLINK-30622 > Project: Flink > Issue Type: New Feature > Components: Connectors / Pulsar >Affects Versions: pulsar-4.0.0 >Reporter: Yufan Sheng >Assignee: Yufan Sheng >Priority: Major > Labels: pull-request-available > Fix For: pulsar-4.0.0 > > > Pulsar client supports [auto > schema|https://pulsar.apache.org/docs/2.10.x/schema-understand/#auto-schema] > if you don't know the schema type of a Pulsar topic in advance. > This can make the message consuming from Pulsar topic which contains multiple > schema extremely easy. We plan to add auto schema support in Pulsar source. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-connector-pulsar] tisonkun merged pull request #19: [FLINK-30622][Connector/Pulsar] Support consuming messages with schema auto-detection from Pulsar.
tisonkun merged PR #19: URL: https://github.com/apache/flink-connector-pulsar/pull/19 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xicm commented on pull request #21902: [FLINK-30974] Add commons-io to flink-sql-connector-hbase
xicm commented on PR #21902: URL: https://github.com/apache/flink/pull/21902#issuecomment-1425066624 @MartijnVisser @chucheng92 Thank you for your suggestions. I am new to flink, is it to update this file https://github.com/apache/flink/blob/master/flink-connectors/flink-sql-connector-hbase-2.2/src/main/resources/META-INF/NOTICE -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-28283) Improving the log of flink when job start and deploy
[ https://issues.apache.org/jira/browse/FLINK-28283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17686825#comment-17686825 ] Xintong Song commented on FLINK-28283: -- I'm also inclined to close the ticket. But let's wait a bit longer for the response from the reporter. > Improving the log of flink when job start and deploy > > > Key: FLINK-28283 > URL: https://issues.apache.org/jira/browse/FLINK-28283 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Affects Versions: 1.14.2 >Reporter: zlzhang0122 >Priority: Major > > When running a large job with many operators and subtasks on flink, the > JobManager and TaskManager will have a huge logs about the subtask executing > msg such as "XXX switched from CREATED to SCHEDULED、XXX switched from > SCHEDULED to DEPLOYING 、XXX switched from DEPLOYING to RUNNING 、XXX switched > from RUNNING to CANCELING、XXX switched from CANCELING to CANCELED", etc. . > Maybe we can do some improvement about this, such as aggregate these msg to > reduce the log, or change the log level and only logs the failure msg and > subtask, etc. Not so sure about the solution, but these msg is really too > much. -- This message was sent by Atlassian Jira (v8.20.10#820010)
