[jira] [Created] (FLINK-35939) Do not set empty config values via ConfigUtils#encodeCollectionToConfig
Ferenc Csaky created FLINK-35939:
Summary: Do not set empty config values via
ConfigUtils#encodeCollectionToConfig
Key: FLINK-35939
URL: https://issues.apache.org/jira/browse/FLINK-35939
Project: Flink
Issue Type: Improvement
Affects Versions: 1.19.1
Reporter: Ferenc Csaky
Fix For: 2.0.0
The {{ConfigUtils#encodeCollectionToConfig}} function only skips to set a given
{{ConfigOption}} value, if that value is null. If the passed collection is
empty, it will set that empty collection.
I think this behavior is less logical and can cause more undesired situations,
when we only set a value if it is not empty AND not null.
Furthermore, the method's
[javadoc|https://github.com/apache/flink/blob/82b628d4730eef32b2f7a022e3b73cb18f950e6e/flink-core/src/main/java/org/apache/flink/configuration/ConfigUtils.java#L73]
describes the logic I just mentioned above, which is in conflict with the
actual implementation and tests, which sets an empty collection.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35699) The flink-kubernetes artifact shades Jackson 2.15.3 from fabric8
Ferenc Csaky created FLINK-35699:
Summary: The flink-kubernetes artifact shades Jackson 2.15.3 from
fabric8
Key: FLINK-35699
URL: https://issues.apache.org/jira/browse/FLINK-35699
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.19.1
Reporter: Ferenc Csaky
Fix For: 1.20.0, 1.19.2
The {{flink-kubernetes}} artifact shades Jackson classes coming through
fabric8, but since Jackson 2.15, Jackson is a [multi-release
JAR|https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.15#jar-changes],
which requires some additional relocations for correct shading.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35695) Release Testing: Verify FLINK-32315: Support local file upload in K8s mode
Ferenc Csaky created FLINK-35695:
Summary: Release Testing: Verify FLINK-32315: Support local file
upload in K8s mode
Key: FLINK-35695
URL: https://issues.apache.org/jira/browse/FLINK-35695
Project: Flink
Issue Type: Sub-task
Components: Runtime / Network
Reporter: Ferenc Csaky
Fix For: 1.20.0
Follow up the test for https://issues.apache.org/jira/browse/FLINK-35533
In Flink 1.20, we proposed integrating Flink's Hybrid Shuffle with Apache
Celeborn through a pluggable remote tier interface. To verify this feature, you
should reference these main two steps.
1. Implement Celeborn tier.
* Implement a new tier factory and tier for Celeborn, including these APIs,
including TierFactory/TierMasterAgent/TierProducerAgent/TierConsumerAgent.
* The implementations should support granular data management at the Segment
level for both client and server sides.
2. Use the implemented tier to shuffle data.
* Compile Flink and Celeborn.
* Deploy Celeborn service
** Deploy a new Celeborn service with the new compiled packages. You can
reference the doc ([https://celeborn.apache.org/docs/latest/]) to deploy the
cluster.
* Add the compiled flink plugin jar (celeborn-client-flink-xxx.jar) to Flink
classpath.
* Configure the options to enable the feature.
** Configure the option
taskmanager.network.hybrid-shuffle.external-remote-tier-factory.class to the
new Celeborn tier classes. Except for this option, the following options should
also be added.
{code:java}
execution.batch-shuffle-mode: ALL_EXCHANGES_HYBRID_FULL
celeborn.master.endpoints:
celeborn.client.shuffle.partition.type: MAP{code}
* Run some test examples(e.g., WordCount) to verify the feature.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35662) Use maven batch mode in k8s-operator CI
Ferenc Csaky created FLINK-35662:
Summary: Use maven batch mode in k8s-operator CI
Key: FLINK-35662
URL: https://issues.apache.org/jira/browse/FLINK-35662
Project: Flink
Issue Type: Improvement
Components: Kubernetes Operator
Reporter: Ferenc Csaky
Fix For: kubernetes-operator-1.10.0
Currently, the GitHub workflows do not use batch mode in the k8s-operator repo,
so there are a lot of lines in the log like this:
{code}
Progress (1): 4.1/14 kB
Progress (1): 8.2/14 kB
Progress (1): 12/14 kB
Progress (1): 14 kB
{code}
To produce logs that are for more easy to navigate, all {{mvn}} calls should
apply the batch-mode option {{-B}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-35649) Bump Flink version to 1.19.1 in k8s-operator
Ferenc Csaky created FLINK-35649: Summary: Bump Flink version to 1.19.1 in k8s-operator Key: FLINK-35649 URL: https://issues.apache.org/jira/browse/FLINK-35649 Project: Flink Issue Type: Improvement Components: Kubernetes Operator Reporter: Ferenc Csaky Fix For: kubernetes-operator-1.10.0 In FLINK-28915 it came up the the operator is not able to utilize the artifact fetching capabilities that was introduced in Flink 1.19 until it is not built on that version. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35625) FLIP-464: Merge "flink run" and "flink run-application"
Ferenc Csaky created FLINK-35625: Summary: FLIP-464: Merge "flink run" and "flink run-application" Key: FLINK-35625 URL: https://issues.apache.org/jira/browse/FLINK-35625 Project: Flink Issue Type: Improvement Components: Client / Job Submission, Command Line Client Reporter: Ferenc Csaky Fix For: 1.20.0 Ticket to track [FLIP-464|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=311626179]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-35114) Remove old Table API implementations
Ferenc Csaky created FLINK-35114: Summary: Remove old Table API implementations Key: FLINK-35114 URL: https://issues.apache.org/jira/browse/FLINK-35114 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky At the moment, the connector has both the old Table sink/source/catalog implementations and the matching Dynamic... implementations as well. Going forward, the deprecated old implementation should be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34931) Update Kudu connector DataStream Source/Sink implementation
Ferenc Csaky created FLINK-34931: Summary: Update Kudu connector DataStream Source/Sink implementation Key: FLINK-34931 URL: https://issues.apache.org/jira/browse/FLINK-34931 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky Update the DataSource API classes to use the current interfaces. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34930) Move existing Kudu connector code from Bahir repo to dedicated repo
Ferenc Csaky created FLINK-34930: Summary: Move existing Kudu connector code from Bahir repo to dedicated repo Key: FLINK-34930 URL: https://issues.apache.org/jira/browse/FLINK-34930 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34929) Create "flink-connector-kudu" repository
Ferenc Csaky created FLINK-34929: Summary: Create "flink-connector-kudu" repository Key: FLINK-34929 URL: https://issues.apache.org/jira/browse/FLINK-34929 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky We should create a "flink-connector-kudu" repositry under the "apache" GitHub organization. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34928) FLIP-439: Externalize Kudu Connector from Bahir
Ferenc Csaky created FLINK-34928: Summary: FLIP-439: Externalize Kudu Connector from Bahir Key: FLINK-34928 URL: https://issues.apache.org/jira/browse/FLINK-34928 Project: Flink Issue Type: Improvement Reporter: Ferenc Csaky Umbrella issue for: https://cwiki.apache.org/confluence/display/FLINK/FLIP-439%3A+Externalize+Kudu+Connector+from+Bahir -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34580) Job run via REST erases "pipeline.classpaths" config
Ferenc Csaky created FLINK-34580:
Summary: Job run via REST erases "pipeline.classpaths" config
Key: FLINK-34580
URL: https://issues.apache.org/jira/browse/FLINK-34580
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.18.1, 1.17.2, 1.19.0
Reporter: Ferenc Csaky
Fix For: 1.20.0
The
[{{JarHandlerContext#applyToConfiguration}}|https://github.com/apache/flink/blob/e0b6c121eaf7aeb2974a45d199e452b022f07d29/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/utils/JarHandlerUtils.java#L134]
creates a {{PackagedProgram}} and then overwrites the {{pipeline.jars}} and
{{pipeline.classpaths}} values according to that newly created
{{{}PackagedProgram{}}}.
Although that [{{PackagedProgram}}
init|https://github.com/apache/flink/blob/e0b6c121eaf7aeb2974a45d199e452b022f07d29/flink-runtime-web/src/main/java/org/apache/flink/runtime/webmonitor/handlers/utils/JarHandlerUtils.java#L185]
does not set {{classpaths}} at all, so it will always overwrites the effective
configuration with an empty value, even if it had something previously.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34506) Do not copy "file://" schemed artifact in standalone application modes
Ferenc Csaky created FLINK-34506: Summary: Do not copy "file://" schemed artifact in standalone application modes Key: FLINK-34506 URL: https://issues.apache.org/jira/browse/FLINK-34506 Project: Flink Issue Type: Bug Components: Client / Job Submission Affects Versions: 1.19.0 Reporter: Ferenc Csaky In standalone application mode, if an artifact is passed via a path witohut prefix, the file will be copied to `user.artifacts.base-dir`, although it should not be, as it can accessable locally. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-34388) Release Testing: Verify FLINK-28915 Support artifact fetching in Standalone and native K8s application mode
Ferenc Csaky created FLINK-34388: Summary: Release Testing: Verify FLINK-28915 Support artifact fetching in Standalone and native K8s application mode Key: FLINK-34388 URL: https://issues.apache.org/jira/browse/FLINK-34388 Project: Flink Issue Type: Sub-task Components: Runtime / Metrics Affects Versions: 1.19.0 Reporter: Ferenc Csaky Fix For: 1.19.0 This ticket covers testing three related features: FLINK-33695, FLINK-33735 and FLINK-33696. Instructions: # Configure Flink to use [Slf4jTraceReporter|https://nightlies.apache.org/flink/flink-docs-master/docs/deployment/trace_reporters/#slf4j] and with enabled *INFO* level logging (can be to console or to a file, doesn't matter). # Start a streaming job with enabled checkpointing. # Let it run for a couple of checkpoints. # Verify presence of a single *JobInitialization* [1] trace logged just after job start up. # Verify presence of a couple of *Checkpoint* [1] traces logged after each successful or failed checkpoint. [1] https://nightlies.apache.org/flink/flink-docs-master/docs/ops/traces/#checkpointing-and-initialization -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33542) Update HBase connector tests to JUnit5
Ferenc Csaky created FLINK-33542: Summary: Update HBase connector tests to JUnit5 Key: FLINK-33542 URL: https://issues.apache.org/jira/browse/FLINK-33542 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33440) Bump flink version on flink-connectors-hbase
Ferenc Csaky created FLINK-33440: Summary: Bump flink version on flink-connectors-hbase Key: FLINK-33440 URL: https://issues.apache.org/jira/browse/FLINK-33440 Project: Flink Issue Type: Improvement Components: Connectors / HBase Reporter: Ferenc Csaky Follow-up the 1.18 release in the connector repo as well. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33353) SQL fails because "TimestampType.kind" is not serialized
Ferenc Csaky created FLINK-33353:
Summary: SQL fails because "TimestampType.kind" is not serialized
Key: FLINK-33353
URL: https://issues.apache.org/jira/browse/FLINK-33353
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.18.0
Reporter: Ferenc Csaky
We have a custom persistent catalog store, which stores tables, views etc. in a
DB. In our application, it is required to utilize the serialized formats of
entities, but the same applies to the Hive, as it functions as a persistent
catalog.
Take the following example SQL:
{code:sql}
CREATE TABLE IF NOT EXISTS `txn_gen` (
`txn_id` INT,
`amount` INT,
`ts` TIMESTAMP(3),
WATERMARK FOR `ts` AS `ts` - INTERVAL '1' SECOND
) WITH (
'connector' = 'datagen',
'fields.txn_id.min' = '1',
'fields.txn_id.max' = '5',
'rows-per-second' = '1'
);
CREATE VIEW IF NOT EXISTS aggr_ten_sec AS
SELECT txn_id,
TUMBLE_ROWTIME(`ts`, INTERVAL '10' SECOND) AS w_row_time,
COUNT(txn_id) AS txn_count
FROM txn_gen
GROUP BY txn_id, TUMBLE(`ts`, INTERVAL '10' SECOND);
SELECT txn_id,
SUM(txn_count),
TUMBLE_START(w_row_time, INTERVAL '20' SECOND) AS total_txn_count
FROM aggr_ten_sec
GROUP BY txn_id, TUMBLE(w_row_time, INTERVAL '20' SECOND);
{code}
This will work without any problems when we simply execute it in a
{{TableEnvironment}}, but it fails with the below error when we try to execute
the query based on the serialized table metadata.
{code}
org.apache.flink.table.api.TableException: Window aggregate can only be defined
over a time attribute column, but TIMESTAMP(3) encountered.
{code}
If there is a view which would require to use ROWTIME, it will be lost and we
cannot recreate the same query from the serialized entites.
Currently in {{TimestampType}} the "kind" field is deliberatly annotated as
{{@Internal}} and is not serialized, although it breaks this functionality.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32811) Add port range support for taskmanager.data.bind-port
Ferenc Csaky created FLINK-32811:
Summary: Add port range support for taskmanager.data.bind-port
Key: FLINK-32811
URL: https://issues.apache.org/jira/browse/FLINK-32811
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration, Runtime / Coordination
Reporter: Ferenc Csaky
Fix For: 1.19.0
Adding this feature could be helpful for installation in a restrictive network
setup. The "port range" support is already available for some other port config
options anyway.
Right now, it is possible to specify a {{taskmanager.data.port}} and
{{taskmanager.data.bind-port}} to be able to support NAT-like setups, although
{{taskmanager.data.port}} is not bound to anything itself, so supporting a port
range there is not an option according to my understanding.
Although, supporting a port range only for {{taskmanager.data.bind-port}} can
be still helpful for anyone who does not require a NAT capability, because if
{{taskmanager.data.bind-port}} is set and {{taskmanager.data.port}} is set to
*0*, then the bound port will be used everywhere.
This change should keep the already possible setups working as is.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-32660) Support external file systems in FileCatalogStore
Ferenc Csaky created FLINK-32660: Summary: Support external file systems in FileCatalogStore Key: FLINK-32660 URL: https://issues.apache.org/jira/browse/FLINK-32660 Project: Flink Issue Type: Sub-task Reporter: Ferenc Csaky Fix For: 1.18.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32174) Update Cloudera product and link in doc page
Ferenc Csaky created FLINK-32174: Summary: Update Cloudera product and link in doc page Key: FLINK-32174 URL: https://issues.apache.org/jira/browse/FLINK-32174 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Ferenc Csaky -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-31085) Add schema option to confluent registry avro formats
Ferenc Csaky created FLINK-31085:
Summary: Add schema option to confluent registry avro formats
Key: FLINK-31085
URL: https://issues.apache.org/jira/browse/FLINK-31085
Project: Flink
Issue Type: Improvement
Reporter: Ferenc Csaky
Fix For: 1.17.0
When using {{avro-confluent}} and {{debezium-avro-confluent}} formats with
schemas already defined in the Confluent Schema Registry, serialization fails,
because Flink uses a default name `record` when converting row types to avro
schema. So if the predefined schema has a different name, the serialization
schema will be incompatible with the registered schema due to name mismatch.
Check [this|https://lists.apache.org/thread/5xppmnqjqwfzxqo4gvd3lzz8wzs566zp]
thread about reproducing the issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-29707) Fix possible comparator violation for "flink list"
Ferenc Csaky created FLINK-29707:
Summary: Fix possible comparator violation for "flink list"
Key: FLINK-29707
URL: https://issues.apache.org/jira/browse/FLINK-29707
Project: Flink
Issue Type: Bug
Components: Command Line Client
Affects Versions: 1.16.0
Reporter: Ferenc Csaky
For the {{list}} CLI option, the code that prints the jobs, there is a
{{startTimeComparator}} definition, which orders the jobs and it is done this
way:
{code:java}
Comparator startTimeComparator =
(o1, o2) -> (int) (o1.getStartTime() - o2.getStartTime());
{code}
In some rare situation this can lead to this:
{code:java}
2022-10-19 09:58:11,690 ERROR org.apache.flink.client.cli.CliFrontend
[] - Error while running the command.
java.lang.IllegalArgumentException: Comparison method violates its general
contract!
at java.util.TimSort.mergeLo(TimSort.java:777) ~[?:1.8.0_312]
at java.util.TimSort.mergeAt(TimSort.java:514) ~[?:1.8.0_312]
at java.util.TimSort.mergeForceCollapse(TimSort.java:457) ~[?:1.8.0_312]
at java.util.TimSort.sort(TimSort.java:254) ~[?:1.8.0_312]
at java.util.Arrays.sort(Arrays.java:1512) ~[?:1.8.0_312]
at java.util.ArrayList.sort(ArrayList.java:1464) ~[?:1.8.0_312]
at java.util.stream.SortedOps$RefSortingSink.end(SortedOps.java:392)
~[?:1.8.0_312]
at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
~[?:1.8.0_312]
at java.util.stream.Sink$ChainedReference.end(Sink.java:258)
~[?:1.8.0_312]
at
java.util.stream.SortedOps$SizedRefSortingSink.end(SortedOps.java:363)
~[?:1.8.0_312]
at
java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:483)
~[?:1.8.0_312]
at
java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
~[?:1.8.0_312]
at
java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150)
~[?:1.8.0_312]
at
java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173)
~[?:1.8.0_312]
at
java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
~[?:1.8.0_312]
at
java.util.stream.ReferencePipeline.forEachOrdered(ReferencePipeline.java:490)
~[?:1.8.0_312]
at
org.apache.flink.client.cli.CliFrontend.printJobStatusMessages(CliFrontend.java:574)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-27441) Scrollbar is missing for particular UI elements (Accumulators, Backpressure, Watermarks)
Ferenc Csaky created FLINK-27441:
Summary: Scrollbar is missing for particular UI elements
(Accumulators, Backpressure, Watermarks)
Key: FLINK-27441
URL: https://issues.apache.org/jira/browse/FLINK-27441
Project: Flink
Issue Type: Bug
Components: Runtime / Web Frontend
Affects Versions: 1.14.3, 1.15.0
Reporter: Ferenc Csaky
The angular version bump introduced a bug, where for {{nzScroll}} does not
support percentage in CSS calc, so the scrollbar will be invisible. There is an
easy workaround, the linked Angular discussion covers it.
Angular issue: https://github.com/NG-ZORRO/ng-zorro-antd/issues/3090
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
