Re: Questions around MemoryStateBackend size limits
Hi + user mail list The limit max state size is because we would send the checkpointed data as a byte array and send it back to jobmanager. If the checkpointed byte stream is too large, the job manager would meet the risk of out-of-memory-error. If you want to use heap-based state-backend, you could use FsStateBackend which actually share the same code as MemoryStateBackend but only checkpoint to external file system storage. 1. If the checkpoint stream size is over limit 5MB, the checkpoint phase on task side would fail due to the size check [1] 2. If the checkpoint stream over the limit, task would fail to report successful checkpoint message to JM [2] 3. Actually, we only care the checkpoint stream size instead of the state size for this field. [1] https://github.com/apache/flink/blob/1c78ab397de524836fd69c6218b1122aa387c251/flink-runtime/src/main/java/org/apache/flink/runtime/state/memory/MemCheckpointStreamFactory.java#L62-L69 [2] https://github.com/apache/flink/blob/1c78ab397de524836fd69c6218b1122aa387c251/flink-runtime/src/main/java/org/apache/flink/runtime/state/TaskStateManagerImpl.java#L117-L122 Best Yun Tang From: vishalovercome Sent: Tuesday, June 2, 2020 13:58 To: dev@flink.apache.org Subject: Questions around MemoryStateBackend size limits The documentation says: *The size of each individual state is by default limited to 5 MB. This value can be increased in the constructor of the MemoryStateBackend.* 1. I want to know what would happen if a job ended up adding elements to a state variable causing its size to exceed 5MB. There are other questions I have around this topic. 2. The doc mentions that akka frame size is the upper limit on state size. What would happen if we were to exceed that as well? Would it cause the program to fail or would it only affect checkpointing (as communication between job manager and task manager would breakdown) 3. If the size is within 5MB but the size of the checkpoint (additions, removals, updates) were to be greater than 5MB (or the akka frame size), then would the checkpoint fail? It will also help if you could provide this information within the documentation itself. -- Sent from: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/
[jira] [Created] (FLINK-18059) Can not execute create/drop catalog statement in sql client
godfrey he created FLINK-18059:
--
Summary: Can not execute create/drop catalog statement in sql
client
Key: FLINK-18059
URL: https://issues.apache.org/jira/browse/FLINK-18059
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.10.1
Reporter: godfrey he
when executing create catalog statement (e.g. {{create CATALOG c1
with('type'='generic_in_memory'}}) in sql client, the following exception will
occur:
Exception in thread "main" org.apache.flink.table.client.SqlClientException:
Unsupported command: CREATE CATALOG
at
org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:355)
at java.util.Optional.ifPresent(Optional.java:159)
at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:213)
at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:142)
at org.apache.flink.table.client.SqlClient.start(SqlClient.java:114)
at org.apache.flink.table.client.SqlClient.main(SqlClient.java:201)
Similar case for {{drop catalog}}.
The reason is CliClient class does not handle CREATE_CATALOG command and
DROP_CATALOG command.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18058) MesosResourceManagerTest.testWorkerStarted:656 » NullPointer
Robert Metzger created FLINK-18058:
--
Summary: MesosResourceManagerTest.testWorkerStarted:656 »
NullPointer
Key: FLINK-18058
URL: https://issues.apache.org/jira/browse/FLINK-18058
Project: Flink
Issue Type: Bug
Components: Deployment / Mesos
Affects Versions: 1.12.0
Reporter: Robert Metzger
https://dev.azure.com/rmetzger/Flink/_build/results?buildId=8113&view=logs&j=764762df-f65b-572b-3d5c-65518c777be4&t=8d823410-c7c7-5a4d-68bb-fa7b08da17b9
{code}
[ERROR] Tests run: 14, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 8.905
s <<< FAILURE! - in
org.apache.flink.mesos.runtime.clusterframework.MesosResourceManagerTest
[ERROR]
testWorkerStarted(org.apache.flink.mesos.runtime.clusterframework.MesosResourceManagerTest)
Time elapsed: 0.219 s <<< ERROR!
java.lang.NullPointerException
at
org.apache.flink.runtime.resourcemanager.ResourceManager.sendSlotReport(ResourceManager.java:406)
at
org.apache.flink.mesos.runtime.clusterframework.MesosResourceManagerTest$7.(MesosResourceManagerTest.java:680)
at
org.apache.flink.mesos.runtime.clusterframework.MesosResourceManagerTest.testWorkerStarted(MesosResourceManagerTest.java:656)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18057) SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: expected:<3> but was:<2>
Robert Metzger created FLINK-18057:
--
Summary:
SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: expected:<3> but
was:<2>
Key: FLINK-18057
URL: https://issues.apache.org/jira/browse/FLINK-18057
Project: Flink
Issue Type: Bug
Components: Runtime / Network
Affects Versions: 1.12.0
Reporter: Robert Metzger
CI:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2524&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=05b74a19-4ee4-5036-c46f-ada307df6cf0
{code}
java.lang.AssertionError: expected:<3> but was:<2>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:834)
at org.junit.Assert.assertEquals(Assert.java:645)
at org.junit.Assert.assertEquals(Assert.java:631)
at
org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest.testConcurrentReadStateAndProcessAndClose(SingleInputGateTest.java:239)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18056) Hive file sink throws exception when the target in-progress file exists.
Yun Gao created FLINK-18056:
---
Summary: Hive file sink throws exception when the target
in-progress file exists.
Key: FLINK-18056
URL: https://issues.apache.org/jira/browse/FLINK-18056
Project: Flink
Issue Type: Bug
Components: Connectors / FileSystem
Reporter: Yun Gao
Fix For: 1.11.0
Currently after failover or restart, the Hive file sink will try to overwrite
the data since the last checkpoint, however, currently neither the in-progress
file is deleted nor hive uses the overwritten mode, thus an exception occurs
after restarting:
{code:java}
org.apache.flink.connectors.hive.FlinkHiveException:
org.apache.flink.table.catalog.exceptions.CatalogException: Failed to create
Hive RecordWriter
at
org.apache.flink.connectors.hive.write.HiveWriterFactory.createRecordWriter(HiveWriterFactory.java:159)
at
org.apache.flink.connectors.hive.write.HiveBulkWriterFactory.create(HiveBulkWriterFactory.java:47)
at
org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriter$HadoopPathBasedBucketWriter.openNewInProgressFile(HadoopPathBasedPartFileWriter.java:234)
at
org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriter$HadoopPathBasedBucketWriter.openNewInProgressFile(HadoopPathBasedPartFileWriter.java:207)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.rollPartFile(Bucket.java:209)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Bucket.write(Bucket.java:200)
at
org.apache.flink.streaming.api.functions.sink.filesystem.Buckets.onElement(Buckets.java:284)
at
org.apache.flink.streaming.api.functions.sink.filesystem.StreamingFileSinkHelper.onElement(StreamingFileSinkHelper.java:104)
at
org.apache.flink.table.filesystem.stream.StreamingFileWriter.processElement(StreamingFileWriter.java:118)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at StreamExecCalc$16.processElement(Unknown Source)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at
org.apache.flink.table.runtime.operators.wmassigners.WatermarkAssignerOperator.processElement(WatermarkAssignerOperator.java:123)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at StreamExecCalc$2.processElement(Unknown Source)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.pushToOperator(OperatorChain.java:717)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:692)
at
org.apache.flink.streaming.runtime.tasks.OperatorChain$CopyingChainingOutput.collect(OperatorChain.java:672)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$NonTimestampContext.collect(StreamSourceContexts.java:104)
at
org.apache.flink.streaming.api.functions.source.datagen.DataGeneratorSource.run(DataGeneratorSource.java:82)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:201)
Caused by: org.apache.flink.table.catalog.exceptions.CatalogExcep
[jira] [Created] (FLINK-18055) Catalog does not exist in SQL Client
godfrey he created FLINK-18055:
--
Summary: Catalog does not exist in SQL Client
Key: FLINK-18055
URL: https://issues.apache.org/jira/browse/FLINK-18055
Project: Flink
Issue Type: Bug
Components: Table SQL / Client
Affects Versions: 1.11.0
Reporter: godfrey he
Flink SQL> show catalogs;
default_catalog
hive
Flink SQL> use catalog hive;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.table.catalog.exceptions.CatalogException: A catalog with name
[`hive`] does not exist.
The reason is {{SqlCommandParser}} adds {{``}} for catalog name, which is
unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18054) Environment should provide API to resume from savepoint
Sergii Mikhtoniuk created FLINK-18054:
-
Summary: Environment should provide API to resume from savepoint
Key: FLINK-18054
URL: https://issues.apache.org/jira/browse/FLINK-18054
Project: Flink
Issue Type: Improvement
Components: Test Infrastructure
Affects Versions: 1.10.1
Reporter: Sergii Mikhtoniuk
Flink's savepoint API lacks symmetry.
While you can stop the job with savepoint as:
{code}
job.stopWithSavepoint(advanceToEndOfTime, checkpointDir)
{code}
... there is no API to resume from one.
This make it very hard to test stop-resume functionality as it forces you to
run a proper cluster and use CLI.
Workaround:
To use savepoints in my "unit" tests I'm resorting to subclassing local
environment and calling {{streamGraphGenerator.setSavepointRestoreSettings}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18053) Savepoints do not preserve watermarks
Sergii Mikhtoniuk created FLINK-18053:
-
Summary: Savepoints do not preserve watermarks
Key: FLINK-18053
URL: https://issues.apache.org/jira/browse/FLINK-18053
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Table SQL / Runtime
Affects Versions: 1.10.1
Reporter: Sergii Mikhtoniuk
Attachments: 1.csv, 2.csv, MyApp.scala
Flink produces invalid result when streaming SQL aggregation is stopped and
resumed from a savepoint.
*Steps to reproduce:*
1) Create an assembly from the attached file.
This job will be reading CSV files as a stream. Files contain fake stock
tickers which will be aggregated with following tumbling window query:
{code:java}
SELECT
TUMBLE_START(event_time, INTERVAL '1' DAY) as event_time,
symbol as symbol,
min(price) as `min`,
max(price) as `max`
FROM Tickers
GROUP BY TUMBLE(event_time, INTERVAL '1' DAY), symbol
{code}
Stream uses punctuated watermarks with max lateness of 1 day
2) Create two CSV files with fake stock tickers:
{{1.csv}}:
{code:java}
2000-01-01 01:00:00.0,A,10
2000-01-01 01:00:00.0,B,20
2000-01-01 02:00:00.0,A,10
2000-01-01 02:00:00.0,B,21
2000-01-02 01:00:00.0,A,12
2000-01-02 01:00:00.0,B,22
2000-01-02 02:00:00.0,A,13
2000-01-02 02:00:00.0,B,23
2000-01-01 03:00:00.0,A,11 // Late arrival - still above watermark
2000-01-03 01:00:00.0,A,14
2000-01-03 01:00:00.0,B,24
2000-01-03 02:00:00.0,A,15
2000-01-03 02:00:00.0,B,25
{code}
{{2.csv}}:
{code:java}
2000-01-01 04:00:00.0,A,12 // Late arrival - under watermark
2000-01-04 01:00:00.0,A,16 // Next values won't be visible in the result, they
only push watermark up
2000-01-04 01:00:00.0,B,26
2000-01-04 02:00:00.0,A,17
2000-01-04 02:00:00.0,B,27
2000-01-05 01:00:00.0,A,18
2000-01-05 01:00:00.0,B,28
{code}
3) Run the job on the folder containing both files. Observed result is as
expected:
{code:java}
2000-01-01,A,10,11
2000-01-01,B,20,21
2000-01-02,A,12,13
2000-01-02,B,22,23
2000-01-03,A,14,15
2000-01-03,B,24,25
{code}
4) Now run the job with only {{1.csv}} in the directory. Produces still correct:
{code:java}
2000-01-01,A,10,11
2000-01-01,B,20,21
{code}
5) Cancel job with savepoint, move {{2.csv}} into the directory. Restart job
from savepoint. Produces incorrect result:
{code:java}
2000-01-01,A,12,12
2000-01-02,A,12,13
2000-01-02,B,22,23
2000-01-03,A,14,15
2000-01-03,B,24,25
{code}
*Expectation:*
We were not supposed to see {{20[^MyApp.scala][^1.csv]00-01-01,A,12,12}}
record, as it should not have passed the watermark check. This tells me that
Flink did not save the watermark in the savepoint.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18052) Elasticsearch7DynamicSinkITCase.testWritingDocumentsNoPrimaryKey fails with NPE
Robert Metzger created FLINK-18052:
--
Summary:
Elasticsearch7DynamicSinkITCase.testWritingDocumentsNoPrimaryKey fails with NPE
Key: FLINK-18052
URL: https://issues.apache.org/jira/browse/FLINK-18052
Project: Flink
Issue Type: Bug
Components: Connectors / ElasticSearch
Affects Versions: 1.11.0
Reporter: Robert Metzger
Fix For: 1.11.0
CI:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2481&view=logs&j=d44f43ce-542c-597d-bf94-b0718c71e5e8&t=34f486e1-e1e4-5dd2-9c06-bfdd9b9c74a8
{code}
2020-05-30T22:32:52.3703325Z [INFO] Running
org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSinkITCase
2020-05-30T22:33:08.3791670Z [ERROR] Tests run: 3, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 16.006 s <<< FAILURE! - in
org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSinkITCase
2020-05-30T22:33:08.3793500Z [ERROR]
testWritingDocumentsNoPrimaryKey(org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSinkITCase)
Time elapsed: 7.671 s <<< ERROR!
2020-05-30T22:33:08.3794313Z java.lang.ArrayIndexOutOfBoundsException: 0
2020-05-30T22:33:08.3794992Zat
org.elasticsearch.search.SearchHits.getAt(SearchHits.java:157)
2020-05-30T22:33:08.3795921Zat
org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch7DynamicSinkITCase.testWritingDocumentsNoPrimaryKey(Elasticsearch7DynamicSinkITCase.java:226)
2020-05-30T22:33:08.3796778Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-05-30T22:33:08.3797675Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-05-30T22:33:08.3798442Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-05-30T22:33:08.3799083Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-05-30T22:33:08.3799740Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-05-30T22:33:08.3800506Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-05-30T22:33:08.3801236Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-05-30T22:33:08.3801938Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-05-30T22:33:08.3802620Zat
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18051) Fail Maven setup on AZP if download fails
Robert Metzger created FLINK-18051: -- Summary: Fail Maven setup on AZP if download fails Key: FLINK-18051 URL: https://issues.apache.org/jira/browse/FLINK-18051 Project: Flink Issue Type: Bug Components: Build System / Azure Pipelines Affects Versions: 1.12.0 Reporter: Robert Metzger Fix For: 1.12.0 Setup maven task is green even though the install was not a success: https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2481&view=logs&j=c88eea3b-64a0-564d-0031-9fdcd7b8abee&t=7f98ac96-cfb0-5c1a-969b-c2a0e48a2291 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18050) Fix the bug of recycling buffer twice once exception in ChannelStateWriteRequestDispatcher#dispatch
Zhijiang created FLINK-18050: Summary: Fix the bug of recycling buffer twice once exception in ChannelStateWriteRequestDispatcher#dispatch Key: FLINK-18050 URL: https://issues.apache.org/jira/browse/FLINK-18050 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing Affects Versions: 1.11.0 Reporter: Zhijiang Assignee: Zhijiang Fix For: 1.11.0, 1.12.0 During ChannelStateWriteRequestDispatcherImpl#dispatch, `request.cancel(e)` is called to recycle the internal buffer of request once exception happens. But for the case of requesting write output, the buffers would be also finally recycled inside ChannelStateCheckpointWriter#write no matter exceptions or not. So the buffers in request will be recycled twice in the case of exception, which would cause further exceptions in the network shuffle process to reference the same buffer. This bug can be reproduced easily via running UnalignedCheckpointITCase#shouldPerformUnalignedCheckpointOnParallelRemoteChannel. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18049) The Flink kafka consumer job will be interrupted if the upstream kafka producer change the AVRO schema
Zheng Hu created FLINK-18049:
Summary: The Flink kafka consumer job will be interrupted if the
upstream kafka producer change the AVRO schema
Key: FLINK-18049
URL: https://issues.apache.org/jira/browse/FLINK-18049
Project: Flink
Issue Type: Bug
Reporter: Zheng Hu
We have encountered a critical case from online services. we have the data
pipeline: (producer) -> (kafka) -> (flink consumer job), and all those records
are encoded in AVRO format. Once the producer changed the AVRO schema , says
adding an extra column to the existing schema and writing few data into the
Kafka.
Then the downstream flink job crashed with the following stacktrace:
{code}
==WARNING== allocating large
array--thread_id[0x7fccd9c16800]--thread_name[Source: Custom Source
(1/1)]--array_size[1590681120 bytes]--array_length[1590681103 elememts]
os_prio=0 tid=0x7fccd9c16800 nid=0x226c0 runnable
at org.shaded.apache.avro.util.Utf8.setByteLength(Utf8.java:78)
at org.shaded.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:261)
at org.shaded.apache.avro.io.BinaryDecoder.readString(BinaryDecoder.java:272)
at
org.shaded.apache.avro.io.ResolvingDecoder.readString(ResolvingDecoder.java:214)
at
org.shaded.apache.avro.generic.GenericDatumReader.readString(GenericDatumReader.java:412)
at
org.shaded.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:181)
at
org.shaded.apache.avro.specific.SpecificDatumReader.readField(SpecificDatumReader.java:116)
at
org.shaded.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:222)
at
org.shaded.apache.avro.generic.GenericDatumReader.readWithoutConversion(GenericDatumReader.java:175)
at
org.shaded.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:153)
at
org.shaded.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:145)
at
org.apache.flink.formats.avro.AvroRowDeserializationSchema.deserialize(AvroRowDeserializationSchema.java:167)
at
org.apache.flink.formats.avro.AvroRowDeserializationSchema.deserialize(AvroRowDeserializationSchema.java:78)
at
org.apache.flink.streaming.util.serialization.KeyedDeserializationSchemaWrapper.deserialize(KeyedDeserializationSchemaWrapper.java:44)
at
org.apache.flink.streaming.connectors.kafka.internal.Kafka09Fetcher.runFetchLoop(Kafka09Fetcher.java:192)
at
org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:771)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:120)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:74)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:129)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:398)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:736)
at java.lang.Thread.run(Thread.java:834)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[ANNOUNCE] Weekly Community Update 2020/22
Dear community, happy to share this week's community update with an update on the upcoming Apache Flink releases: Flink 1.11 and Flink Stateful Functions 2.1. With the community focused on release testing the dev@ mailing list remains relatively quiet. Flink Development == * [releases] Release Testing for Flink 1.11 is progressing. To follow the testing efforts check out the Flink 1.11 burndown board [1] in the Flink Jira. Stephan proposes to backport FLIP-126 to Flink 1.11 (after the feature freeze) as it is an isolated change and to avoid breaking the newly added source interface again in the next release. [2] * [releases] Apache Flink-shaded 11.0 was released. Flink 1.11 will depend on it. [3] * [releases] Gordon just published the first release candidate for Apache Flink Stateful Functions 2.1. [4] * [savepoints] I started a small discussion about documenting (breaking) backwards compatibility of Apache Flink's savepoint format. [5] [1] https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=364&projectKey=FLINK [2] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Backpoint-FLIP-126-watermarks-integration-with-FLIP-27-tp41897p41999.html [3] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/ANNOUNCE-Apache-Flink-shaded-11-0-released-tp42056.html [4] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/VOTE-Apache-Flink-Stateful-Functions-2-1-0-release-candidate-1-tp42061.html [5] http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Document-Backwards-Compatibility-of-Savepoints-tp41903.html Notable Bugs == * Flink 1.10.1 seemed to have reintroduced an issue with Kerberos-secured MapR environments. [6] [6] https://issues.apache.org/jira/browse/FLINK-18045 Events, Blog Posts, Misc === * Ververica has added a second training to Flink Forward Global (Oct 20) shifting the two conference days back by day. The new dates are 19th/20th Oct for training, and 21st/22nd Oct for conference/talks. [7] Pre-registration is already opened. [8] [7] https://twitter.com/FlinkForward/status/1265281578676166658 [8] https://www.flink-forward.org/global-2020 Cheers, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
[jira] [Created] (FLINK-18048) "--host" option could not take effect for standalone application cluster
Yang Wang created FLINK-18048:
-
Summary: "--host" option could not take effect for standalone
application cluster
Key: FLINK-18048
URL: https://issues.apache.org/jira/browse/FLINK-18048
Project: Flink
Issue Type: Bug
Components: Deployment / Scripts
Affects Versions: 1.11.0, 1.10.2
Reporter: Yang Wang
Fix For: 1.11.0, 1.10.2
When we use the following command to start a Flink application cluster, the
specified hostname could not take effect. The root cause is {{HOST_OPTION}} is
not added to options in
{{StandaloneApplicationClusterConfigurationParserFactory}}. It will be a
critical issue when we deploy Flink on container environment. Because users
usually want to specify a given hostname.
{code:java}
./bin/standalone-job.sh start --host external-hostname --job-classname
org.apache.flink.streaming.examples.join.WindowJoin
{code}
For the old {{StandaloneJobClusterConfigurationParserFactory}}, it has the same
issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: I want to contribute to Apache Flink
Hi Pony, Welcome to the community! Currently, there are no special permissions required to contribute to Flink. Just ping a committer if you want to work on some JIRA ticket and someone will assign the ticket to you. Here is some information about how to contribute to Flink [1]. [1] https://flink.apache.org/contributing/contribute-code.html Best, Yangze Guo On Mon, Jun 1, 2020 at 5:06 PM 以梦为马 wrote: > > Hi Guys, > > I want to contribute to Apache Flink. > Would you please give me the permission as a contributor? > My JIRA ID is pony712.
I want to contribute to Apache Flink
Hi Guys, I want to contribute to Apache Flink. Would you please give me the permission as a contributor? My JIRA ID is pony712.
[jira] [Created] (FLINK-18047) [DDL] ExecutotionEnvironment set the parallelism did not take effect
pengnengsong created FLINK-18047: Summary: [DDL] ExecutotionEnvironment set the parallelism did not take effect Key: FLINK-18047 URL: https://issues.apache.org/jira/browse/FLINK-18047 Project: Flink Issue Type: Improvement Components: Connectors / FileSystem Affects Versions: 1.10.1, 1.10.0 Environment: ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); EnviromentSettings streamSettings = EnviromentSetting.newInstance().useBlinkPlanner().inStreamingMode().build(); TableEnviroment tEnv = TableEnviroment.create(streamSettings); env.setParallelism(1); CREATE TABLE source( ... )WITH( 'connector.type' = 'filesystem', ... ) Reporter: pengnengsong using filesystem batch write csv data to file, we set parallelism to 1, but output 4 files. There is no such problem when writing with streams. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[VOTE] Apache Flink Stateful Functions 2.1.0, release candidate #1
Hi everyone, Please review and vote on the *release candidate #1* for the version 2.1.0 of Apache Flink Stateful Functions, as follows: [ ] +1, Approve the release [ ] -1, Do not approve the release (please provide specific comments) ***Testing Guideline*** You can find here [1] a page in the project wiki on instructions for testing. To cast a vote, it is not necessary to perform all listed checks, but please mention which checks you have performed when voting. ***Release Overview*** As an overview, the release consists of the following: a) Stateful Functions canonical source distribution, to be deployed to the release repository at dist.apache.org b) Stateful Functions Python SDK distributions to be deployed to PyPI c) Maven artifacts to be deployed to the Maven Central Repository ***Staging Areas to Review*** The staging areas containing the above mentioned artifacts are as follows, for your review: * All artifacts for a) and b) can be found in the corresponding dev repository at dist.apache.org [2] * All artifacts for c) can be found at the Apache Nexus Repository [3] All artifacts are singed with the key 1C1E2394D3194E1944613488F320986D35C33D6A [4] Other links for your review: * JIRA release notes [5] * source code tag "release-2.1.0-rc1" [6] [7] ***Vote Duration*** The vote will be open for at least 72 hours *(target end date is Wednesday, Jun. 3rd).* It is adopted by majority approval, with at least 3 PMC affirmative votes. Thanks, Gordon [1] https://cwiki.apache.org/confluence/display/FLINK/Verifying+a+Flink+Stateful+Functions+Release [2] https://dist.apache.org/repos/dist/dev/flink/flink-statefun-2.1.0-rc1/ [3] https://repository.apache.org/content/repositories/orgapacheflink-1373/ [4] https://dist.apache.org/repos/dist/release/flink/KEYS [5] https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12347861 [6] https://gitbox.apache.org/repos/asf?p=flink-statefun.git;a=tag;h=a372c69501b8816609f9e90872b1a1e10ab66e8e [7] https://github.com/apache/flink-statefun/tree/release-2.1.0-rc1
[jira] [Created] (FLINK-18046) Decimal column stats not supported for Hive table
Rui Li created FLINK-18046: -- Summary: Decimal column stats not supported for Hive table Key: FLINK-18046 URL: https://issues.apache.org/jira/browse/FLINK-18046 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: Rui Li Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18045) Newest version reintroduced a bug causing not working on secured MapR
Bartosz Krasiński created FLINK-18045:
-
Summary: Newest version reintroduced a bug causing not working on
secured MapR
Key: FLINK-18045
URL: https://issues.apache.org/jira/browse/FLINK-18045
Project: Flink
Issue Type: Bug
Components: Deployment / YARN
Affects Versions: 1.10.1
Reporter: Bartosz Krasiński
I was not able to run Flink 1.10.1 on YARN on a a secured MapR cluster, but the
previous version (1.10.0) works fine.
After some investigation it looks like during some refactoring, checking if the
enabled security method is kerberos was removed, effectively reintroducing
https://issues.apache.org/jira/browse/FLINK-5949
Refactoring commit:
[https://github.com/apache/flink/commit/8751e69037d8a9b1756b75eed62a368c3ef29137]
My proposal would be to bring back the kerberos check:
{code:java}
loginUser.getAuthenticationMethod() ==
UserGroupInformation.AuthenticationMethod.KERBEROS
{code}
and add an unit test for that case to prevent it from happening again
I'm happy to prepare a PR after reaching consensus
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
