[jira] [Closed] (FLINK-16158) SqlClient showdown when executing update if not start cluster
[
https://issues.apache.org/jira/browse/FLINK-16158?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed FLINK-16158.
--
Resolution: Duplicate
> SqlClient showdown when executing update if not start cluster
> -
>
> Key: FLINK-16158
> URL: https://issues.apache.org/jira/browse/FLINK-16158
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: hailong wang
>Priority: Minor
> Fix For: 1.11.0
>
>
> If not start cluster, we execute update just as follow:
> {code:java}
> insert into sinks Select product from sources;
> {code}
> It will throw error:
> {code:java}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Unexpected exception. This is a bug. Please consider filing an
> issue.Exception in thread "main"
> org.apache.flink.table.client.SqlClientException: Unexpected exception. This
> is a bug. Please consider filing an issue. at
> org.apache.flink.table.client.SqlClient.main(SqlClient.java:190)Caused by:
> java.lang.RuntimeException: Error running SQL job. at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:606)
> at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdate(LocalExecutor.java:527)
> at
> org.apache.flink.table.client.cli.CliClient.callInsert(CliClient.java:537) at
> org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:299)
> at java.util.Optional.ifPresent(Optional.java:159) at
> org.apache.flink.table.client.cli.CliClient.open(CliClient.java:200) at
> org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:125) at
> org.apache.flink.table.client.SqlClient.start(SqlClient.java:104) at
> org.apache.flink.table.client.SqlClient.main(SqlClient.java:178)Caused by:
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobSubmissionException: Failed to submit
> JobGraph. at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1895) at
> org.apache.flink.table.client.gateway.local.LocalExecutor.executeUpdateInternal(LocalExecutor.java:603)
> ... 8 moreCaused by: org.apache.flink.runtime.client.JobSubmissionException:
> Failed to submit JobGraph. at
> org.apache.flink.client.program.rest.RestClusterClient.lambda$submitJob$7(RestClusterClient.java:359)
> at
> java.util.concurrent.CompletableFuture.uniExceptionally(CompletableFuture.java:870)
> at
> java.util.concurrent.CompletableFuture$UniExceptionally.tryFire(CompletableFuture.java:852)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils.lambda$retryOperationWithDelay$8(FutureUtils.java:287)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$1(RestClient.java:342)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:500)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:493)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:472)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:413)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:538)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:531)
> at
> org.apache.flink.shaded.netty4.io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:111)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.fulfillConnectPromise(AbstractNioChannel.java:323)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:685)
> at
> org.apache.flink.shaded.netty4.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:632)
> at
> or
[GitHub] [flink] flinkbot commented on pull request #12264: [FLINK-17668][netty] Release partitions asynchronously
flinkbot commented on pull request #12264: URL: https://github.com/apache/flink/pull/12264#issuecomment-631339639 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 19c5f57b94cc56b70002031618c32d9e6f68effb (Wed May 20 08:56:08 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-15333) Using FlinkCostFactory rather than RelOptCostImpl in blink planner
[
https://issues.apache.org/jira/browse/FLINK-15333?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed FLINK-15333.
--
Resolution: Not A Problem
> Using FlinkCostFactory rather than RelOptCostImpl in blink planner
> ---
>
> Key: FLINK-15333
> URL: https://issues.apache.org/jira/browse/FLINK-15333
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.0
>Reporter: Leonard Xu
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.11.0
>
>
> When I test FLINK SQL in flink 1.10, I found an exception which is a bug and
> need to fix.
> {code:java}
> // Some comments here
> Exception in thread "main" java.lang.ClassCastException:
> org.apache.calcite.plan.RelOptCostImpl$Factory cannot be cast to
> org.apache.flink.table.planner.plan.cost.FlinkCostFactory
> at
> org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalExchange.computeSelfCost(CommonPhysicalExchange.scala:50)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdNonCumulativeCost.getNonCumulativeCost(FlinkRelMdNonCumulativeCost.scala:41)
> at
> GeneratedMetadataHandler_NonCumulativeCost.getNonCumulativeCost_$(Unknown
> Source)
> at
> GeneratedMetadataHandler_NonCumulativeCost.getNonCumulativeCost(Unknown
> Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getNonCumulativeCost(RelMetadataQuery.java:301)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost.getCumulativeCost(FlinkRelMdCumulativeCost.scala:38)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost_$(Unknown
> Source)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost(Unknown
> Source)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost_$(Unknown
> Source)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost(Unknown
> Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getCumulativeCost(RelMetadataQuery.java:282)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost$$anonfun$getCumulativeCost$1.apply(FlinkRelMdCumulativeCost.scala:41)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost$$anonfun$getCumulativeCost$1.apply(FlinkRelMdCumulativeCost.scala:40)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost.getCumulativeCost(FlinkRelMdCumulativeCost.scala:39)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost_$(Unknown
> Source)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost(Unknown
> Source)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost_$(Unknown
> Source)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost(Unknown
> Source)
> at
> org.apache.calcite.rel.metadata.RelMetadataQuery.getCumulativeCost(RelMetadataQuery.java:282)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost$$anonfun$getCumulativeCost$1.apply(FlinkRelMdCumulativeCost.scala:41)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost$$anonfun$getCumulativeCost$1.apply(FlinkRelMdCumulativeCost.scala:40)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.metadata.FlinkRelMdCumulativeCost.getCumulativeCost(FlinkRelMdCumulativeCost.scala:39)
> at GeneratedMetadataHandler_CumulativeCost.getCumulativeCost_$(Un
[GitHub] [flink] zentol opened a new pull request #12264: [FLINK-17668][netty] Release partitions asynchronously
zentol opened a new pull request #12264: URL: https://github.com/apache/flink/pull/12264 With this PR the local release of partitions in the `NettyShuffleEnvironment` is executed asynchronously. This primarily includes the deletion of files. We have seen instances where this operation was taking a long time, blocking the TaskExecutor main thread. For this we are re-using the existing `ioExecutor` from the `TaskExecutor` and also pass it into the `NettyShuffleEnvironment`. This executor was so far only used for handling log file requests. The number of threads of this executor was increased from 1 -> 2. Given that at most 1 thread should be busy handling log file requests (?), we should at least 1 thread processing the partition releases, which should be sufficient as we so far only used 1 thread as well (a busy one at that). We still want these partitions to be cleaned up in a timely fashion; not for correctness, but to reduce disk usage. There is a new (hidden) option for increasing this thread count, but ideally users should never have to use it. There are ultimately 3 places where we could introduce the asynchronous behavior; in the TaskExecutor, PartitionTracker or ShuffleEnvironment. Doing this in the TaskExecutor would imply doing all operations on the PartitionTracker with the ioExecutor, since the tracker requires single-threaded access. This would add complexity where we want it the least, and could easily lead to bugs since we start switching back and forth between executors. The PartitionTracker is a good candidate; it can selectively use the executor for the release call into the ShuffleEnvironment, and would guard the TaskExecutor against such blocking calls regardless of ShuffleEnvironment implementations. So far the tracker was not aware of any asynchronous stuff however, and there may be value in keeping it that way. For this PR I opted for the ShuffleEnvironment. It seems reasonable that the local release should be a non-blocking operation; any ShuffleEnvironment that communicates to the outside will necessarily have to do some non-blocking operations, and being provided an Executor through the ShuffleEnvironmentContext may make this easier, although it has the downside that the ShuffleEnvironment may over-use it and potentially deny other other usages of the executor. On the flip-side, this implies that we only use the executor when necessary; for a ShuffleEnvironment that implements the local release in a non-blocking fashion the safeguard in the PartitionTracker would just be a waste of resources. One thing left to do is adding a test to ensure a blocking partition release is not blocking the TaskExecutor main thread. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-14875) Blink planner should pass the right ExecutionConfig to the creation of serializer
[
https://issues.apache.org/jira/browse/FLINK-14875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated FLINK-14875:
---
Fix Version/s: (was: 1.11.0)
1.12.0
> Blink planner should pass the right ExecutionConfig to the creation of
> serializer
> -
>
> Key: FLINK-14875
> URL: https://issues.apache.org/jira/browse/FLINK-14875
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jing Zhang
>Priority: Major
> Fix For: 1.12.0
>
> Attachments: ImmutableCollectionKryoDeserializerITCase.java
>
>
> If source contains data which has immutable collection, the exception will be
> thrown out:
> {code:java}
> Caused by: com.esotericsoftware.kryo.KryoException:
> java.lang.UnsupportedOperationException
> Serialization trace:
> logTags_ (com.aliyun.openservices.log.common.Logs$LogGroup)
> mLogGroup (com.aliyun.openservices.log.common.LogGroupData)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:125)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readObjectOrNull(Kryo.java:730)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:113)
> at
> com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:528)
> at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:761)
> at
> org.apache.flink.api.java.typeutils.runtime.kryo.KryoSerializer.deserialize(KryoSerializer.java:315)
> at
> org.apache.flink.api.common.typeutils.base.ListSerializer.deserialize(ListSerializer.java:138)
> at
> org.apache.flink.api.common.typeutils.base.ListSerializer.deserialize(ListSerializer.java:47)
> at
> org.apache.flink.util.InstantiationUtil.deserializeFromByteArray(InstantiationUtil.java:463)
> at
> org.apache.flink.table.dataformat.BinaryRow.getGeneric(BinaryRow.java:440)
> at BaseRowSerializerProjection$52.apply(Unknown Source)
> at BaseRowSerializerProjection$52.apply(Unknown Source)
> at
> org.apache.flink.table.typeutils.BaseRowSerializer.baseRowToBinary(BaseRowSerializer.java:250)
> at
> org.apache.flink.table.typeutils.BaseRowSerializer.serializeToPages(BaseRowSerializer.java:285)
> at
> org.apache.flink.table.typeutils.BaseRowSerializer.serializeToPages(BaseRowSerializer.java:55)
> at
> org.apache.flink.table.runtime.sort.BinaryInMemorySortBuffer.write(BinaryInMemorySortBuffer.java:190)
> at
> org.apache.flink.table.runtime.sort.BinaryExternalSorter.write(BinaryExternalSorter.java:540)
> ... 10 more
> Caused by: java.lang.UnsupportedOperationException
> at
> java.util.Collections$UnmodifiableCollection.add(Collections.java:1055)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:109)
> at
> com.esotericsoftware.kryo.serializers.CollectionSerializer.read(CollectionSerializer.java:22)
> at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:679)
> at
> com.esotericsoftware.kryo.serializers.ObjectField.read(ObjectField.java:106)
> ... 27 more
> {code}
> the exception could also appears in a simple ITCase in attachments.
> I find similar problems in [How to set Unmodifiable collection serializer of
> Kryo in Spark
> code|https://stackoverflow.com/questions/46818293/how-to-set-unmodifiable-collection-serializer-of-kryo-in-spark-code],
> is there any way to set unmodifiable collection serializer of Kryo in at
> present?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12232: [FLINK-15947] Finish moving scala expression DSL to flink-table-api-scala
flinkbot edited a comment on pull request #12232: URL: https://github.com/apache/flink/pull/12232#issuecomment-630355116 ## CI report: * 5925ef19171dea0dd42cd7f13875b7b05598f4b0 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1767) * 6df9602ad51db30a39d5a8c6ed6e750025ff7429 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1919) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
flinkbot edited a comment on pull request #11906: URL: https://github.com/apache/flink/pull/11906#issuecomment-619214462 ## CI report: * 2e339ca93fcf4461ddb3502b49ab34083fc96cf6 UNKNOWN * 0e1a42d1f6f38f2e1e92db036b55a5f54a49402f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1864) * 66afd5253c17fae0a41bc38f41338a69268ca4ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1917) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12096: [FLINK-16074][docs-zh] Translate the Overview page for State & Fault Tolerance into Chinese
flinkbot edited a comment on pull request #12096: URL: https://github.com/apache/flink/pull/12096#issuecomment-627237312 ## CI report: * b414987409b6b4e072ac41798e53ab30d326675a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1815) * d5ca90e68a87b35c5969ef79b099164d850381ff Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1918) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12181: [FLINK-17645][runtime] Fix SafetyNetCloseableRegistry constructor bug.
flinkbot edited a comment on pull request #12181: URL: https://github.com/apache/flink/pull/12181#issuecomment-629344595 ## CI report: * bd9add8e480455265ca95b863601f6608918b334 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1907) * 05e0b2b0379e0b05c62631147b82711c32f11fcb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`
[
https://issues.apache.org/jira/browse/FLINK-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed FLINK-15066.
--
Resolution: Won't Fix
> Cannot run multiple `insert into csvTable values ()`
>
>
> Key: FLINK-15066
> URL: https://issues.apache.org/jira/browse/FLINK-15066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Kurt Young
>Assignee: Jingsong Lee
>Priority: Major
> Fix For: 1.11.0
>
>
> I created a csv table in sql client, and tried to insert some data into this
> table.
> The first insert into success, but the second one failed with exception:
> {code:java}
> // Caused by: java.io.IOException: File or directory /.../xxx.csv already
> exists. Existing files and directories are not overwritten in NO_OVERWRITE
> mode. Use OVERWRITE mode to overwrite existing files and directories.at
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:817)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15399) Join with a LookupableTableSource：java.lang.RuntimeException: while converting XXXX Caused by: java.lang.AssertionError: Field ordinal 26 is invalid for type
[
https://issues.apache.org/jira/browse/FLINK-15399?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed FLINK-15399.
--
Resolution: Fixed
> Join with a LookupableTableSource：java.lang.RuntimeException: while
> converting Caused by: java.lang.AssertionError: Field ordinal 26 is
> invalid for type
> ---
>
> Key: FLINK-15399
> URL: https://issues.apache.org/jira/browse/FLINK-15399
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.9.1
> Environment: jdk1.8.0_211
>Reporter: Rockey Cui
>Priority: Major
> Fix For: 1.11.0
>
> Attachments: JoinTest-1.0-SNAPSHOT.jar
>
>
>
> {code:java}
> //代码占位符
> public static void main(String[] args) throws Exception {
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(env,
> settings);
> env.setParallelism(1);
> DataStreamSource stringDataStreamSource1 = env.fromElements(
> "HA"
> );
> String[] fields1 = new String[]{"ORD_ID", "PS_PARTKEY", "PS_SUPPKEY",
> "PS_AVAILQTY", "PS_SUPPLYCOST", "PS_COMMENT"
> // key
> , "PS_INT", "PS_LONG"
> , "PS_DOUBLE8", "PS_DOUBLE14", "PS_DOUBLE15"
> , "PS_NUMBER1", "PS_NUMBER2", "PS_NUMBER3", "PS_NUMBER4"
> , "PS_DATE", "PS_TIMESTAMP", "PS_DATE_EVENT",
> "PS_TIMESTAMP_EVENT"};
> TypeInformation[] types1 = new TypeInformation[]{Types.STRING,
> Types.INT, Types.LONG, Types.LONG, Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE, Types.DOUBLE
> , Types.LONG, Types.LONG, Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP, Types.SQL_DATE,
> Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation1 = new RowTypeInfo(types1, fields1);
> DataStream stream1 = stringDataStreamSource1.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673356L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation1.getArity());
> }
> }).returns(typeInformation1);
> tableEnv.registerDataStream("FUN_1", stream1, String.join(",",
> typeInformation1.getFieldNames()) + ",PROCTIME.proctime");
> DataStreamSource stringDataStreamSource2 = env.fromElements(
> "HA"
> );
> String[] fields2 = new String[]{"C_NAME", "C_ADDRESS", "C_NATIONKEY"
> // key
> , "C_INT", "C_LONG"
> , "C_DOUBLE8", "C_DOUBLE14"
> , "C_DATE_EVENT", "C_TIMESTAMP_EVENT"};
> TypeInformation[] types2 = new TypeInformation[]{Types.STRING,
> Types.STRING, Types.LONG
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.SQL_DATE, Types.SQL_TIMESTAMP};
> RowTypeInfo typeInformation2 = new RowTypeInfo(types2, fields2);
> DataStream stream2 = stringDataStreamSource2.map(new
> MapFunction() {
> private static final long serialVersionUID = 2349572544179673349L;
> @Override
> public Row map(String s) {
> return new Row(typeInformation2.getArity());
> }
> }).returns(typeInformation2);
> tableEnv.registerDataStream("FUN_2", stream2, String.join(",",
> typeInformation2.getFieldNames()) + ",PROCTIME.proctime");
> MyLookupTableSource tableSource = MyLookupTableSource.newBuilder()
> .withFieldNames(new String[]{
> "S_NAME", "S_ADDRESS", "S_PHONE"
> , "S_ACCTBAL", "S_COMMENT"
> // key
> , "S_INT", "S_LONG"
> , "S_DOUBLE8", "S_DOUBLE14"
> , "S_DOUBLE15", "S_DATE_EVENT", "S_TIMESTAMP_EVENT"})
> .withFieldTypes(new TypeInformation[]{
> Types.STRING, Types.STRING, Types.STRING
> , Types.DOUBLE, Types.STRING
> // key
> , Types.INT, Types.LONG
> , Types.DOUBLE, Types.DOUBLE
> , Types.DOUBLE, Types.SQL_DATE, Types.SQL_TIMESTAMP})
> .build();
> tableEnv.registerTableSource("INFO", tableSource);
> String sql = "SELECT LN(F.PS_INT),LOG(F2.C_INT,1)\n" +
> " FROM (SELECT *\n" +
> " FROM FUN_1 F1\n" +
> " JOIN INFO FOR SYSTEM_TIME AS OF F1.PROCTIME D1\n" +
>
[jira] [Closed] (FLINK-15613) execute sql appear "java.lang.IndexOutOfBoundsException"
[ https://issues.apache.org/jira/browse/FLINK-15613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed FLINK-15613. -- Resolution: Duplicate > execute sql appear "java.lang.IndexOutOfBoundsException" > > > Key: FLINK-15613 > URL: https://issues.apache.org/jira/browse/FLINK-15613 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 > Environment: *The input data is:* > 0.0 > 1004.30 > -34.84 > 1.2345678901234E200 > 1.2345678901234E-200 > *The sql-client conf is:* > execution: > planner: blink > type: batch >Reporter: xiaojin.wy >Priority: Major > Fix For: 1.11.0 > > > *The sql is* : > CREATE TABLE `int8_tbl` ( > q1 bigint, q2 bigint > ) WITH ( > 'connector.path'='/test_join/sources/int8_tbl.csv', > 'format.empty-column-as-null'='true', > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > 'format.type'='csv' > ); > select * from int8_tbl i1 left join (select * from int8_tbl i2 join (select > 123 as x) ss on i2.q1 = x) as i3 on i1.q2 = i3.q2 order by 1, 2; > > *The output after exciting the sql is :* > [ERROR] Could not execute SQL statement. Reason: > Caused by: java.lang.IndexOutOfBoundsException: Index: 1, Size: 1 at > java.util.ArrayList.rangeCheck(ArrayList.java:653) at > java.util.ArrayList.get(ArrayList.java:429) at > org.apache.calcite.sql2rel.SqlToRelConverter$LookupContext.findRel(SqlToRelConverter.java:5300) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookup(SqlToRelConverter.java:4424) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.lookupExp(SqlToRelConverter.java:4369) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:3720) > at > org.apache.calcite.sql2rel.SqlToRelConverter.access$2200(SqlToRelConverter.java:217) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4796) > at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.visit(SqlToRelConverter.java:4092) > at org.apache.calcite.sql.SqlIdentifier.accept(SqlIdentifier.java:317) at > org.apache.calcite.sql2rel.SqlToRelConverter$Blackboard.convertExpression(SqlToRelConverter.java:4656) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectList(SqlToRelConverter.java:3939) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:670) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3181) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2124) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2005) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2085) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:646) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:627) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3181) > at > org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:563) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:148) > at > org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:135) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:522) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:436) > at > org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:154) > at > org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:66) > at > org.apache.flink.table.api.internal.TableEnvironmentImpl.sqlQuery(TableEnvironmentImpl.java:464) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.lambda$createTable$16(LocalExecutor.java:783) > at > org.apache.flink.table.client.gateway.local.ExecutionContext.wrapClassLoader(ExecutionContext.java:231) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.createTable(LocalExecutor.java:783) > ... 9 more -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17819) Yarn error unhelpful when forgetting HADOOP_CLASSPATH
[
https://issues.apache.org/jira/browse/FLINK-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111941#comment-17111941
]
Till Rohrmann commented on FLINK-17819:
---
Thanks for looking into the issue [~kkl0u]. I think improving the error message
will be super helpful for our users.
> Yarn error unhelpful when forgetting HADOOP_CLASSPATH
> -
>
> Key: FLINK-17819
> URL: https://issues.apache.org/jira/browse/FLINK-17819
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Arvid Heise
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: usability
> Fix For: 1.11.0
>
>
> When running
> {code:bash}
> flink run -m yarn-cluster -yjm 1768 -ytm 50072 -ys 32 ...
> {code}
> without some export HADOOP_CLASSPATH, we get the unhelpful message
> {noformat}
> Could not build the program from JAR file: JAR file does not exist: -yjm
> {noformat}
> I'd expect something like
> {noformat}
> yarn-cluster can only be used with exported HADOOP_CLASSPATH, see for
> more information{noformat}
>
> I suggest to load a stub for YarnCluster deployment if the actual
> implementation fails to load, which prints this error when used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17819) Yarn error unhelpful when forgetting HADOOP_CLASSPATH
[
https://issues.apache.org/jira/browse/FLINK-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-17819:
--
Priority: Critical (was: Major)
> Yarn error unhelpful when forgetting HADOOP_CLASSPATH
> -
>
> Key: FLINK-17819
> URL: https://issues.apache.org/jira/browse/FLINK-17819
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Arvid Heise
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: usability
>
> When running
> {code:bash}
> flink run -m yarn-cluster -yjm 1768 -ytm 50072 -ys 32 ...
> {code}
> without some export HADOOP_CLASSPATH, we get the unhelpful message
> {noformat}
> Could not build the program from JAR file: JAR file does not exist: -yjm
> {noformat}
> I'd expect something like
> {noformat}
> yarn-cluster can only be used with exported HADOOP_CLASSPATH, see for
> more information{noformat}
>
> I suggest to load a stub for YarnCluster deployment if the actual
> implementation fails to load, which prints this error when used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17819) Yarn error unhelpful when forgetting HADOOP_CLASSPATH
[
https://issues.apache.org/jira/browse/FLINK-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann updated FLINK-17819:
--
Fix Version/s: 1.11.0
> Yarn error unhelpful when forgetting HADOOP_CLASSPATH
> -
>
> Key: FLINK-17819
> URL: https://issues.apache.org/jira/browse/FLINK-17819
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Arvid Heise
>Assignee: Kostas Kloudas
>Priority: Critical
> Labels: usability
> Fix For: 1.11.0
>
>
> When running
> {code:bash}
> flink run -m yarn-cluster -yjm 1768 -ytm 50072 -ys 32 ...
> {code}
> without some export HADOOP_CLASSPATH, we get the unhelpful message
> {noformat}
> Could not build the program from JAR file: JAR file does not exist: -yjm
> {noformat}
> I'd expect something like
> {noformat}
> yarn-cluster can only be used with exported HADOOP_CLASSPATH, see for
> more information{noformat}
>
> I suggest to load a stub for YarnCluster deployment if the actual
> implementation fails to load, which prints this error when used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15503) FileUploadHandlerTest.testMixedMultipart and FileUploadHandlerTest. testUploadCleanupOnUnknownAttribute failed on Azure
[
https://issues.apache.org/jira/browse/FLINK-15503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111939#comment-17111939
]
Till Rohrmann commented on FLINK-15503:
---
[~rmetzger] could you re-run the tests with FLINK-17725? I think this should
fix the problem.
> FileUploadHandlerTest.testMixedMultipart and FileUploadHandlerTest.
> testUploadCleanupOnUnknownAttribute failed on Azure
> ---
>
> Key: FLINK-15503
> URL: https://issues.apache.org/jira/browse/FLINK-15503
> Project: Flink
> Issue Type: Bug
> Components: Runtime / REST, Tests
>Affects Versions: 1.10.0
>Reporter: Till Rohrmann
>Priority: Critical
> Labels: test-stability
> Fix For: 1.10.0
>
>
> The tests {{FileUploadHandlerTest.testMixedMultipart}} and
> {{FileUploadHandlerTest. testUploadCleanupOnUnknownAttribute}} failed on
> Azure with
> {code}
> 2020-01-07T09:32:06.9840445Z [ERROR]
> testUploadCleanupOnUnknownAttribute(org.apache.flink.runtime.rest.FileUploadHandlerTest)
> Time elapsed: 12.457 s <<< ERROR!
> 2020-01-07T09:32:06.9850865Z java.net.SocketTimeoutException: timeout
> 2020-01-07T09:32:06.9851650Z at
> org.apache.flink.runtime.rest.FileUploadHandlerTest.testUploadCleanupOnUnknownAttribute(FileUploadHandlerTest.java:234)
> 2020-01-07T09:32:06.9852910Z Caused by: java.net.SocketException: Socket
> closed
> 2020-01-07T09:32:06.9853465Z at
> org.apache.flink.runtime.rest.FileUploadHandlerTest.testUploadCleanupOnUnknownAttribute(FileUploadHandlerTest.java:234)
> 2020-01-07T09:32:06.9853855Z
> 2020-01-07T09:32:06.9854362Z [ERROR]
> testMixedMultipart(org.apache.flink.runtime.rest.FileUploadHandlerTest) Time
> elapsed: 10.091 s <<< ERROR!
> 2020-01-07T09:32:06.9855125Z java.net.SocketTimeoutException: Read timed out
> 2020-01-07T09:32:06.9855652Z at
> org.apache.flink.runtime.rest.FileUploadHandlerTest.testMixedMultipart(FileUploadHandlerTest.java:154)
> 2020-01-07T09:32:06.9856034Z
> {code}
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=4159&view=results
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] cshuo commented on pull request #12208: [FLINK-17651][table-planner-blink] DecomposeGroupingSetsRule generat…
cshuo commented on pull request #12208: URL: https://github.com/apache/flink/pull/12208#issuecomment-631334364 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-15397) Streaming and batch has different value in the case of count function
[
https://issues.apache.org/jira/browse/FLINK-15397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen updated FLINK-15397:
---
Fix Version/s: (was: 1.11.0)
> Streaming and batch has different value in the case of count function
> -
>
> Key: FLINK-15397
> URL: https://issues.apache.org/jira/browse/FLINK-15397
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.0
>Reporter: xiaojin.wy
>Priority: Major
>
> *The sql is:*
> CREATE TABLE `testdata` (
> a INT,
> b INT
> ) WITH (
>
> 'connector.path'='/defender_test_data/daily_regression_batch_spark_1.10/test_group_agg/sources/testdata.csv',
> 'format.empty-column-as-null'='true',
> 'format.field-delimiter'='|',
> 'connector.type'='filesystem',
> 'format.derive-schema'='true',
> 'format.type'='csv'
> );
> SELECT COUNT(1) FROM testdata WHERE false;
> If the configuration's type is batch ,the result will be 0, but if the
> configuration is streaming, there will be no value;
> *The configuration is:*
> execution:
> planner: blink
> type: streaming
> *The input data is:*
> {code:java}
> 1|1
> 1|2
> 2|1
> 2|2
> 3|1
> 3|2
> |1
> 3|
> |
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15289) Run sql appear error of "Zero-length character strings have no serializable string representation".
[ https://issues.apache.org/jira/browse/FLINK-15289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen updated FLINK-15289: --- Fix Version/s: (was: 1.11.0) > Run sql appear error of "Zero-length character strings have no serializable > string representation". > --- > > Key: FLINK-15289 > URL: https://issues.apache.org/jira/browse/FLINK-15289 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.0 >Reporter: xiaojin.wy >Priority: Major > > *The sql is:* > CREATE TABLE `INT8_TBL` ( > q1 BIGINT, > q2 BIGINT > ) WITH ( > 'format.field-delimiter'=',', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_batch_postgres_1.10/test_bigint/sources/INT8_TBL.csv', > 'format.type'='csv' > ); > SELECT '' AS five, q1 AS plus, -q1 AS xm FROM INT8_TBL; > *The error detail is :* > 2019-12-17 15:35:07,026 ERROR org.apache.flink.table.client.SqlClient - SQL > Client must stop. Unexpected exception. This is a bug. Please consider filing > an issue. > org.apache.flink.table.api.TableException: Zero-length character strings > have no serializable string representation. > at > org.apache.flink.table.types.logical.CharType.asSerializableString(CharType.java:116) > at > org.apache.flink.table.descriptors.DescriptorProperties.putTableSchema(DescriptorProperties.java:218) > at > org.apache.flink.table.catalog.CatalogTableImpl.toProperties(CatalogTableImpl.java:75) > at > org.apache.flink.table.factories.TableFactoryUtil.findAndCreateTableSink(TableFactoryUtil.java:85) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryAndPersistInternal(LocalExecutor.java:688) > at > org.apache.flink.table.client.gateway.local.LocalExecutor.executeQueryAndPersist(LocalExecutor.java:488) > at org.apache.flink.table.client.cli.CliClient.callSelect(CliClient.java:601) > at > org.apache.flink.table.client.cli.CliClient.callCommand(CliClient.java:385) > at java.util.Optional.ifPresent(Optional.java:159) > at > org.apache.flink.table.client.cli.CliClient.submitSQLFile(CliClient.java:271) > at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:125) > at org.apache.flink.table.client.SqlClient.start(SqlClient.java:104) > at org.apache.flink.table.client.SqlClient.main(SqlClient.java:180) > *The input data is:* > 123,456 > 123,4567890123456789 > 4567890123456789,123 > 4567890123456789,4567890123456789 > 4567890123456789,-4567890123456789 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111903#comment-17111903 ] Kostas Kloudas edited comment on FLINK-17745 at 5/20/20, 8:44 AM: -- [~Echo Lee] Given the discussion here, I opened a discussion in the ML with title "[DISCUSS] Remove dependency shipping through nested jars during job submission". Let's move the discussion there so that the whole community can participate. was (Author: kkl0u): [~Echo Lee] Given the discussion here, I opened a discussion in the ML with title "[DISCUSS] Remove dependency shipping through nested jars during job submission. ". Let's move the discussion there so that the whole community can participate. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Echo Lee >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111903#comment-17111903 ] Kostas Kloudas edited comment on FLINK-17745 at 5/20/20, 8:44 AM: -- [~Echo Lee] Given the discussion here, I opened a discussion in the ML with title "[DISCUSS] Remove dependency shipping through nested jars during job submission. ". Let's move the discussion there so that the whole community can participate. was (Author: kkl0u): [~Echo Lee] Is it ok if we open a discussion in the ML to see also what other members of the community have to say about it? As you also mentioned, this way of submission has some problems related to redundant transfers and it is (so far) a hidden way of submitting jobs. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Echo Lee >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-14982) YARN IT Test Case log config is mistakenly disabled
[ https://issues.apache.org/jira/browse/FLINK-14982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Till Rohrmann updated FLINK-14982: -- Fix Version/s: (was: 1.11.0) > YARN IT Test Case log config is mistakenly disabled > --- > > Key: FLINK-14982 > URL: https://issues.apache.org/jira/browse/FLINK-14982 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.10.0 >Reporter: Zhenqiu Huang >Assignee: Zhenqiu Huang >Priority: Major > > The [FLINK-14630] Make the yarn APPLICATION_LOG_CONFIG_FILE an internal > option changed how log config is shipped in YarnClusterDescritor. Currently, > we need to rely on the yarn.log-config-file to specify which log file to ship > in flink conf. But currently all YARN IT test cases haven't enabled it. It > will cause the IT test to fail catch issue by looking into JM, TM log files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wanglijie95 commented on a change in pull request #12181: [FLINK-17645][runtime] Fix SafetyNetCloseableRegistry constructor bug.
wanglijie95 commented on a change in pull request #12181:
URL: https://github.com/apache/flink/pull/12181#discussion_r427838651
##
File path:
flink-core/src/test/java/org/apache/flink/core/fs/SafetyNetCloseableRegistryTest.java
##
@@ -198,4 +198,29 @@ public void testReaperThreadSpawnAndStop() throws
Exception {
}
Assert.assertFalse(SafetyNetCloseableRegistry.isReaperThreadRunning());
}
+
+ /**
+* Test whether failure to start thread in {@link
SafetyNetCloseableRegistry}
+* constructor can lead to failure of subsequent state check.
+*/
+ @Test
+ public void testReaperThreadStartFailed() throws Exception {
+
+ try {
+ new SafetyNetCloseableRegistry(() -> new
OutOfMemoryReaperThread());
+ } catch (java.lang.OutOfMemoryError error) {
+ }
+
+ // the OOM error will not lead to failure of subsequent
constructor call.
+ SafetyNetCloseableRegistry closeableRegistry = new
SafetyNetCloseableRegistry();
+ closeableRegistry.close();
+ }
+
+ static class OutOfMemoryReaperThread extends
SafetyNetCloseableRegistry.CloseableReaperThread {
Review comment:
done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-17183) the 'create table [if not exists]' syntax is not supported
[
https://issues.apache.org/jira/browse/FLINK-17183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Chen closed FLINK-17183.
--
Resolution: Not A Problem
Close because it is a feature on the road-map.
> the 'create table [if not exists]' syntax is not supported
> --
>
> Key: FLINK-17183
> URL: https://issues.apache.org/jira/browse/FLINK-17183
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / API
>Affects Versions: 1.10.0
>Reporter: muzimusi
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments: create_table_if_not_exists.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The 'create table [if not exists]' syntax is not supported by
> FlinkSqlParserImpl. For example, the following example will throw an error:
>
> {code:java}
> CREATE TABLE IF NOT EXISTS default_catalog.default_database.access_log_hive (
> source_ip VARCHAR,
> target_ip VARCHAR,
> behavior VARCHAR,
> comm VARCHAR,
> ts TIMESTAMP(3)
> ) WITH (
> 'key' = 'value',
> ...
> )
> {code}
> The error message is like this：
> {noformat}
> Caused by: org.apache.flink.table.api.SqlParserException: SQL parse failed.
> Encountered "NOT" at line 1, column 17.
> Was expecting one of:
> {{ }}
> {{ "WITH" ...}}
> {{ "COMMENT" ...}}
> {{ "PARTITIONED" ...}}
> {{ "(" ...}}
> {{ "." ...}}{noformat}
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
wuchong commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427835703
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
String pkName = null;
- while (rs.next()) {
+ while (rs.next()) {
String columnName = rs.getString("COLUMN_NAME");
- pkName = rs.getString("PK_NAME");
+ pkName = rs.getString("PK_NAME"); // all the PK_NAME
should be the same
int keySeq = rs.getInt("KEY_SEQ");
- if (columnsWithIndex == null) {
- columnsWithIndex = new ArrayList<>();
- }
- columnsWithIndex.add(new
AbstractMap.SimpleEntry<>(Integer.valueOf(keySeq), columnName));
+ keySeqColumnName.put(keySeq - 1, columnName); //
KEY_SEQ is 1-based index
}
- if (columnsWithIndex != null) {
- // sort columns by KEY_SEQ
-
columnsWithIndex.sort(Comparator.comparingInt(Map.Entry::getKey));
- List cols =
columnsWithIndex.stream().map(Map.Entry::getValue).collect(Collectors.toList());
- return UniqueConstraint.primaryKey(pkName, cols);
+ List pkFields = Arrays.asList(new
String[keySeqColumnName.size()]); // initialize size
+ keySeqColumnName.forEach(pkFields::set);
+ if (!pkFields.isEmpty()) {
+ if (pkName == null) {
+ // PK_NAME maybe null according to the javadoc,
+ // generate an unique name for the primary key
+ pkName = "pk_" + String.join("_", pkFields);
+ }
+ return Optional.of(UniqueConstraint.primaryKey(pkName,
pkFields));
+ } else {
Review comment:
Yes... this is a guideline to return Optional instead of null unless
there is a performance concern.
> Always use Optional to return nullable values in the API/public methods
except the case of a proven performance concern.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
wuchong commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427835703
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
String pkName = null;
- while (rs.next()) {
+ while (rs.next()) {
String columnName = rs.getString("COLUMN_NAME");
- pkName = rs.getString("PK_NAME");
+ pkName = rs.getString("PK_NAME"); // all the PK_NAME
should be the same
int keySeq = rs.getInt("KEY_SEQ");
- if (columnsWithIndex == null) {
- columnsWithIndex = new ArrayList<>();
- }
- columnsWithIndex.add(new
AbstractMap.SimpleEntry<>(Integer.valueOf(keySeq), columnName));
+ keySeqColumnName.put(keySeq - 1, columnName); //
KEY_SEQ is 1-based index
}
- if (columnsWithIndex != null) {
- // sort columns by KEY_SEQ
-
columnsWithIndex.sort(Comparator.comparingInt(Map.Entry::getKey));
- List cols =
columnsWithIndex.stream().map(Map.Entry::getValue).collect(Collectors.toList());
- return UniqueConstraint.primaryKey(pkName, cols);
+ List pkFields = Arrays.asList(new
String[keySeqColumnName.size()]); // initialize size
+ keySeqColumnName.forEach(pkFields::set);
+ if (!pkFields.isEmpty()) {
+ if (pkName == null) {
+ // PK_NAME maybe null according to the javadoc,
+ // generate an unique name for the primary key
+ pkName = "pk_" + String.join("_", pkFields);
+ }
+ return Optional.of(UniqueConstraint.primaryKey(pkName,
pkFields));
+ } else {
Review comment:
Yes... this is a guideline to return Optional instead of null unless
there is a performance concern:
https://flink.apache.org/contributing/code-style-and-quality-java.html#java-optional
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
wuchong commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427834480
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
String pkName = null;
- while (rs.next()) {
+ while (rs.next()) {
String columnName = rs.getString("COLUMN_NAME");
- pkName = rs.getString("PK_NAME");
+ pkName = rs.getString("PK_NAME"); // all the PK_NAME
should be the same
int keySeq = rs.getInt("KEY_SEQ");
- if (columnsWithIndex == null) {
- columnsWithIndex = new ArrayList<>();
- }
- columnsWithIndex.add(new
AbstractMap.SimpleEntry<>(Integer.valueOf(keySeq), columnName));
+ keySeqColumnName.put(keySeq - 1, columnName); //
KEY_SEQ is 1-based index
}
- if (columnsWithIndex != null) {
- // sort columns by KEY_SEQ
-
columnsWithIndex.sort(Comparator.comparingInt(Map.Entry::getKey));
- List cols =
columnsWithIndex.stream().map(Map.Entry::getValue).collect(Collectors.toList());
- return UniqueConstraint.primaryKey(pkName, cols);
+ List pkFields = Arrays.asList(new
String[keySeqColumnName.size()]); // initialize size
+ keySeqColumnName.forEach(pkFields::set);
Review comment:
I stored the index as the key in the map, so we can simply call the
`set()` by index on the `List`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
wuchong commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427833904
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
Review comment:
The modern GC algorithms are actually optimized for creating many many
small objects that are short lived, that is basically the 99% heuristic for
Java objects in every program. So I think the GC overhead is very low, besides
it's in the client side. I think it's fine to early initialize the map.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #12232: [FLINK-15947] Finish moving scala expression DSL to flink-table-api-scala
dawidwys commented on pull request #12232: URL: https://github.com/apache/flink/pull/12232#issuecomment-631324483 Thank you all for joining the discussion! Thanks @twalthr for the review. I'll run the tests and after that I will merge that change. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17800) RocksDB optimizeForPointLookup results in missing time windows
[
https://issues.apache.org/jira/browse/FLINK-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li reassigned FLINK-17800:
-
Fix Version/s: 1.11.0
Assignee: Yun Tang
Priority: Critical (was: Major)
> RocksDB optimizeForPointLookup results in missing time windows
> --
>
> Key: FLINK-17800
> URL: https://issues.apache.org/jira/browse/FLINK-17800
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Yordan Pavlov
>Assignee: Yun Tang
>Priority: Critical
> Fix For: 1.11.0
>
> Attachments: MissingWindows.scala
>
>
> +My Setup:+
> We have been using the _RocksDb_ option of _optimizeForPointLookup_ and
> running version 1.7 for years. Upon upgrading to Flink 1.10 we started
> receiving a strange behavior of missing time windows on a streaming Flink
> job. For the purpose of testing I experimented with previous Flink version
> and (1.8, 1.9, 1.9.3) and non of them showed the problem
>
> A sample of the code demonstrating the problem is here:
> {code:java}
> val datastream = env
> .addSource(KafkaSource.keyedElements(config.kafkaElements,
> List(config.kafkaBootstrapServer)))
> val result = datastream
> .keyBy( _ => 1)
> .timeWindow(Time.milliseconds(1))
> .print()
> {code}
>
>
> The source consists of 3 streams (being either 3 Kafka partitions or 3 Kafka
> topics), the elements in each of the streams are separately increasing. The
> elements generate increasing timestamps using an event time and start from 1,
> increasing by 1. The first partitions would consist of timestamps 1, 2, 10,
> 15..., the second of 4, 5, 6, 11..., the third of 3, 7, 8, 9...
>
> +What I observe:+
> The time windows would open as I expect for the first 127 timestamps. Then
> there would be a huge gap with no opened windows, if the source has many
> elements, then next open window would be having a timestamp in the thousands.
> A gap of hundred of elements would be created with what appear to be 'lost'
> elements. Those elements are not reported as late (if tested with the
> ._sideOutputLateData_ operator). The way we have been using the option is by
> setting in inside the config like so:
> ??etherbi.rocksDB.columnOptions.optimizeForPointLookup=268435456??
> We have been using it for performance reasons as we have huge RocksDB state
> backend.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17625) Fix ArrayIndexOutOfBoundsException in AppendOnlyTopNFunction
[
https://issues.apache.org/jira/browse/FLINK-17625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111922#comment-17111922
]
Danny Chen commented on FLINK-17625:
Hi, [~lsy], are you working on this ?
> Fix ArrayIndexOutOfBoundsException in AppendOnlyTopNFunction
>
>
> Key: FLINK-17625
> URL: https://issues.apache.org/jira/browse/FLINK-17625
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: dalongliu
>Priority: Major
> Fix For: 1.11.0, 1.10.2
>
>
> This is reported in user mailing list:
> http://apache-flink.147419.n8.nabble.com/sql-topN-ArrayIndexOutOfBoundsException-td3008.html
> We should check list is not empty before removing the last element in
> {{AppendOnlyTopNFunction#processElementWithoutRowNumber}}.
> {code:java}
> ava.lang.ArrayIndexOutOfBoundsException: -1
> at java.util.ArrayList.elementData(ArrayList.java:422)
> at java.util.ArrayList.remove(ArrayList.java:499)
> at
> org.apache.flink.table.runtime.operators.rank.AppendOnlyTopNFunction.processElementWithoutRowNumber(AppendOnlyTopNFunction.java:205)
> at
> org.apache.flink.table.runtime.operators.rank.AppendOnlyTopNFunction.processElement(AppendOnlyTopNFunction.java:120)
> at
> org.apache.flink.table.runtime.operators.rank.AppendOnlyTopNFunction.processElement(AppendOnlyTopNFunction.java:46)
> at
> org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:85)
> at
> org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:173)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:151)
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:128)
> at
> org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:487)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wanglijie95 commented on a change in pull request #12181: [FLINK-17645][runtime] Fix SafetyNetCloseableRegistry constructor bug.
wanglijie95 commented on a change in pull request #12181:
URL: https://github.com/apache/flink/pull/12181#discussion_r427831205
##
File path:
flink-core/src/test/java/org/apache/flink/core/fs/SafetyNetCloseableRegistryTest.java
##
@@ -198,4 +198,29 @@ public void testReaperThreadSpawnAndStop() throws
Exception {
}
Assert.assertFalse(SafetyNetCloseableRegistry.isReaperThreadRunning());
}
+
+ /**
+* Test whether failure to start thread in {@link
SafetyNetCloseableRegistry}
+* constructor can lead to failure of subsequent state check.
+*/
+ @Test
+ public void testReaperThreadStartFailed() throws Exception {
+
+ try {
+ new SafetyNetCloseableRegistry(() -> new
OutOfMemoryReaperThread());
+ } catch (java.lang.OutOfMemoryError error) {
+ }
+
Review comment:
OK, I will add the check.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-17073) Slow checkpoint cleanup causing OOMs
[
https://issues.apache.org/jira/browse/FLINK-17073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-17073:
---
Fix Version/s: (was: 1.11.0)
1.12.0
> Slow checkpoint cleanup causing OOMs
>
>
> Key: FLINK-17073
> URL: https://issues.apache.org/jira/browse/FLINK-17073
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.7.3, 1.8.0, 1.9.0, 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Priority: Major
> Fix For: 1.12.0
>
>
> A user reported that he sees a decline in checkpoint cleanup speed when
> upgrading from Flink 1.7.2 to 1.10.0. The result is that a lot of cleanup
> tasks are waiting in the execution queue occupying memory. Ultimately, the JM
> process dies with an OOM.
> Compared to Flink 1.7.2, we introduced a dedicated {{ioExecutor}} which is
> used by the {{HighAvailabilityServices}} (FLINK-11851). Before, we use the
> {{AkkaRpcService}} thread pool which was a {{ForkJoinPool}} with a max
> parallelism of 64. Now it is a {{FixedThreadPool}} with as many threads as
> CPU cores. This change might have caused the decline in completed checkpoint
> discard throughput. This suspicion needs to be validated before trying to fix
> it!
> [1]
> https://lists.apache.org/thread.html/r390e5d775878918edca0b6c9f18de96f828c266a888e34ed30ce8494%40%3Cuser.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12232: [FLINK-15947] Finish moving scala expression DSL to flink-table-api-scala
flinkbot edited a comment on pull request #12232: URL: https://github.com/apache/flink/pull/12232#issuecomment-630355116 ## CI report: * 5925ef19171dea0dd42cd7f13875b7b05598f4b0 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1767) * 6df9602ad51db30a39d5a8c6ed6e750025ff7429 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
flinkbot edited a comment on pull request #12256: URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695 ## CI report: * edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12096: [FLINK-16074][docs-zh] Translate the Overview page for State & Fault Tolerance into Chinese
flinkbot edited a comment on pull request #12096: URL: https://github.com/apache/flink/pull/12096#issuecomment-627237312 ## CI report: * b414987409b6b4e072ac41798e53ab30d326675a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1815) * d5ca90e68a87b35c5969ef79b099164d850381ff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12263: [FLINK-16998][core] Support backwards compatibility for upgraded RowSerializer
flinkbot edited a comment on pull request #12263: URL: https://github.com/apache/flink/pull/12263#issuecomment-631274882 ## CI report: * 5e0f9df0a404a5d88b8762238ec37b903b9f0e4b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1910) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
flinkbot edited a comment on pull request #11906: URL: https://github.com/apache/flink/pull/11906#issuecomment-619214462 ## CI report: * 2e339ca93fcf4461ddb3502b49ab34083fc96cf6 UNKNOWN * 0e1a42d1f6f38f2e1e92db036b55a5f54a49402f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1864) * 66afd5253c17fae0a41bc38f41338a69268ca4ff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17657) jdbc not support read BIGINT UNSIGNED field
[ https://issues.apache.org/jira/browse/FLINK-17657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111920#comment-17111920 ] Danny Chen commented on FLINK-17657: Hi [~zhanglun], would you plan to fix this in release 1.11.0 ? > jdbc not support read BIGINT UNSIGNED field > --- > > Key: FLINK-17657 > URL: https://issues.apache.org/jira/browse/FLINK-17657 > Project: Flink > Issue Type: Bug > Components: Connectors / JDBC, Table SQL / Client >Affects Versions: 1.10.0, 1.10.1 >Reporter: lun zhang >Priority: Major > Fix For: 1.11.0 > > Attachments: env.yaml, excetion.txt > > > I use sql client read mysql table, but I found I can't read a table contain > `BIGINT UNSIGNED` field. It will > Caused by: java.lang.ClassCastException: java.math.BigInteger cannot be cast > to java.lang.Long > > MySQL table: > > create table tb > ( > id BIGINT UNSIGNED auto_increment > primary key, > cooper BIGINT(19) null , > user_sex VARCHAR(2) null > ); > > my env yaml is env.yaml . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17756) Drop table/view shouldn't take effect on each other
[ https://issues.apache.org/jira/browse/FLINK-17756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111916#comment-17111916 ] Danny Chen commented on FLINK-17756: Hi, [~ykt836], i would like to take this issue ~ > Drop table/view shouldn't take effect on each other > --- > > Key: FLINK-17756 > URL: https://issues.apache.org/jira/browse/FLINK-17756 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Reporter: Kurt Young >Priority: Major > Fix For: 1.11.0 > > > Currently "DROP VIEW" can successfully drop a table, and "DROP TABLE" can > successfully a view. We should disable this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17651) DecomposeGroupingSetsRule generates wrong plan when there exist distinct agg and simple agg with same filter
[
https://issues.apache.org/jira/browse/FLINK-17651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111913#comment-17111913
]
Danny Chen commented on FLINK-17651:
cc [~godfreyhe], can you help to review this PR, thanks ~
> DecomposeGroupingSetsRule generates wrong plan when there exist distinct agg
> and simple agg with same filter
>
>
> Key: FLINK-17651
> URL: https://issues.apache.org/jira/browse/FLINK-17651
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.10.1
>Reporter: Shuo Cheng
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Consider adding the following test case to
> org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateITCaseBase. As
> you can see, the actual result is wrong.
>
> {code:java}
> @Test
> def testSimpleAndDistinctAggWithCommonFilter(): Unit = {
> val sql =
> """
> |SELECT
> | h,
> | COUNT(1) FILTER(WHERE d > 1),
> | COUNT(1) FILTER(WHERE d < 2),
> | COUNT(DISTINCT e) FILTER(WHERE d > 1)
> |FROM Table5
> |GROUP BY h
> |""".stripMargin
> checkResult(
> sql,
> Seq(
> row(1,4,1,4),
> row(2,7,0,7),
> row(3,3,0,3)
> )
> )
> }
> Results
> == Correct Result == == Actual Result ==
> 1,4,1,41,0,1,4
> 2,7,0,72,0,0,7
> 3,3,0,33,0,0,3
> {code}
> The problem lies in `DecomposeGroupingSetsRule`, which omits filter arg of
> aggregate call when doing some processing.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] fpompermaier commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
fpompermaier commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427826635
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
String pkName = null;
- while (rs.next()) {
+ while (rs.next()) {
String columnName = rs.getString("COLUMN_NAME");
- pkName = rs.getString("PK_NAME");
+ pkName = rs.getString("PK_NAME"); // all the PK_NAME
should be the same
int keySeq = rs.getInt("KEY_SEQ");
- if (columnsWithIndex == null) {
- columnsWithIndex = new ArrayList<>();
- }
- columnsWithIndex.add(new
AbstractMap.SimpleEntry<>(Integer.valueOf(keySeq), columnName));
+ keySeqColumnName.put(keySeq - 1, columnName); //
KEY_SEQ is 1-based index
}
- if (columnsWithIndex != null) {
- // sort columns by KEY_SEQ
-
columnsWithIndex.sort(Comparator.comparingInt(Map.Entry::getKey));
- List cols =
columnsWithIndex.stream().map(Map.Entry::getValue).collect(Collectors.toList());
- return UniqueConstraint.primaryKey(pkName, cols);
+ List pkFields = Arrays.asList(new
String[keySeqColumnName.size()]); // initialize size
+ keySeqColumnName.forEach(pkFields::set);
+ if (!pkFields.isEmpty()) {
+ if (pkName == null) {
+ // PK_NAME maybe null according to the javadoc,
+ // generate an unique name for the primary key
+ pkName = "pk_" + String.join("_", pkFields);
+ }
+ return Optional.of(UniqueConstraint.primaryKey(pkName,
pkFields));
+ } else {
Review comment:
This else could be removed, we could keep just the return statement..or
do the code style guidelines to proceed in this way?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fpompermaier commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
fpompermaier commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427825676
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
String pkName = null;
- while (rs.next()) {
+ while (rs.next()) {
String columnName = rs.getString("COLUMN_NAME");
- pkName = rs.getString("PK_NAME");
+ pkName = rs.getString("PK_NAME"); // all the PK_NAME
should be the same
int keySeq = rs.getInt("KEY_SEQ");
- if (columnsWithIndex == null) {
- columnsWithIndex = new ArrayList<>();
- }
- columnsWithIndex.add(new
AbstractMap.SimpleEntry<>(Integer.valueOf(keySeq), columnName));
+ keySeqColumnName.put(keySeq - 1, columnName); //
KEY_SEQ is 1-based index
}
- if (columnsWithIndex != null) {
- // sort columns by KEY_SEQ
-
columnsWithIndex.sort(Comparator.comparingInt(Map.Entry::getKey));
- List cols =
columnsWithIndex.stream().map(Map.Entry::getValue).collect(Collectors.toList());
- return UniqueConstraint.primaryKey(pkName, cols);
+ List pkFields = Arrays.asList(new
String[keySeqColumnName.size()]); // initialize size
+ keySeqColumnName.forEach(pkFields::set);
Review comment:
Are you sure that foreach is ordered by the entry key?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fpompermaier commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
fpompermaier commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427825676
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
String pkName = null;
- while (rs.next()) {
+ while (rs.next()) {
String columnName = rs.getString("COLUMN_NAME");
- pkName = rs.getString("PK_NAME");
+ pkName = rs.getString("PK_NAME"); // all the PK_NAME
should be the same
int keySeq = rs.getInt("KEY_SEQ");
- if (columnsWithIndex == null) {
- columnsWithIndex = new ArrayList<>();
- }
- columnsWithIndex.add(new
AbstractMap.SimpleEntry<>(Integer.valueOf(keySeq), columnName));
+ keySeqColumnName.put(keySeq - 1, columnName); //
KEY_SEQ is 1-based index
}
- if (columnsWithIndex != null) {
- // sort columns by KEY_SEQ
-
columnsWithIndex.sort(Comparator.comparingInt(Map.Entry::getKey));
- List cols =
columnsWithIndex.stream().map(Map.Entry::getValue).collect(Collectors.toList());
- return UniqueConstraint.primaryKey(pkName, cols);
+ List pkFields = Arrays.asList(new
String[keySeqColumnName.size()]); // initialize size
+ keySeqColumnName.forEach(pkFields::set);
Review comment:
I Are you sure that foreach is ordered by the entry key?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17174) Conversion to relational algebra failed to preserve datatypes when using Table API but succeed when using SQL query
[ https://issues.apache.org/jira/browse/FLINK-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111907#comment-17111907 ] Danny Chen commented on FLINK-17174: This is an duplicate issue of FLINK-16160, so i would close it, feel free to re-open it when the bug still exists. > Conversion to relational algebra failed to preserve datatypes when using > Table API but succeed when using SQL query > --- > > Key: FLINK-17174 > URL: https://issues.apache.org/jira/browse/FLINK-17174 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Yiwen Tu >Priority: Critical > Fix For: 1.11.0 > > Attachments: image-2020-04-16-12-54-19-574.png, > image-2020-04-16-13-08-33-752.png, image-2020-04-16-13-08-59-149.png, > image-2020-04-16-13-12-32-212.png > > > I read data from kafka source and the source table has the fields of 'ts' > which has the row time attributes, after that I add an column whose type is > int and default value is 0. After that, the kafka source will be join with a > table. And the code is below: > !image-2020-04-16-13-08-33-752.png|width=2329,height=253! > The Error is : > !image-2020-04-16-13-08-59-149.png! > But when I use SqlQuery, it works: > !image-2020-04-16-13-12-32-212.png|width=1230,height=47! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17174) Conversion to relational algebra failed to preserve datatypes when using Table API but succeed when using SQL query
[ https://issues.apache.org/jira/browse/FLINK-17174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Danny Chen closed FLINK-17174. -- Resolution: Duplicate > Conversion to relational algebra failed to preserve datatypes when using > Table API but succeed when using SQL query > --- > > Key: FLINK-17174 > URL: https://issues.apache.org/jira/browse/FLINK-17174 > Project: Flink > Issue Type: Bug > Components: Table SQL / API, Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: Yiwen Tu >Priority: Critical > Fix For: 1.11.0 > > Attachments: image-2020-04-16-12-54-19-574.png, > image-2020-04-16-13-08-33-752.png, image-2020-04-16-13-08-59-149.png, > image-2020-04-16-13-12-32-212.png > > > I read data from kafka source and the source table has the fields of 'ts' > which has the row time attributes, after that I add an column whose type is > int and default value is 0. After that, the kafka source will be join with a > table. And the code is below: > !image-2020-04-16-13-08-33-752.png|width=2329,height=253! > The Error is : > !image-2020-04-16-13-08-59-149.png! > But when I use SqlQuery, it works: > !image-2020-04-16-13-12-32-212.png|width=1230,height=47! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17820) Memory threshold is ignored for channel state
[
https://issues.apache.org/jira/browse/FLINK-17820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-17820:
---
Priority: Critical (was: Major)
> Memory threshold is ignored for channel state
> -
>
> Key: FLINK-17820
> URL: https://issues.apache.org/jira/browse/FLINK-17820
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Runtime / Task
>Affects Versions: 1.11.0
>Reporter: Roman Khachatryan
>Assignee: Roman Khachatryan
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Config parameter state.backend.fs.memory-threshold is ignored for channel
> state. Causing each subtask to have a file per checkpoint. Regardless of the
> size of channel state (of this subtask).
> This also causes slow cleanup and delays the next checkpoint.
>
> The problem is that {{ChannelStateCheckpointWriter.finishWriteAndResult}}
> calls flush(); which actually flushes the data on disk.
>
> From FSDataOutputStream.flush Javadoc:
> A completed flush does not mean that the data is necessarily persistent. Data
> persistence can is only assumed after calls to close() or sync().
>
> Possible solutions:
> 1. not to flush in {{ChannelStateCheckpointWriter.finishWriteAndResult (which
> can lead to data loss in a wrapping stream).}}
> {{2. change }}{{FsCheckpointStateOutputStream.flush behavior}}
> {{3. wrap }}{{FsCheckpointStateOutputStream to prevent flush}}{{}}{{}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111903#comment-17111903 ] Kostas Kloudas commented on FLINK-17745: [~Echo Lee] Is it ok if we open a discussion in the ML to see also what other members of the community have to say about it? As you also mentioned, this way of submission has some problems related to redundant transfers and it is (so far) a hidden way of submitting jobs. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Echo Lee >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] fpompermaier commented on a change in pull request #11906: [FLINK-17356][jdbc][postgres] Support PK and Unique constraints
fpompermaier commented on a change in pull request #11906:
URL: https://github.com/apache/flink/pull/11906#discussion_r427823029
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/catalog/AbstractJdbcCatalog.java
##
@@ -126,31 +124,33 @@ public String getBaseUrl() {
// -- retrieve PK constraint --
- protected UniqueConstraint getPrimaryKey(DatabaseMetaData metaData,
String schema, String table) throws SQLException {
+ protected Optional getPrimaryKey(DatabaseMetaData
metaData, String schema, String table) throws SQLException {
// According to the Javadoc of
java.sql.DatabaseMetaData#getPrimaryKeys,
// the returned primary key columns are ordered by COLUMN_NAME,
not by KEY_SEQ.
// We need to sort them based on the KEY_SEQ value.
ResultSet rs = metaData.getPrimaryKeys(null, schema, table);
- List> columnsWithIndex = null;
+ Map keySeqColumnName = new HashMap<>();
Review comment:
I didn't initialize any structure for a table before PK discovery in
order to avoid GC pressure (If you have thousands of tables maybe this could
save some memory). What do you think?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12258: [FLINK-17820][task][checkpointing] Don't flush channel state to disk explicitly
flinkbot edited a comment on pull request #12258: URL: https://github.com/apache/flink/pull/12258#issuecomment-631108764 ## CI report: * 8d01ba80d36c07517d7493cef13d6ab634c01e18 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1909) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12261: [FLINK-17823][network] Resolve the race condition while releasing RemoteInputChannel
flinkbot edited a comment on pull request #12261: URL: https://github.com/apache/flink/pull/12261#issuecomment-631229356 ## CI report: * 26afeb03aa30f84994a8aa85ca2d223d44672067 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1905) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17819) Yarn error unhelpful when forgetting HADOOP_CLASSPATH
[
https://issues.apache.org/jira/browse/FLINK-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas reassigned FLINK-17819:
--
Assignee: Kostas Kloudas
> Yarn error unhelpful when forgetting HADOOP_CLASSPATH
> -
>
> Key: FLINK-17819
> URL: https://issues.apache.org/jira/browse/FLINK-17819
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Arvid Heise
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: usability
>
> When running
> {code:bash}
> flink run -m yarn-cluster -yjm 1768 -ytm 50072 -ys 32 ...
> {code}
> without some export HADOOP_CLASSPATH, we get the unhelpful message
> {noformat}
> Could not build the program from JAR file: JAR file does not exist: -yjm
> {noformat}
> I'd expect something like
> {noformat}
> yarn-cluster can only be used with exported HADOOP_CLASSPATH, see for
> more information{noformat}
>
> I suggest to load a stub for YarnCluster deployment if the actual
> implementation fails to load, which prints this error when used.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-17826) Add missing custom query support on new jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-17826:
---
Assignee: Flavio Pompermaier
> Add missing custom query support on new jdbc connector
> --
>
> Key: FLINK-17826
> URL: https://issues.apache.org/jira/browse/FLINK-17826
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC
>Reporter: Jark Wu
>Assignee: Flavio Pompermaier
>Priority: Major
> Fix For: 1.11.0
>
>
> In FLINK-17361, we added custom query on JDBC tables, but missing to add the
> same ability on new jdbc connector (i.e.
> {{JdbcDynamicTableSourceSinkFactory}}).
> In the new jdbc connector, maybe we should call it {{scan.query}} to keep
> consistent with other scan options, besides we need to make {{"table-name"}}
> optional, but add validation that "table-name" and "scan.query" shouldn't
> both be empty, and "table-name" must not be empty when used as sink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17826) Add missing custom query support on new jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111895#comment-17111895
]
Flavio Pompermaier commented on FLINK-17826:
Ok, it should not be too complicated. Assign the issue to me
> Add missing custom query support on new jdbc connector
> --
>
> Key: FLINK-17826
> URL: https://issues.apache.org/jira/browse/FLINK-17826
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC
>Reporter: Jark Wu
>Assignee: Flavio Pompermaier
>Priority: Major
> Fix For: 1.11.0
>
>
> In FLINK-17361, we added custom query on JDBC tables, but missing to add the
> same ability on new jdbc connector (i.e.
> {{JdbcDynamicTableSourceSinkFactory}}).
> In the new jdbc connector, maybe we should call it {{scan.query}} to keep
> consistent with other scan options, besides we need to make {{"table-name"}}
> optional, but add validation that "table-name" and "scan.query" shouldn't
> both be empty, and "table-name" must not be empty when used as sink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17795) Add an example to show how to leverage GPU resources
[ https://issues.apache.org/jira/browse/FLINK-17795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yangze Guo updated FLINK-17795: --- Priority: Critical (was: Major) > Add an example to show how to leverage GPU resources > > > Key: FLINK-17795 > URL: https://issues.apache.org/jira/browse/FLINK-17795 > Project: Flink > Issue Type: Task > Components: Examples >Reporter: Yangze Guo >Priority: Critical > Fix For: 1.11.0 > > > Add an example to show how to leverage GPU resources. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17826) Add missing custom query support on new jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111894#comment-17111894
]
Jark Wu commented on FLINK-17826:
-
Hi [~f.pompermaier], the options and validation should be added in
{{JdbcDynamicTableSourceSinkFactory}}, besides we may need to update
{{JdbcDynamicTableSource}} just like what you updated on the
{{JdbcTableSource}}.
> Add missing custom query support on new jdbc connector
> --
>
> Key: FLINK-17826
> URL: https://issues.apache.org/jira/browse/FLINK-17826
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.11.0
>
>
> In FLINK-17361, we added custom query on JDBC tables, but missing to add the
> same ability on new jdbc connector (i.e.
> {{JdbcDynamicTableSourceSinkFactory}}).
> In the new jdbc connector, maybe we should call it {{scan.query}} to keep
> consistent with other scan options, besides we need to make {{"table-name"}}
> optional, but add validation that "table-name" and "scan.query" shouldn't
> both be empty, and "table-name" must not be empty when used as sink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17470) Flink task executor process permanently hangs on `flink-daemon.sh stop`, deletes PID file
[
https://issues.apache.org/jira/browse/FLINK-17470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111891#comment-17111891
]
Stephan Ewen commented on FLINK-17470:
--
What JVM version are you using? Maybe there is different behavior in newer JVM
versions?
Nothing changed in The JvmShutdownGuard since many releases, so curious that
this changed reproducibly in your setup.
> Flink task executor process permanently hangs on `flink-daemon.sh stop`,
> deletes PID file
> -
>
> Key: FLINK-17470
> URL: https://issues.apache.org/jira/browse/FLINK-17470
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0
> Environment:
> {code:java}
> $ uname -a
> Linux hostname.local 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux
> $ lsb_release -a
> LSB Version: :core-4.1-amd64:core-4.1-noarch
> Distributor ID: CentOS
> Description: CentOS Linux release 7.7.1908 (Core)
> Release: 7.7.1908
> Codename: Core
> {code}
> Flink version 1.10
>
>Reporter: Hunter Herman
>Priority: Major
> Attachments: flink_jstack.log, flink_mixed_jstack.log
>
>
> Hi Flink team!
> We've attempted to upgrade our flink 1.9 cluster to 1.10, but are
> experiencing reproducible instability on shutdown. Speciically, it appears
> that the `kill` issued in the `stop` case of flink-daemon.sh is causing the
> task executor process to hang permanently. Specifically, the process seems to
> be hanging in the
> `org.apache.flink.runtime.util.JvmShutdownSafeguard$DelayedTerminator.run` in
> a `Thread.sleep()` call. I think this is a bizarre behavior. Also note that
> every thread in the process is BLOCKED. on a `pthread_cond_wait` call. Is
> this an OS level issue? Banging my head on a wall here. See attached stack
> traces for details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-17774) supports all kinds of changes for select result
[ https://issues.apache.org/jira/browse/FLINK-17774?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-17774: --- Assignee: godfrey he > supports all kinds of changes for select result > --- > > Key: FLINK-17774 > URL: https://issues.apache.org/jira/browse/FLINK-17774 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Legacy Planner, Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > [FLINK-17252|https://issues.apache.org/jira/browse/FLINK-17252] has supported > select query, however only append change is supported. because > [FLINK-16998|https://issues.apache.org/jira/browse/FLINK-16998] is not > finished. This issue aims to support all kinds of changes based on > FLINK-16998. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17826) Add missing custom query support on new jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111889#comment-17111889
]
Flavio Pompermaier commented on FLINK-17826:
I hope to find the time within the end of this week. Could you just give me
some hint of what I need to do (which classes should I edit)?
> Add missing custom query support on new jdbc connector
> --
>
> Key: FLINK-17826
> URL: https://issues.apache.org/jira/browse/FLINK-17826
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.11.0
>
>
> In FLINK-17361, we added custom query on JDBC tables, but missing to add the
> same ability on new jdbc connector (i.e.
> {{JdbcDynamicTableSourceSinkFactory}}).
> In the new jdbc connector, maybe we should call it {{scan.query}} to keep
> consistent with other scan options, besides we need to make {{"table-name"}}
> optional, but add validation that "table-name" and "scan.query" shouldn't
> both be empty, and "table-name" must not be empty when used as sink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] dawidwys commented on a change in pull request #12232: [FLINK-15947] Finish moving scala expression DSL to flink-table-api-scala
dawidwys commented on a change in pull request #12232: URL: https://github.com/apache/flink/pull/12232#discussion_r427812311 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/expressions/package.scala ## @@ -21,8 +21,8 @@ package org.apache.flink.table * This package contains the base class of AST nodes and all the expression language AST classes. * Expression trees should not be manually constructed by users. They are implicitly constructed * from the implicit DSL conversions in - * [[org.apache.flink.table.api.scala.ImplicitExpressionConversions]] and - * [[org.apache.flink.table.api.scala.ImplicitExpressionOperations]]. For the Java API, + * [[org.apache.flink.table.api.bridge.scala.ImplicitExpressionConversions]] and + * [[org.apache.flink.table.api.bridge.scala.ImplicitExpressionOperations]]. For the Java API, Review comment: Or shall I just remove this file? I don't see any reason for it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17018) DefaultExecutionSlotAllocator allocates slots in bulks ignoring slot sharing
[ https://issues.apache.org/jira/browse/FLINK-17018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu reassigned FLINK-17018: --- Assignee: Zhu Zhu > DefaultExecutionSlotAllocator allocates slots in bulks ignoring slot sharing > > > Key: FLINK-17018 > URL: https://issues.apache.org/jira/browse/FLINK-17018 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.11.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > The DefaultExecutionSlotAllocator should allocate slot in bulks, that means > the bulks of slot requests will be sent together and fail if any of the > request fails. > Note this is a first step to fully functional bulk slot allocation. Current > limitations would be: > 1. Slot sharing will be ignored > 2. Co-location constraints are not allowed > 3. intra-bulk input location preferences will be ignored -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] twalthr commented on a change in pull request #12232: [FLINK-15947] Finish moving scala expression DSL to flink-table-api-scala
twalthr commented on a change in pull request #12232:
URL: https://github.com/apache/flink/pull/12232#discussion_r427791839
##
File path:
flink-table/flink-table-api-scala/src/main/scala/org/apache/flink/table/api/package.scala
##
@@ -39,7 +37,7 @@ import
org.apache.flink.table.api.{ImplicitExpressionConversions, ImplicitExpres
*
* Please refer to the website documentation about how to construct and run
table programs.
*/
-package object api /* extends ImplicitExpressionConversions */ {
+package object api extends ImplicitExpressionConversions {
// This package object should extend from ImplicitExpressionConversions but
would clash with
Review comment:
update comment
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/expressions/package.scala
##
@@ -21,8 +21,8 @@ package org.apache.flink.table
* This package contains the base class of AST nodes and all the expression
language AST classes.
* Expression trees should not be manually constructed by users. They are
implicitly constructed
* from the implicit DSL conversions in
- * [[org.apache.flink.table.api.scala.ImplicitExpressionConversions]] and
- * [[org.apache.flink.table.api.scala.ImplicitExpressionOperations]]. For the
Java API,
+ * [[org.apache.flink.table.api.bridge.scala.ImplicitExpressionConversions]]
and
+ * [[org.apache.flink.table.api.bridge.scala.ImplicitExpressionOperations]].
For the Java API,
Review comment:
update last sentence
##
File path:
flink-table/flink-table-api-scala/src/main/scala/org/apache/flink/table/api/expressionDsl.scala
##
@@ -33,7 +33,7 @@ import java.sql.{Date, Time, Timestamp}
import java.time.{LocalDate, LocalDateTime, LocalTime}
import java.util.{List => JList, Map => JMap}
Review comment:
nit: Shall we move `ImplicitExpressionConversions` and
`ImplicitExpressionOperations` into dedicated files? Personally, I don't like
`expressionDsl.scala` much. `TableConversions` and `DataStreamConvertions` are
also dedicated files.
##
File path: docs/dev/table/common.md
##
@@ -552,7 +552,9 @@ val revenue = orders
// execute query
{% endhighlight %}
-**Note:** The Scala Table API uses Scala Symbols, which start with a single
tick (`'`) to reference the attributes of a `Table`. The Table API uses Scala
implicits. Make sure to import `org.apache.flink.api.scala._` and
`org.apache.flink.table.api.scala._` in order to use Scala implicit conversions.
+**Note:** The Scala Table API uses Scala Symbols, which start with a single
tick (`'`) to reference the attributes of a `Table`. The Table API uses Scala
implicits. Make sure to import
Review comment:
`The Scala Table API uses Scala Symbols, which start with a single tick
(`'`)`, update?
##
File path:
flink-scala-shell/src/main/scala/org/apache/flink/api/scala/FlinkILoop.scala
##
@@ -23,7 +23,7 @@ import org.apache.flink.api.java.{JarHelper,
ScalaShellEnvironment, ScalaShellSt
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import org.apache.flink.configuration.Configuration
import org.apache.flink.table.api.EnvironmentSettings
-import org.apache.flink.table.api.scala.{BatchTableEnvironment,
StreamTableEnvironment}
+import org.apache.flink.table.api.bridge.scala.{BatchTableEnvironment,
StreamTableEnvironment}
Review comment:
Update the default imports in the lower part of this class.
##
File path:
flink-table/flink-table-api-scala-bridge/src/main/scala/org/apache/flink/table/api/bridge/scala/package.scala
##
@@ -50,7 +50,7 @@ import _root_.scala.language.implicitConversions
* Please refer to the website documentation about how to construct and run
table programs that are
* connected to the DataStream API.
*/
-package object scala extends ImplicitExpressionConversions {
+package object scala {
// This package object should not extend from ImplicitExpressionConversions
but would clash with
Review comment:
update comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17745) PackagedProgram' extractedTempLibraries and jarfiles may be duplicate
[ https://issues.apache.org/jira/browse/FLINK-17745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111881#comment-17111881 ] Kostas Kloudas commented on FLINK-17745: [~Echo Lee] I also think that we can close this issue and open a new one about documenting this way of submitting a job. I already closed the PR although I consider that this way of submission a bit tricky and I would not be opposed to remove it in the future. > PackagedProgram' extractedTempLibraries and jarfiles may be duplicate > - > > Key: FLINK-17745 > URL: https://issues.apache.org/jira/browse/FLINK-17745 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Echo Lee >Assignee: Kostas Kloudas >Priority: Major > Labels: pull-request-available > > When i submit a flink app with a fat jar, PackagedProgram will extracted temp > libraries by the fat jar, and add to pipeline.jars, and the pipeline.jars > contains fat jar and temp libraries. I don't think we should add fat jar to > the pipeline.jars if extractedTempLibraries is not empty. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-17721) AbstractHadoopFileSystemITTest .cleanupDirectoryWithRetry fails with AssertionError
[
https://issues.apache.org/jira/browse/FLINK-17721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-17721:
--
Assignee: Xintong Song
> AbstractHadoopFileSystemITTest .cleanupDirectoryWithRetry fails with
> AssertionError
>
>
> Key: FLINK-17721
> URL: https://issues.apache.org/jira/browse/FLINK-17721
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Tests
>Reporter: Robert Metzger
>Assignee: Xintong Song
>Priority: Critical
> Fix For: 1.11.0
>
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1343&view=logs&j=961f8f81-6b52-53df-09f6-7291a2e4af6a&t=2f99feaa-7a9b-5916-4c1c-5e61f395079e
> {code}
> [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 34.079 s <<< FAILURE! - in
> org.apache.flink.fs.s3hadoop.HadoopS3FileSystemITCase
> [ERROR] org.apache.flink.fs.s3hadoop.HadoopS3FileSystemITCase Time elapsed:
> 21.334 s <<< FAILURE!
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:86)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertFalse(Assert.java:64)
> at org.junit.Assert.assertFalse(Assert.java:74)
> at
> org.apache.flink.runtime.fs.hdfs.AbstractHadoopFileSystemITTest.cleanupDirectoryWithRetry(AbstractHadoopFileSystemITTest.java:162)
> at
> org.apache.flink.runtime.fs.hdfs.AbstractHadoopFileSystemITTest.teardown(AbstractHadoopFileSystemITTest.java:149)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u commented on pull request #12233: [FLINK-17745] Remove nested jar handling from PackagedProgram
kl0u commented on pull request #12233: URL: https://github.com/apache/flink/pull/12233#issuecomment-631300927 This PR is closed unmerged. For details see the discussion on the JIRA issue. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] kl0u closed pull request #12233: [FLINK-17745] Remove nested jar handling from PackagedProgram
kl0u closed pull request #12233: URL: https://github.com/apache/flink/pull/12233 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17351) CheckpointCoordinator and CheckpointFailureManager ignores checkpoint timeouts
[
https://issues.apache.org/jira/browse/FLINK-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-17351:
--
Assignee: Yuan Mei
> CheckpointCoordinator and CheckpointFailureManager ignores checkpoint timeouts
> --
>
> Key: FLINK-17351
> URL: https://issues.apache.org/jira/browse/FLINK-17351
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.2, 1.10.0
>Reporter: Piotr Nowojski
>Assignee: Yuan Mei
>Priority: Critical
> Fix For: 1.11.0
>
>
> As described in point 2:
> https://issues.apache.org/jira/browse/FLINK-17327?focusedCommentId=17090576&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17090576
> (copy of description from above linked comment):
> The logic in how {{CheckpointCoordinator}} handles checkpoint timeouts is
> broken. In your [~qinjunjerry] examples, your job should have failed after
> first checkpoint failure, but checkpoints were time outing on
> CheckpointCoordinator after 5 seconds, before {{FlinkKafkaProducer}} was
> detecting Kafka failure after 2 minutes. Those timeouts were not checked
> against {{setTolerableCheckpointFailureNumber(...)}} limit, so the job was
> keep going with many timed out checkpoints. Now funny thing happens:
> FlinkKafkaProducer detects Kafka failure. Funny thing is that it depends
> where the failure was detected:
> a) on processing record? no problem, job will failover immediately once
> failure is detected (in this example after 2 minutes)
> b) on checkpoint? heh, the failure is reported to {{CheckpointCoordinator}}
> *and gets ignored, as PendingCheckpoint has already been discarded 2 minutes
> ago* :) So theoretically the checkpoints can keep failing forever and the job
> will not restart automatically, unless something else fails.
> Even more funny things can happen if we mix FLINK-17350 . or b) with
> intermittent external system failure. Sink reports an exception, transaction
> was lost/aborted, Sink is in failed state, but if there will be a happy
> coincidence that it manages to accept further records, this exception can be
> lost and all of the records in those failed checkpoints will be lost forever
> as well. In all of the examples that [~qinjunjerry] posted it hasn't
> happened. {{FlinkKafkaProducer}} was not able to recover after the initial
> failure and it was keep throwing exceptions until the job finally failed (but
> much later then it should have). And that's not guaranteed anywhere.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17827) scala-shell.sh should fail early if no mode is specified, or have default logging settings
[ https://issues.apache.org/jira/browse/FLINK-17827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-17827: - Priority: Critical (was: Major) > scala-shell.sh should fail early if no mode is specified, or have default > logging settings > -- > > Key: FLINK-17827 > URL: https://issues.apache.org/jira/browse/FLINK-17827 > Project: Flink > Issue Type: Improvement > Components: Scala Shell >Affects Versions: 1.11.0 >Reporter: Chesnay Schepler >Priority: Critical > > The scala-shell has multiple modes it can run in: local, remote and yarn. > It is mandatory to specify such a mode, but this is only enforced on the > scala side, not in the bash script. > The problem is that the scala-shell script derives the log4j properties from > the mode, and if no mode is set, then the log4j properties are empty. > This leads to a warning from slf4j that no logger was defined and all that. > Either scala-shell.sh should fail early if no mode is specified, or it should > have some default logging settings (e.g., the ones for local/remote). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17827) scala-shell.sh should fail early if no mode is specified, or have default logging settings
Chesnay Schepler created FLINK-17827: Summary: scala-shell.sh should fail early if no mode is specified, or have default logging settings Key: FLINK-17827 URL: https://issues.apache.org/jira/browse/FLINK-17827 Project: Flink Issue Type: Improvement Components: Scala Shell Affects Versions: 1.11.0 Reporter: Chesnay Schepler The scala-shell has multiple modes it can run in: local, remote and yarn. It is mandatory to specify such a mode, but this is only enforced on the scala side, not in the bash script. The problem is that the scala-shell script derives the log4j properties from the mode, and if no mode is set, then the log4j properties are empty. This leads to a warning from slf4j that no logger was defined and all that. Either scala-shell.sh should fail early if no mode is specified, or it should have some default logging settings (e.g., the ones for local/remote). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17096) Minibatch Group Agg support state ttl
[ https://issues.apache.org/jira/browse/FLINK-17096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-17096: --- Affects Version/s: 1.11.0 > Minibatch Group Agg support state ttl > - > > Key: FLINK-17096 > URL: https://issues.apache.org/jira/browse/FLINK-17096 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Affects Versions: 1.9.0, 1.10.0, 1.11.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > > At the moment, MiniBatch Group Agg include Local/Global doesn`t support State > TTL, for streaming job, it will lead to OOM in long time running, so we need > to make state data expire after ttl, the solution is that use incremental > cleanup feature refer to FLINK-16581 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17096) Minibatch Group Agg support state ttl
[ https://issues.apache.org/jira/browse/FLINK-17096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-17096: --- Fix Version/s: (was: 1.11.0) 1.12.0 > Minibatch Group Agg support state ttl > - > > Key: FLINK-17096 > URL: https://issues.apache.org/jira/browse/FLINK-17096 > Project: Flink > Issue Type: New Feature > Components: Table SQL / Runtime >Affects Versions: 1.9.0, 1.10.0 >Reporter: dalongliu >Assignee: dalongliu >Priority: Critical > Labels: pull-request-available > Fix For: 1.12.0 > > > At the moment, MiniBatch Group Agg include Local/Global doesn`t support State > TTL, for streaming job, it will lead to OOM in long time running, so we need > to make state data expire after ttl, the solution is that use incremental > cleanup feature refer to FLINK-16581 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16931) Large _metadata file lead to JobManager not responding when restart
[
https://issues.apache.org/jira/browse/FLINK-16931?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-16931:
---
Fix Version/s: (was: 1.11.0)
1.12.0
> Large _metadata file lead to JobManager not responding when restart
> ---
>
> Key: FLINK-16931
> URL: https://issues.apache.org/jira/browse/FLINK-16931
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing, Runtime / Coordination
>Affects Versions: 1.9.2, 1.10.0, 1.11.0
>Reporter: Lu Niu
>Assignee: Lu Niu
>Priority: Critical
> Fix For: 1.12.0
>
>
> When _metadata file is big, JobManager could never recover from checkpoint.
> It fall into a loop that fetch checkpoint -> JM timeout -> restart. Here is
> related log:
> {code:java}
> 2020-04-01 17:08:25,689 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Recovering checkpoints from ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore - Found
> 3 checkpoints in ZooKeeper.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to fetch 3 checkpoints from storage.
> 2020-04-01 17:08:25,698 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 50.
> 2020-04-01 17:08:48,589 INFO
> org.apache.flink.runtime.checkpoint.ZooKeeperCompletedCheckpointStore -
> Trying to retrieve checkpoint 51.
> 2020-04-01 17:09:12,775 INFO org.apache.flink.yarn.YarnResourceManager - The
> heartbeat of JobManager with id 02500708baf0bb976891c391afd3d7d5 timed out.
> {code}
> Digging into the code, looks like ExecutionGraph::restart runs in JobMaster
> main thread and finally calls
> ZooKeeperCompletedCheckpointStore::retrieveCompletedCheckpoint which download
> file form DFS. The main thread is basically blocked for a while because of
> this. One possible solution is to making the downloading part async. More
> things might need to consider as the original change tries to make it
> single-threaded. [https://github.com/apache/flink/pull/7568]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16975) Add docs for FileSystem connector
[ https://issues.apache.org/jira/browse/FLINK-16975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-16975: - Fix Version/s: 1.11.0 > Add docs for FileSystem connector > - > > Key: FLINK-16975 > URL: https://issues.apache.org/jira/browse/FLINK-16975 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.11.0 >Reporter: Leonard Xu >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111869#comment-17111869 ] Yu Li commented on FLINK-15507: --- Thanks for volunteering [~Zakelly], just assigned to you. > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Assignee: Zakelly Lan >Priority: Blocker > Fix For: 1.11.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16245) Use a delegating classloader as the user code classloader to prevent class leaks.
[ https://issues.apache.org/jira/browse/FLINK-16245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-16245: --- Fix Version/s: (was: 1.11.0) 1.12.0 > Use a delegating classloader as the user code classloader to prevent class > leaks. > - > > Key: FLINK-16245 > URL: https://issues.apache.org/jira/browse/FLINK-16245 > Project: Flink > Issue Type: Improvement > Components: Runtime / Task >Reporter: Stephan Ewen >Assignee: Arvid Heise >Priority: Critical > Labels: pull-request-available, usability > Fix For: 1.12.0 > > Time Spent: 10m > Remaining Estimate: 0h > > As reported in FLINK-11205, a reference to the user-code ClassLoader can be > held by some libraries, causing class leaks. > One way to circumvent this class leak is if the ClassLoader that we set as > the user-code ClassLoader is a delegating ClassLoader to the real class > loader, and when closing the user code ClassLoader we null out the reference. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-16165) NestedLoopJoin fallback to HashJoin when build records number is very large
[ https://issues.apache.org/jira/browse/FLINK-16165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski reassigned FLINK-16165: -- Assignee: Jingsong Lee > NestedLoopJoin fallback to HashJoin when build records number is very large > --- > > Key: FLINK-16165 > URL: https://issues.apache.org/jira/browse/FLINK-16165 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Critical > Fix For: 1.11.0 > > > Now, if the statistic is not so accurate, maybe choose wrong join type to > nested loop join. > If build records number is very large, this lead to very slow join, the user > looks like the join is in Hang. > It is a stability problem, We should fallback to hash join in runtime to > avoid this hang. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li reassigned FLINK-15507: - Assignee: Zakelly Lan > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Assignee: Zakelly Lan >Priority: Blocker > Fix For: 1.11.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15507) Activate local recovery for RocksDB backends by default
[ https://issues.apache.org/jira/browse/FLINK-15507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111865#comment-17111865 ] Zakelly Lan commented on FLINK-15507: - I would like to take this ticket. > Activate local recovery for RocksDB backends by default > --- > > Key: FLINK-15507 > URL: https://issues.apache.org/jira/browse/FLINK-15507 > Project: Flink > Issue Type: Improvement > Components: Runtime / State Backends >Reporter: Stephan Ewen >Priority: Blocker > Fix For: 1.11.0 > > > For the RocksDB state backend, local recovery has no overhead when > incremental checkpoints are used. > It should be activated by default, because it greatly helps with recovery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17800) RocksDB optimizeForPointLookup results in missing time windows
[
https://issues.apache.org/jira/browse/FLINK-17800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111864#comment-17111864
]
Yordan Pavlov commented on FLINK-17800:
---
[~yunta]
Yes the attachment reproduces the problem, it is an even simpler code than what
I originally described as it does not have a Kafka source.
I did check with disabling the managed memory:
{noformat}
main] INFO org.apache.flink.configuration.GlobalConfiguration - Loading
configuration property: state.backend.rocksdb.memory.managed, false{noformat}
that does not prevent the problem from happening.
> RocksDB optimizeForPointLookup results in missing time windows
> --
>
> Key: FLINK-17800
> URL: https://issues.apache.org/jira/browse/FLINK-17800
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.10.0, 1.10.1
>Reporter: Yordan Pavlov
>Priority: Major
> Attachments: MissingWindows.scala
>
>
> +My Setup:+
> We have been using the _RocksDb_ option of _optimizeForPointLookup_ and
> running version 1.7 for years. Upon upgrading to Flink 1.10 we started
> receiving a strange behavior of missing time windows on a streaming Flink
> job. For the purpose of testing I experimented with previous Flink version
> and (1.8, 1.9, 1.9.3) and non of them showed the problem
>
> A sample of the code demonstrating the problem is here:
> {code:java}
> val datastream = env
> .addSource(KafkaSource.keyedElements(config.kafkaElements,
> List(config.kafkaBootstrapServer)))
> val result = datastream
> .keyBy( _ => 1)
> .timeWindow(Time.milliseconds(1))
> .print()
> {code}
>
>
> The source consists of 3 streams (being either 3 Kafka partitions or 3 Kafka
> topics), the elements in each of the streams are separately increasing. The
> elements generate increasing timestamps using an event time and start from 1,
> increasing by 1. The first partitions would consist of timestamps 1, 2, 10,
> 15..., the second of 4, 5, 6, 11..., the third of 3, 7, 8, 9...
>
> +What I observe:+
> The time windows would open as I expect for the first 127 timestamps. Then
> there would be a huge gap with no opened windows, if the source has many
> elements, then next open window would be having a timestamp in the thousands.
> A gap of hundred of elements would be created with what appear to be 'lost'
> elements. Those elements are not reported as late (if tested with the
> ._sideOutputLateData_ operator). The way we have been using the option is by
> setting in inside the config like so:
> ??etherbi.rocksDB.columnOptions.optimizeForPointLookup=268435456??
> We have been using it for performance reasons as we have huge RocksDB state
> backend.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tzulitai commented on a change in pull request #12263: [FLINK-16998][core] Support backwards compatibility for upgraded RowSerializer
tzulitai commented on a change in pull request #12263:
URL: https://github.com/apache/flink/pull/12263#discussion_r427782378
##
File path:
flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/RowSerializer.java
##
@@ -367,36 +367,42 @@ public int getVersion() {
/**
* A {@link TypeSerializerSnapshot} for RowSerializer.
*/
- // TODO not fully functional yet due to FLINK-17520
public static final class RowSerializerSnapshot extends
CompositeTypeSerializerSnapshot {
private static final int VERSION = 3;
- private static final int VERSION_WITHOUT_ROW_KIND = 2;
+ private static final int LAST_VERSION_WITHOUT_ROW_KIND = 2;
- private boolean legacyModeEnabled = false;
+ private int version = VERSION;
public RowSerializerSnapshot() {
super(RowSerializer.class);
}
RowSerializerSnapshot(RowSerializer serializerInstance) {
super(serializerInstance);
+ this.version = translateVersion(serializerInstance);
}
@Override
protected int getCurrentOuterSnapshotVersion() {
- return VERSION;
+ return version;
Review comment:
This method is only ever relevant for when writing snapshots and not
used on restore.
Therefore, this should always be the latest version, and not the read older
version.
```suggestion
return VERSION;
```
##
File path:
flink-core/src/test/java/org/apache/flink/api/java/typeutils/runtime/RowSerializerUpgradeTest.java
##
@@ -76,14 +84,16 @@ public RowSerializerUpgradeTest(TestSpecification
testSpecification) {
public static final class RowSerializerSetup implements
TypeSerializerUpgradeTestBase.PreUpgradeSetup {
@Override
public TypeSerializer createPriorSerializer() {
- return stringLongRowSupplier();
+ return createRowSerializer(true);
Review comment:
To really clarify this, I think we should make the `RowSerializer`
constructor that allows passing in the `legacyModeEnabled` flag private, to be
only usable by the `RowSerializerSnapshot#createOuterSerializer`. This concern
should not be leaked into tests.
The bottom line is, the concern of creating an old serializer with previous
formats should only be visible to the snapshots.
##
File path:
flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/RowSerializer.java
##
@@ -367,36 +367,42 @@ public int getVersion() {
/**
* A {@link TypeSerializerSnapshot} for RowSerializer.
*/
- // TODO not fully functional yet due to FLINK-17520
public static final class RowSerializerSnapshot extends
CompositeTypeSerializerSnapshot {
private static final int VERSION = 3;
- private static final int VERSION_WITHOUT_ROW_KIND = 2;
+ private static final int LAST_VERSION_WITHOUT_ROW_KIND = 2;
- private boolean legacyModeEnabled = false;
+ private int version = VERSION;
Review comment:
Maybe rename this to `readVersion`, to better convey its difference with
the static `VERSION`.
##
File path:
flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/RowSerializer.java
##
@@ -367,36 +367,42 @@ public int getVersion() {
/**
* A {@link TypeSerializerSnapshot} for RowSerializer.
*/
- // TODO not fully functional yet due to FLINK-17520
public static final class RowSerializerSnapshot extends
CompositeTypeSerializerSnapshot {
private static final int VERSION = 3;
- private static final int VERSION_WITHOUT_ROW_KIND = 2;
+ private static final int LAST_VERSION_WITHOUT_ROW_KIND = 2;
- private boolean legacyModeEnabled = false;
+ private int version = VERSION;
public RowSerializerSnapshot() {
super(RowSerializer.class);
}
RowSerializerSnapshot(RowSerializer serializerInstance) {
super(serializerInstance);
+ this.version = translateVersion(serializerInstance);
}
@Override
protected int getCurrentOuterSnapshotVersion() {
- return VERSION;
+ return version;
}
@Override
protected void readOuterSnapshot(
int readOuterSnapshotVersion,
DataInputView in,
ClassLoader userCodeClassLoader) {
- if (
[jira] [Updated] (FLINK-10241) Reduce performance/stability impact of latency metrics
[ https://issues.apache.org/jira/browse/FLINK-10241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-10241: --- Fix Version/s: (was: 1.11.0) 1.12.0 > Reduce performance/stability impact of latency metrics > -- > > Key: FLINK-10241 > URL: https://issues.apache.org/jira/browse/FLINK-10241 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Affects Versions: 1.5.0, 1.6.0, 1.7.0 >Reporter: Chesnay Schepler >Assignee: Chesnay Schepler >Priority: Critical > Fix For: 1.12.0 > > > Umbrella issue for performance/stability improvements around the latency > metrics. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12230: [FLINK-17504][docs] Update Chinese translation of Getting Started / O…
flinkbot edited a comment on pull request #12230: URL: https://github.com/apache/flink/pull/12230#issuecomment-630205457 ## CI report: * 2f0ca570ff878cd12f999570590a08fa75efcc6b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1903) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
flinkbot edited a comment on pull request #12256: URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695 ## CI report: * 265fa7767926382824b2894b2662d3a93a76b176 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885) * edab8aef00bcac9a530501fa64165f344d7c74e7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1916) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12258: [FLINK-17820][task][checkpointing] Don't flush channel state to disk explicitly
flinkbot edited a comment on pull request #12258: URL: https://github.com/apache/flink/pull/12258#issuecomment-631108764 ## CI report: * cf629e225bc323888017be5d5a86c7c89a2b76bd Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1892) * 8d01ba80d36c07517d7493cef13d6ab634c01e18 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1909) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12263: [FLINK-16998][core] Support backwards compatibility for upgraded RowSerializer
flinkbot edited a comment on pull request #12263: URL: https://github.com/apache/flink/pull/12263#issuecomment-631274882 ## CI report: * 5e0f9df0a404a5d88b8762238ec37b903b9f0e4b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1910) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12090: [FLINK-17622][connectors / jdbc] Remove useless switch for decimal in PostgresCatalog
flinkbot edited a comment on pull request #12090: URL: https://github.com/apache/flink/pull/12090#issuecomment-626999634 ## CI report: * c6f506b50b8728300915c3353e606a807ac1c0c5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1849) * 29252f270a654406deb02ed0f0e552605c476b68 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1915) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12181: [FLINK-17645][runtime] Fix SafetyNetCloseableRegistry constructor bug.
zhuzhurk commented on a change in pull request #12181:
URL: https://github.com/apache/flink/pull/12181#discussion_r427794339
##
File path:
flink-core/src/test/java/org/apache/flink/core/fs/SafetyNetCloseableRegistryTest.java
##
@@ -198,4 +198,29 @@ public void testReaperThreadSpawnAndStop() throws
Exception {
}
Assert.assertFalse(SafetyNetCloseableRegistry.isReaperThreadRunning());
}
+
+ /**
+* Test whether failure to start thread in {@link
SafetyNetCloseableRegistry}
+* constructor can lead to failure of subsequent state check.
+*/
+ @Test
+ public void testReaperThreadStartFailed() throws Exception {
+
+ try {
+ new SafetyNetCloseableRegistry(() -> new
OutOfMemoryReaperThread());
+ } catch (java.lang.OutOfMemoryError error) {
+ }
+
Review comment:
We can add `isReaperThreadRunning` checks after the failed creation and
the succeeded creation.
This helps to verify that a reaper thread is really created and running.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Assigned] (FLINK-16986) Enhance the OperatorEvent handling guarantee during checkpointing.
[
https://issues.apache.org/jira/browse/FLINK-16986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen reassigned FLINK-16986:
Assignee: Stephan Ewen (was: Jiangjie Qin)
> Enhance the OperatorEvent handling guarantee during checkpointing.
> --
>
> Key: FLINK-16986
> URL: https://issues.apache.org/jira/browse/FLINK-16986
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Jiangjie Qin
>Assignee: Stephan Ewen
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> When the {{CheckpointCoordinator}} takes a checkpoint, the checkpointing
> order is following:
> # {{CheckpointCoordinator}} triggers checkpoint on each
> {{OperatorCoordinator}}
> # Each {{OperatorCoordinator}} takes a snapshot.
> # Right after taking the snapshot, the {{CheckpointCoordinator}} sends a
> {{CHECKPOINT_FIN}} marker through the {{OperatorContext}}.
> # Once the {{OperatorContext}} sees {{CHECKPOINT_FIN}} marker, it will wait
> for all the previous events are acked and suspend the event gateway to the
> operators by buffering the future {{OperatorEvents}} sent from the
> {{OperatorCoordinator}} to the operators without actually sending them out.
> # The {{CheckpointCoordinator}} waits until all the {{OperatorCoordinator}}s
> finish step 2-4 and then triggers the task snapshots.
> # The suspension of an event gateway to an operator can be lifted after all
> the subtasks of that operator has finished their task checkpoint.
> The mechanism above guarantees all the {{OperatorEvents}} sent before taking
> the operator coordinator snapshot are handled by the operator before the task
> snapshots are taken.
> An operator can use this mechanism to know whether an {{OperatorEvent}} it
> sent to the coordinator is included in the upcoming checkpoint or not. What
> it has to do is to ask the operator coordinator to ACK that OperatorEvent. If
> the ACK is received before the operator takes the next snapshot, that
> OperatorEvent must have been handled and checkpointed by the
> OperatorCoordinator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on a change in pull request #12181: [FLINK-17645][runtime] Fix SafetyNetCloseableRegistry constructor bug.
zhuzhurk commented on a change in pull request #12181:
URL: https://github.com/apache/flink/pull/12181#discussion_r427792440
##
File path:
flink-core/src/test/java/org/apache/flink/core/fs/SafetyNetCloseableRegistryTest.java
##
@@ -198,4 +198,29 @@ public void testReaperThreadSpawnAndStop() throws
Exception {
}
Assert.assertFalse(SafetyNetCloseableRegistry.isReaperThreadRunning());
}
+
+ /**
+* Test whether failure to start thread in {@link
SafetyNetCloseableRegistry}
+* constructor can lead to failure of subsequent state check.
+*/
+ @Test
+ public void testReaperThreadStartFailed() throws Exception {
+
+ try {
+ new SafetyNetCloseableRegistry(() -> new
OutOfMemoryReaperThread());
+ } catch (java.lang.OutOfMemoryError error) {
+ }
+
+ // the OOM error will not lead to failure of subsequent
constructor call.
+ SafetyNetCloseableRegistry closeableRegistry = new
SafetyNetCloseableRegistry();
+ closeableRegistry.close();
+ }
+
+ static class OutOfMemoryReaperThread extends
SafetyNetCloseableRegistry.CloseableReaperThread {
Review comment:
can be private
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17817) CollectResultFetcher fails with EOFException in AggregateReduceGroupingITCase
[
https://issues.apache.org/jira/browse/FLINK-17817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111829#comment-17111829
]
Caizhi Weng edited comment on FLINK-17817 at 5/20/20, 7:17 AM:
---
I remembered that FLINK-17774 actually solves this problem. To handle object
reuse (precisely {{RowData}} reuse), collect sink in FLINK-17774 will serialize
the values in {{invoke}} method, so there is no serializing in socket server
thread. Let's wait for FLINK-17774 to be merged so that this problem can also
be solved.
was (Author: tsreaper):
I remembered that FLINK-17774 actually solves this problem. To handle object
reuse, collect sink in FLINK-17774 will serialize the values in {{invoke}}
method, so there is no serializing in socket server thread. Let's wait for
FLINK-17774 to be merged so that this problem can also be solved.
> CollectResultFetcher fails with EOFException in AggregateReduceGroupingITCase
> -
>
> Key: FLINK-17817
> URL: https://issues.apache.org/jira/browse/FLINK-17817
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Blocker
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1826&view=logs&j=e25d5e7e-2a9c-5589-4940-0b638d75a414&t=f83cd372-208c-5ec4-12a8-337462457129
> {code}
> 2020-05-19T10:34:18.3224679Z [ERROR]
> testSingleAggOnTable_SortAgg(org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase)
> Time elapsed: 7.537 s <<< ERROR!
> 2020-05-19T10:34:18.3225273Z java.lang.RuntimeException: Failed to fetch next
> result
> 2020-05-19T10:34:18.3227634Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:92)
> 2020-05-19T10:34:18.3228518Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:63)
> 2020-05-19T10:34:18.3229170Z at
> org.apache.flink.shaded.guava18.com.google.common.collect.Iterators.addAll(Iterators.java:361)
> 2020-05-19T10:34:18.3229863Z at
> org.apache.flink.shaded.guava18.com.google.common.collect.Lists.newArrayList(Lists.java:160)
> 2020-05-19T10:34:18.3230586Z at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.executeQuery(BatchTestBase.scala:300)
> 2020-05-19T10:34:18.3231303Z at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.check(BatchTestBase.scala:141)
> 2020-05-19T10:34:18.3231996Z at
> org.apache.flink.table.planner.runtime.utils.BatchTestBase.checkResult(BatchTestBase.scala:107)
> 2020-05-19T10:34:18.3232847Z at
> org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase.testSingleAggOnTable(AggregateReduceGroupingITCase.scala:176)
> 2020-05-19T10:34:18.3233694Z at
> org.apache.flink.table.planner.runtime.batch.sql.agg.AggregateReduceGroupingITCase.testSingleAggOnTable_SortAgg(AggregateReduceGroupingITCase.scala:122)
> 2020-05-19T10:34:18.3234461Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-19T10:34:18.3234983Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-19T10:34:18.3235632Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-19T10:34:18.3236615Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-19T10:34:18.3237256Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-19T10:34:18.3237965Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-19T10:34:18.3238750Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-19T10:34:18.3239314Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-19T10:34:18.3239838Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-19T10:34:18.3240362Z at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2020-05-19T10:34:18.3240803Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-05-19T10:34:18.3243624Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-05-19T10:34:18.3244531Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-19T10:34:18.3245325Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-19T10:34:18.3246086Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:5
[jira] [Assigned] (FLINK-17822) Nightly Flink CLI end-to-end test failed with "JavaGcCleanerWrapper$PendingCleanersRunner cannot access class jdk.internal.misc.SharedSecrets" in Java 11
[
https://issues.apache.org/jira/browse/FLINK-17822?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin reassigned FLINK-17822:
---
Assignee: Andrey Zagrebin
> Nightly Flink CLI end-to-end test failed with
> "JavaGcCleanerWrapper$PendingCleanersRunner cannot access class
> jdk.internal.misc.SharedSecrets" in Java 11
> --
>
> Key: FLINK-17822
> URL: https://issues.apache.org/jira/browse/FLINK-17822
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task, Tests
>Affects Versions: 1.11.0
>Reporter: Dian Fu
>Assignee: Andrey Zagrebin
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
>
> Instance:
> https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/1887/logs/600
> {code}
> 2020-05-19T21:59:39.8829043Z 2020-05-19 21:59:25,193 ERROR
> org.apache.flink.util.JavaGcCleanerWrapper [] - FATAL
> UNEXPECTED - Failed to invoke waitForReferenceProcessing
> 2020-05-19T21:59:39.8829849Z java.lang.IllegalAccessException: class
> org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunner cannot
> access class jdk.internal.misc.SharedSecrets (in module java.base) because
> module java.base does not export jdk.internal.misc to unnamed module @54e3658c
> 2020-05-19T21:59:39.8830707Z at
> jdk.internal.reflect.Reflection.newIllegalAccessException(Reflection.java:361)
> ~[?:?]
> 2020-05-19T21:59:39.8831166Z at
> java.lang.reflect.AccessibleObject.checkAccess(AccessibleObject.java:591)
> ~[?:?]
> 2020-05-19T21:59:39.8831744Z at
> java.lang.reflect.Method.invoke(Method.java:558) ~[?:?]
> 2020-05-19T21:59:39.8832596Z at
> org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunner.getJavaLangRefAccess(JavaGcCleanerWrapper.java:362)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8833667Z at
> org.apache.flink.util.JavaGcCleanerWrapper$PendingCleanersRunner.tryRunPendingCleaners(JavaGcCleanerWrapper.java:351)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8834712Z at
> org.apache.flink.util.JavaGcCleanerWrapper$CleanerManager.tryRunPendingCleaners(JavaGcCleanerWrapper.java:207)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8835686Z at
> org.apache.flink.util.JavaGcCleanerWrapper.tryRunPendingCleaners(JavaGcCleanerWrapper.java:158)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8836652Z at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.reserveMemory(UnsafeMemoryBudget.java:94)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8838033Z at
> org.apache.flink.runtime.memory.UnsafeMemoryBudget.verifyEmpty(UnsafeMemoryBudget.java:64)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8839259Z at
> org.apache.flink.runtime.memory.MemoryManager.verifyEmpty(MemoryManager.java:172)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8840148Z at
> org.apache.flink.runtime.taskexecutor.slot.TaskSlot.verifyMemoryFreed(TaskSlot.java:311)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8841035Z at
> org.apache.flink.runtime.taskexecutor.slot.TaskSlot.lambda$closeAsync$1(TaskSlot.java:301)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8841603Z at
> java.util.concurrent.CompletableFuture.uniRunNow(CompletableFuture.java:815)
> ~[?:?]
> 2020-05-19T21:59:39.8842069Z at
> java.util.concurrent.CompletableFuture.uniRunStage(CompletableFuture.java:799)
> ~[?:?]
> 2020-05-19T21:59:39.8842844Z at
> java.util.concurrent.CompletableFuture.thenRun(CompletableFuture.java:2121)
> ~[?:?]
> 2020-05-19T21:59:39.8843828Z at
> org.apache.flink.runtime.taskexecutor.slot.TaskSlot.closeAsync(TaskSlot.java:300)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8844790Z at
> org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl.freeSlotInternal(TaskSlotTableImpl.java:404)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8845754Z at
> org.apache.flink.runtime.taskexecutor.slot.TaskSlotTableImpl.freeSlot(TaskSlotTableImpl.java:365)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8846842Z at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.freeSlotInternal(TaskExecutor.java:1589)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8847711Z at
> org.apache.flink.runtime.taskexecutor.TaskExecutor.freeSlot(TaskExecutor.java:967)
> ~[flink-dist_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]
> 2020-05-19T21:59:39.8848295Z at
[jira] [Updated] (FLINK-16986) Enhance the OperatorEvent handling guarantee during checkpointing.
[
https://issues.apache.org/jira/browse/FLINK-16986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-16986:
-
Priority: Blocker (was: Major)
> Enhance the OperatorEvent handling guarantee during checkpointing.
> --
>
> Key: FLINK-16986
> URL: https://issues.apache.org/jira/browse/FLINK-16986
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> When the {{CheckpointCoordinator}} takes a checkpoint, the checkpointing
> order is following:
> # {{CheckpointCoordinator}} triggers checkpoint on each
> {{OperatorCoordinator}}
> # Each {{OperatorCoordinator}} takes a snapshot.
> # Right after taking the snapshot, the {{CheckpointCoordinator}} sends a
> {{CHECKPOINT_FIN}} marker through the {{OperatorContext}}.
> # Once the {{OperatorContext}} sees {{CHECKPOINT_FIN}} marker, it will wait
> for all the previous events are acked and suspend the event gateway to the
> operators by buffering the future {{OperatorEvents}} sent from the
> {{OperatorCoordinator}} to the operators without actually sending them out.
> # The {{CheckpointCoordinator}} waits until all the {{OperatorCoordinator}}s
> finish step 2-4 and then triggers the task snapshots.
> # The suspension of an event gateway to an operator can be lifted after all
> the subtasks of that operator has finished their task checkpoint.
> The mechanism above guarantees all the {{OperatorEvents}} sent before taking
> the operator coordinator snapshot are handled by the operator before the task
> snapshots are taken.
> An operator can use this mechanism to know whether an {{OperatorEvent}} it
> sent to the coordinator is included in the upcoming checkpoint or not. What
> it has to do is to ask the operator coordinator to ACK that OperatorEvent. If
> the ACK is received before the operator takes the next snapshot, that
> OperatorEvent must have been handled and checkpointed by the
> OperatorCoordinator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-16986) Enhance the OperatorEvent handling guarantee during checkpointing.
[
https://issues.apache.org/jira/browse/FLINK-16986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-16986:
-
Fix Version/s: 1.11.0
> Enhance the OperatorEvent handling guarantee during checkpointing.
> --
>
> Key: FLINK-16986
> URL: https://issues.apache.org/jira/browse/FLINK-16986
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Checkpointing
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> When the {{CheckpointCoordinator}} takes a checkpoint, the checkpointing
> order is following:
> # {{CheckpointCoordinator}} triggers checkpoint on each
> {{OperatorCoordinator}}
> # Each {{OperatorCoordinator}} takes a snapshot.
> # Right after taking the snapshot, the {{CheckpointCoordinator}} sends a
> {{CHECKPOINT_FIN}} marker through the {{OperatorContext}}.
> # Once the {{OperatorContext}} sees {{CHECKPOINT_FIN}} marker, it will wait
> for all the previous events are acked and suspend the event gateway to the
> operators by buffering the future {{OperatorEvents}} sent from the
> {{OperatorCoordinator}} to the operators without actually sending them out.
> # The {{CheckpointCoordinator}} waits until all the {{OperatorCoordinator}}s
> finish step 2-4 and then triggers the task snapshots.
> # The suspension of an event gateway to an operator can be lifted after all
> the subtasks of that operator has finished their task checkpoint.
> The mechanism above guarantees all the {{OperatorEvents}} sent before taking
> the operator coordinator snapshot are handled by the operator before the task
> snapshots are taken.
> An operator can use this mechanism to know whether an {{OperatorEvent}} it
> sent to the coordinator is included in the upcoming checkpoint or not. What
> it has to do is to ask the operator coordinator to ACK that OperatorEvent. If
> the ACK is received before the operator takes the next snapshot, that
> OperatorEvent must have been handled and checkpointed by the
> OperatorCoordinator.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12090: [FLINK-17622][connectors / jdbc] Remove useless switch for decimal in PostgresCatalog
flinkbot edited a comment on pull request #12090: URL: https://github.com/apache/flink/pull/12090#issuecomment-626999634 ## CI report: * c6f506b50b8728300915c3353e606a807ac1c0c5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1849) * 29252f270a654406deb02ed0f0e552605c476b68 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12256: [FLINK-17018][runtime] DefaultExecutionSlotAllocator allocates slots in bulks
flinkbot edited a comment on pull request #12256: URL: https://github.com/apache/flink/pull/12256#issuecomment-631025695 ## CI report: * 265fa7767926382824b2894b2662d3a93a76b176 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1885) * edab8aef00bcac9a530501fa64165f344d7c74e7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17541) Support inline structured types
[ https://issues.apache.org/jira/browse/FLINK-17541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Timo Walther closed FLINK-17541. Fix Version/s: 1.11.0 Resolution: Fixed Fixed in 1.12.0: 11b5330563b59e085ad4755c9f9a3da6f206c1c3 Fixed in 1.11.0: d255bef5daca2aab238ed9e00ddf14a00193a5a4 > Support inline structured types > --- > > Key: FLINK-17541 > URL: https://issues.apache.org/jira/browse/FLINK-17541 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > > Many locations in the code base already support structured types. The runtime > treats them as row types. However, some final work is needed to support > structured types though the stack. We start with inline structured types. > Registered structured types in catalog are covered in a different issue. > Inline structured types are a prerequisite to enable aggregate functions in > FLIP-65 again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] TsReaper commented on pull request #12262: [FLINK-17817][hotfix] Fix serializer thread safe problem in CollectSinkFunction
TsReaper commented on pull request #12262: URL: https://github.com/apache/flink/pull/12262#issuecomment-631280841 Closing because #12199 also solves this problem. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17826) Add missing custom query support on new jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111840#comment-17111840
]
Jark Wu commented on FLINK-17826:
-
Hi [~zjwang], I would like to treat this one as a bug fix and should be fixed
in 1.11, otherwise there is feature regression on the new jdbc connector. What
do you think?
> Add missing custom query support on new jdbc connector
> --
>
> Key: FLINK-17826
> URL: https://issues.apache.org/jira/browse/FLINK-17826
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.11.0
>
>
> In FLINK-17361, we added custom query on JDBC tables, but missing to add the
> same ability on new jdbc connector (i.e.
> {{JdbcDynamicTableSourceSinkFactory}}).
> In the new jdbc connector, maybe we should call it {{scan.query}} to keep
> consistent with other scan options, besides we need to make {{"table-name"}}
> optional, but add validation that "table-name" and "scan.query" shouldn't
> both be empty, and "table-name" must not be empty when used as sink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17826) Add missing custom query support on new jdbc connector
[
https://issues.apache.org/jira/browse/FLINK-17826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17111837#comment-17111837
]
Jark Wu commented on FLINK-17826:
-
Would you like to take this? [~f.pompermaier]
> Add missing custom query support on new jdbc connector
> --
>
> Key: FLINK-17826
> URL: https://issues.apache.org/jira/browse/FLINK-17826
> Project: Flink
> Issue Type: Sub-task
> Components: Connectors / JDBC
>Reporter: Jark Wu
>Priority: Major
> Fix For: 1.11.0
>
>
> In FLINK-17361, we added custom query on JDBC tables, but missing to add the
> same ability on new jdbc connector (i.e.
> {{JdbcDynamicTableSourceSinkFactory}}).
> In the new jdbc connector, maybe we should call it {{scan.query}} to keep
> consistent with other scan options, besides we need to make {{"table-name"}}
> optional, but add validation that "table-name" and "scan.query" shouldn't
> both be empty, and "table-name" must not be empty when used as sink.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] gyfora commented on pull request #12250: [FLINK-17619] Backport to 1.11
gyfora commented on pull request #12250: URL: https://github.com/apache/flink/pull/12250#issuecomment-631279205 merged This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
