[jira] [Created] (FLINK-24679) Configuration parsing exceptions may leak sensitive data
Chesnay Schepler created FLINK-24679: Summary: Configuration parsing exceptions may leak sensitive data Key: FLINK-24679 URL: https://issues.apache.org/jira/browse/FLINK-24679 Project: Flink Issue Type: Bug Components: Runtime / Configuration Affects Versions: 1.13.0 Reporter: Chesnay Schepler Various exceptions thrown by the Configuration when a parsing error occurs print the problematic value as-is. This means a typo by the user may result in all or part of a sensitive value to be leaked. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [Discuss] Planning Flink 1.15
I think it is important that most of the people who contribute new features are available during the testing/stabilization period because otherwise there is a risk that the release gets delayed. Hence +1 for moving the feature freeze to the 6th of February. Cheers, Till On Thu, Oct 28, 2021 at 7:00 AM Guowei Ma wrote: > Thanks for volunteering as our release managers Till, Yun and Joe! > > +1 for feature freeze on 2.6 ,which is more convenient for us to plan 1.15 > related work. > > Best, > Guowei > > > On Thu, Oct 28, 2021 at 12:01 PM Xintong Song > wrote: > > > Thanks for kicking this off, Joe. > > > > +1 for Till, Yun and Joe as the release managers. > > > > Also +1 for feature freeze on 2.6, which would make it easier for folks > > from China to dedicate to the release testing. > > > > Thank you~ > > > > Xintong Song > > > > > > > > On Wed, Oct 27, 2021 at 11:04 PM Yun Gao > > wrote: > > > > > Hi all, > > > > > > Very thanks @Joe for starting the discussion! And it is really > honorable > > > to be the release manager of 1.15~ > > > > > > And for the releasing date, there might be some concerns since Jan. > 17th > > > is closed to the Spring Festival Holiday (1.31 to 2.6) and many people > > > might also start the holiday further in advance, it might have some > > > influence to our releasing test / stabilization period. Thus perhaps > > > another option might be we slightly extend the code freeze date to > bypass > > > the holiday till Feb. 6th ? WDYT ? > > > > > > Thank you~ > > > > > > Best, > > > Yun > > > > > > > > > > > > > > > -- > > > From:Israel Ekpo > > > Send Time:2021 Oct. 26 (Tue.) 08:09 > > > To:dev > > > Subject:Re: [Discuss] Planning Flink 1.15 > > > > > > Thanks for the update, Joe. > > > > > > Looking forward to this release. > > > > > > On Mon, Oct 25, 2021 at 11:13 AM Johannes Moser > > wrote: > > > > > > > Hi all, > > > > > > > > As people have already started working on their 1.15. > > > > contribution, we'd like to start the discussion for > > > > the release setup. > > > > > > > > - Release managers: As a team of three seems to have > > > > worked perfectly fine, we'd like to suggest Till, Yun Gao > > > > & Joe as the release managers for 1.15. > > > > > > > > - Timeline: 1.14 was released at the end of September and > > > > aiming for a 4 months release cycle including one months > > > > of stabilisation would lead to a feature freeze date at the > > > > end of December, which would make the European holiday > > > > season a bit stressful. One option would have been to aim for early > > > > December, but we decided to go for the 17th of January. > > > > Such that we also have some buffer before the Chinese new > > > > year. > > > > > > > > - Bi-weekly sync: We'd also like to setup a bi-weekly sync again > > > > starting from the 9th of November at 9am CET/4pm CST. > > > > > > > > - Collecting features: As last time it would be helpful to have > > > > a rough overview of the efforts that will likely be included in > > > > this release. We have created a wiki page [1] for collecting such > > > > information. We'd like to kindly ask all committers to fill in the > > > > page with features that they intend to work on. > > > > > > > > Just copy pasting what we included into the planning email > > > > for 1.14, because it still applies: > > > > > > > > - Stability of master: This has been an issue during the 1.13 & 1.14 > > > > feature freeze phase and it is still going on. We encourage every > > > > committer to not merge PRs through the Github button, but do this > > > > manually, with caution for the commits merged after the CI being > > > > triggered. It would be appreciated to always build the project before > > > > merging to master. > > > > > > > > - Documentation: Please try to see documentation as an integrated > > > > part of the engineering process and don't push it to the feature > > > > freeze phase or even after. You might even think about going > > > > documentation first. We, as the Flink community, are adding great > > > > stuff, that is pushing the limits of streaming data processors, with > > > > every release. We should also make this stuff usable for our users by > > > > documenting it well. > > > > > > > > - Promotion of 1.15: What applies to documentation also applies > > > > to all the activity around the release. We encourage every > contributor > > > > to also think about, plan and prepare activities like blog posts and > > > talk, > > > > that will promote and spread the release once it is done. > > > > > > > > Please let us know what you think. > > > > > > > > Thank you~ > > > > Till, Yun Gao & Joe > > > > > > > > [1] https://cwiki.apache.org/confluence/display/FLINK/1.15+Release > > > > > > > > > >
[jira] [Created] (FLINK-24680) Expose UserCodeClassLoader in OperatorCoordinator.Context for registering release hooks
Qingsheng Ren created FLINK-24680: - Summary: Expose UserCodeClassLoader in OperatorCoordinator.Context for registering release hooks Key: FLINK-24680 URL: https://issues.apache.org/jira/browse/FLINK-24680 Project: Flink Issue Type: Bug Components: Runtime / Coordination Affects Versions: 1.14.0 Reporter: Qingsheng Ren Fix For: 1.15.0 Currently `OperatorCoordinator.Context` only exposes `ClassLoader` for accessing user code class loader, which doesn't support adding release hooks in `OperatorCoordinator`, like sync releasing class loader and closing operator coordinator. We need to expose `UserCodeClassLoader` in the context fo registering hooks in coordinator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[DISCUSS] FLIP-189: SQL Client Usability Improvements
Hi all, I want to start a discussion about FLIP-189: SQL Client Usability Improvements. The main changes in this FLIP: - Flink sql client parser improvements so that sql client does not ask for ; inside a quoted string or a comment - use prompt to show what sql client is waiting for - introduce syntax highlighting - improve completion For more detailed changes, please refer to FLIP-189[1]. [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-189%3A+SQL+Client+Usability+Improvements Look forward to your feedback. -- Best regards, Sergey
[jira] [Created] (FLINK-24681) org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined offset with no reset policy for partitions
qinghuan wang created FLINK-24681:
-
Summary:
org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined
offset with no reset policy for partitions
Key: FLINK-24681
URL: https://issues.apache.org/jira/browse/FLINK-24681
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka
Affects Versions: 1.14.0
Reporter: qinghuan wang
When create a Kafka Table
{code:java}
CREATE TABLE KafkaTable (
...
) WITH (
'connector' = 'kafka',
'topic' = 'user_behavior',
'properties.bootstrap.servers' = '192.168.3.244:9092',
'properties.group.id' = 'testGroup',
'format' = 'csv'
);
{code}
An exception throws:
{code:java}
Caused by: org.apache.kafka.clients.consumer.NoOffsetForPartitionException:
Undefined offset with no reset policy for partitions: [user_behavior-0]Caused
by: org.apache.kafka.clients.consumer.NoOffsetForPartitionException: Undefined
offset with no reset policy for partitions: [haikang-face-recognition-0] at
org.apache.kafka.clients.consumer.internals.SubscriptionState.resetMissingPositions(SubscriptionState.java:631)
at
org.apache.kafka.clients.consumer.KafkaConsumer.updateFetchPositions(KafkaConsumer.java:2343)
at
org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1725)
at
org.apache.kafka.clients.consumer.KafkaConsumer.position(KafkaConsumer.java:1684)
at
org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader.removeEmptySplits(KafkaPartitionSplitReader.java:375)
at
org.apache.flink.connector.kafka.source.reader.KafkaPartitionSplitReader.handleSplitsChanges(KafkaPartitionSplitReader.java:260)
at
org.apache.flink.connector.base.source.reader.fetcher.AddSplitsTask.run(AddSplitsTask.java:51)
at
org.apache.flink.connector.base.source.reader.fetcher.SplitFetcher.runOnce(SplitFetcher.java:142)
... 7 common frames omitted{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] Creating an external connector repository
Thank you all, for the nice discussion! >From my point of view, I very much like the idea of putting connectors in a separate repository. But I would argue it should be part of Apache Flink, similar to flink-statefun, flink-ml, etc. I share many of the reasons for that: - As argued many times, reduces complexity of the Flink repo, increases response times of CI, etc. - Much lower barrier of contribution, because an unstable connector would not de-stabilize the whole build. Of course, we would need to make sure we set this up the right way, with connectors having individual CI runs, build status, etc. But it certainly seems possible. I would argue some points a bit different than some cases made before: (a) I believe the separation would increase connector stability. Because it really forces us to work with the connectors against the APIs like any external developer. A mono repo is somehow the wrong thing if you in practice want to actually guarantee stable internal APIs at some layer. Because the mono repo makes it easy to just change something on both sides of the API (provider and consumer) seamlessly. Major refactorings in Flink need to keep all connector API contracts intact, or we need to have a new version of the connector API. (b) We may even be able to go towards more lightweight and automated releases over time, even if we stay in Apache Flink with that repo. This isn't yet fully aligned with the Apache release policies, yet, but there are board discussions about whether there can be bot-triggered releases (by dependabot) and how that could fit into the Apache process. This doesn't seem to be quite there just yet, but seeing that those start is a good sign, and there is a good chance we can do some things there. I am not sure whether we should let bots trigger releases, because a final human look at things isn't a bad thing, especially given the popularity of software supply chain attacks recently. I do share Chesnay's concerns about complexity in tooling, though. Both release tooling and test tooling. They are not incompatible with that approach, but they are a task we need to tackle during this change which will add additional work. On Tue, Oct 26, 2021 at 10:31 AM Arvid Heise wrote: > Hi folks, > > I think some questions came up and I'd like to address the question of the > timing. > > Could you clarify what release cadence you're thinking of? There's quite > > a big range that fits "more frequent than Flink" (per-commit, daily, > > weekly, bi-weekly, monthly, even bi-monthly). > > The short answer is: as often as needed: > - If there is a CVE in a dependency and we need to bump it - release > immediately. > - If there is a new feature merged, release soonish. We may collect a few > successive features before a release. > - If there is a bugfix, release immediately or soonish depending on the > severity and if there are workarounds available. > > We should not limit ourselves; the whole idea of independent releases is > exactly that you release as needed. There is no release planning or > anything needed, you just go with a release as if it was an external > artifact. > > (1) is the connector API already stable? > > From another discussion thread [1], connector API is far from stable. > > Currently, it's hard to build connectors against multiple Flink versions. > > There are breaking API changes both in 1.12 -> 1.13 and 1.13 -> 1.14 and > > maybe also in the future versions, because Table related APIs are still > > @PublicEvolving and new Sink API is still @Experimental. > > > > The question is: what is stable in an evolving system? We recently > discovered that the old SourceFunction needed to be refined such that > cancellation works correctly [1]. So that interface is in Flink since 7 > years, heavily used also outside, and we still had to change the contract > in a way that I'd expect any implementer to recheck their implementation. > It might not be necessary to change anything and you can probably change > the the code for all Flink versions but still, the interface was not stable > in the closest sense. > > If we focus just on API changes on the unified interfaces, then we expect > one more change to Sink API to support compaction. For Table API, there > will most likely also be some changes in 1.15. So we could wait for 1.15. > > But I'm questioning if that's really necessary because we will add more > functionality beyond 1.15 without breaking API. For example, we may add > more unified connector metrics. If you want to use it in your connector, > you have to support multiple Flink versions anyhow. So rather then focusing > the discussion on "when is stuff stable", I'd rather focus on "how can we > support building connectors against multiple Flink versions" and make it as > painless as possible. > > Chesnay pointed out to use different branches for different Flink versions > which sounds like a good suggestion. With a mono-repo, we can't use > branches differently
[jira] [Created] (FLINK-24682) Unify the -C option behavior in both yarn application and per-job mode
Biao Geng created FLINK-24682:
-
Summary: Unify the -C option behavior in both yarn application and
per-job mode
Key: FLINK-24682
URL: https://issues.apache.org/jira/browse/FLINK-24682
Project: Flink
Issue Type: Improvement
Components: Deployment / YARN
Affects Versions: 1.12.3
Environment: flink 1.12.3
yarn 2.8.5
Reporter: Biao Geng
Recently, when switching the job submission mode from per-job mode to
application mode on yarn, we found the behavior of '-C' ('–-classpath') is
somehow misleading:
In per-job mode, the `main()` method of the program is executed in the local
machine and '-C' option works well when we use it to specify some local user
jars like -C file://xx.jar.
But in application mode, this option works differently: as the `main()` method
will be executed on the job manager in the cluster, it is unclear where the url
like `file://xx.jar` points. It seems that `file://xx.jar` is located
on the job manager machine in the cluster due to the code. If that is true, it
may mislead users as in per-job mode, it refers to the the jars in the client
machine.
In summary, if we can unify the -C option behavior in both yarn application and
per-job mode, it would help users to switch to application mode more smoothly
and more importantly, it makes it much easier to specify some local jars, that
should be loaded by UserClassLoader, on the client machine.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24683) Add Support for Azure Synapse Analytics
Israel Ekpo created FLINK-24683: --- Summary: Add Support for Azure Synapse Analytics Key: FLINK-24683 URL: https://issues.apache.org/jira/browse/FLINK-24683 Project: Flink Issue Type: Improvement Components: API / DataStream Affects Versions: 1.15.0 Reporter: Israel Ekpo Fix For: 1.15.0 The objective of this improvement is to add support for Azure Synapse Analytics so that users can stream data out of Flink into Azure Synapse. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] FLIP-189: SQL Client Usability Improvements
Hi Sergey, Thanks for taking the initiative to create a FLIP and propose improvements on the SQL client. All usability improvements on the SQL client are highly appreciated, especially for new users of Flink. Multi-line support is definitely one of those things I've run into myself. I do think it would be quite nice if there would be some kind of POC which could show (some of) the proposed improvements. Is that something that might be easily feasible? Best regards, Martijn On Thu, 28 Oct 2021 at 11:02, Sergey Nuyanzin wrote: > Hi all, > > I want to start a discussion about FLIP-189: SQL Client Usability > Improvements. > > The main changes in this FLIP: > > - Flink sql client parser improvements so >that sql client does not ask for ; inside a quoted string or a comment > - use prompt to show what sql client is waiting for > - introduce syntax highlighting > - improve completion > > For more detailed changes, please refer to FLIP-189[1]. > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-189%3A+SQL+Client+Usability+Improvements > > > > Look forward to your feedback. > > -- > Best regards, > Sergey >
[jira] [Created] (FLINK-24684) Port to string casting rules to the new CastRule interface
Francesco Guardiani created FLINK-24684: --- Summary: Port to string casting rules to the new CastRule interface Key: FLINK-24684 URL: https://issues.apache.org/jira/browse/FLINK-24684 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Francesco Guardiani Assignee: Francesco Guardiani -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24685) Use the new casting rules in TableResult#print
Francesco Guardiani created FLINK-24685: --- Summary: Use the new casting rules in TableResult#print Key: FLINK-24685 URL: https://issues.apache.org/jira/browse/FLINK-24685 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Francesco Guardiani Assignee: Francesco Guardiani -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24686) Make doc clear on AsyncFunction::timeout() overriding
Jun Qin created FLINK-24686:
---
Summary: Make doc clear on AsyncFunction::timeout() overriding
Key: FLINK-24686
URL: https://issues.apache.org/jira/browse/FLINK-24686
Project: Flink
Issue Type: Improvement
Components: Documentation
Reporter: Jun Qin
Assignee: Jun Qin
Sometimes, a user overrides {{AsyncFunction::timeout()}} with an empty method
or with only logging code. This causes the timeout does not signal back to the
framework and job stuck especially when using {{orderedWait()}}. Opening this
Jira to make the doc clear on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24687) flink-table uber jar should not include flink-connector-files dependency
Thomas Weise created FLINK-24687: Summary: flink-table uber jar should not include flink-connector-files dependency Key: FLINK-24687 URL: https://issues.apache.org/jira/browse/FLINK-24687 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.13.3, 1.14.0 Reporter: Thomas Weise flink-table-common currently depends on flink-connector-files due to BulkReaderFormatFactory. Since the connector isn't relocated and the jar file is part of the distributions lib directory, it conflicts with application packaged connector dependencies, including flink-connector-base -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Shading in flink-table-blink and upgrade compatibility issue
Hi Timo, Thanks for the reply, I filed https://issues.apache.org/jira/browse/FLINK-24687 The file connector dependency is via flink-table-common and there are main source references (see below). Thomas [ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project flink-table-common: Compilation failure: Compilation failure: [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/FileSystemValidator.java:[22,44] package org.apache.flink.connector.file.sink does not exist [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/FileSystemValidator.java:[23,43] package org.apache.flink.connector.file.src does not exist [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[23,43] package org.apache.flink.connector.file.src does not exist [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[24,50] package org.apache.flink.connector.file.src.reader does not exist [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[35,39] cannot find symbol [ERROR] symbol: class BulkFormat [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/factories/BulkReaderFormatFactory.java:[35,59] cannot find symbol [ERROR] symbol: class FileSourceSplit [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[22,43] package org.apache.flink.connector.file.src does not exist [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[23,50] package org.apache.flink.connector.file.src.reader does not exist [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[31,63] cannot find symbol [ERROR] symbol: class BulkFormat [ERROR] /Users/thomas/src/flink/flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/format/BulkDecodingFormat.java:[31,77] cannot find symbol [ERROR] symbol: class FileSourceSplit On Tue, Oct 26, 2021 at 5:13 AM Timo Walther wrote: > > Hi Thomas, > > thanks for your feedback. > > The error that you are experiencing is definitely a bug in 1.13.3 and > the missing method should be reintroduced in the next patch version to > make code compiled against older patch versions run again. > > Regarding the discussion points: > > I agree that flink-table-blink uber jar should not contain a dependency > to flink-connector-base. Even filesystem connectors should be optional > and put in a dedicated module that is not in /lib by default. > > With having the flink-table-blink uber jar in /lib we would like to > improve the SQL experience as this API is as important as the DataStream > API nowadays. But the depenencies should be minimal nevertheless. > > Regards, > Timo > > > On 23.10.21 00:59, Thomas Weise wrote: > > Hi, > > > > As part of upgrading to Flink 1.13.3 from 1.13.2 we run into the > > following problem with KafkaSource (Flink distribution is 1.13.2 and > > the application was built with 1.13.3): > > > > java.lang.NoSuchMethodError: 'void > > org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager.(org.apache.flink.connector.base.source.reader.synchronization.FutureCompletingBlockingQueue, > > java.util.function.Supplier, java.util.function.Consumer)' > > at > > org.apache.flink.connector.kafka.source.reader.fetcher.KafkaSourceFetcherManager.(KafkaSourceFetcherManager.java:67) > > at > > org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:160) > > at > > org.apache.flink.connector.kafka.source.KafkaSource.createReader(KafkaSource.java:127) > > > > It turns out that flink-table-blink_2.12-1.13.2.jar contains > > flink-connector-base and because that jar is under lib the 1.13.2 > > connector base gets picked up instead of the one bundled in the > > application jar. > > > > (The constructor in > > org.apache.flink.connector.base.source.reader.fetcher.SingleThreadFetcherManager > > was added in 1.13.3.) > > > > There are a few points I would like to discuss: > > > > 1) Version compatibility: A *patch* version should ideally not > > introduce such a change, it should be forward and backward compatible. > > Hopefully this will be the case after 1.14 with stable source API. > > 2) flink-table-blink - if it is meant to be self contained and usable > > as a library - should not leak its shaded dependencies. It contains > > FileSource and other deps from Flink, can those be relocated? > > 3) Do we need flink-table
[jira] [Created] (FLINK-24688) yarn.application-attempt-failures-validity-interval link url is invalid
baizhendong created FLINK-24688: --- Summary: yarn.application-attempt-failures-validity-interval link url is invalid Key: FLINK-24688 URL: https://issues.apache.org/jira/browse/FLINK-24688 Project: Flink Issue Type: Bug Components: Deployment / YARN Affects Versions: 1.14.0, 1.13.0, 1.12.0, 1.11.0, 1.10.0, 1.9.0 Reporter: baizhendong Fix For: 1.9.4 yarn.application-attempt-failures-validity-interval property was added in YarnConfigOptions since [FLINK-12472][yarn], but now the link `https://hortonworks.com/blog/apache-hadoop-yarn-hdp-2-2-fault-tolerance-features-long-running-services/` is not available.So we should update this link. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24689) Add log file's time info in loglist
Fei Feng created FLINK-24689: Summary: Add log file's time info in loglist Key: FLINK-24689 URL: https://issues.apache.org/jira/browse/FLINK-24689 Project: Flink Issue Type: Improvement Components: Runtime / Web Frontend Affects Versions: 1.14.0, 1.12.2 Reporter: Fei Feng Fix For: 1.14.0 Flink web ui support show all log file info in log dir, but only has file name and file size now。 When I search and locate problem, I don't now which one log file that I should open(for example: taskmanager.log rotated and retained multi history log file)。 If there is a time info (for example mtime) in loglist's view, it will guide me to choose correct log file and analyse problem exactly and quickly。or I must open every log file inefficiently If no one made this improvement , I will be happy to fix it -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24690) Clarification of buffer size threshold calculation in BufferDebloater
Anton Kalashnikov created FLINK-24690: - Summary: Clarification of buffer size threshold calculation in BufferDebloater Key: FLINK-24690 URL: https://issues.apache.org/jira/browse/FLINK-24690 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.14.0 Reporter: Anton Kalashnikov It looks like that the variable `skipUpdate` in BufferDebloater#recalculateBufferSize is calculated in a not obvious way. For example if `taskmanager.network.memory.buffer-debloat.threshold-percentages` is set to 50(means 50%) then it will be something like: * 32000 -> 16000(possible) * 32000 -> 17000(not possible) * 16000 -> 24000(not possible) - but 16000 + 50% = 24000 * 16000 -> 32000(only this possible) This happens because the algorithm takes into account only the largest value. So in example of `16000 -> 24000` it would calculate 50% of 24000 so only transit from 12000 -> 24000 possible. In general, this approach is not so bad especially on small values (instead of 256 ->374, the minimum possible transit is 256 -> 512). But we should clarify it somewhere with test or javadoc or both. Also, we can discuss changing this algorithm to a more natural implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-24691) FLINK SQL SUM() causes a precision error
Zongwen Li created FLINK-24691:
--
Summary: FLINK SQL SUM() causes a precision error
Key: FLINK-24691
URL: https://issues.apache.org/jira/browse/FLINK-24691
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.14.0
Environment: flink release-1.14.0
Reporter: Zongwen Li
code:
{code:java}
tableEnv.executeSql("select sum(cast( 1.03520274 as decimal(32, 8))) as num
").print();
{code}
expected:
1.03520274
actual:
1.03520270
This function is normal in version 1.13.x.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Re: [NOTICE] CiBot improvements
Hi Chesnay, I meet with something weird. I submit a pull request https://github.com/apache/flink/pull/17571, it trigger a ci ( https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25470&view=results) which fails at first time because of python module failure. I try to run 'run azure' for two times, nothing happened which is different from the behavior in the past. I ask some other guys, they said they also meet with this problem. Please help me check whether it is expected, thanks a lot. [image: image.png] Best, JING ZHANG Yuepeng Pan 于2021年10月12日周二 下午8:08写道： > Thank you for your effort! > Best, > Roc > > At 2021-10-12 14:07:18, "Arvid Heise" wrote: > >Awesome! > > > >On Tue, Oct 12, 2021 at 3:11 AM Guowei Ma wrote: > > > >> Thanks for your effort! > >> > >> Best, > >> Guowei > >> > >> > >> On Mon, Oct 11, 2021 at 9:26 PM Stephan Ewen wrote: > >> > >> > Great initiative, thanks for doing this! > >> > > >> > On Mon, Oct 11, 2021 at 10:52 AM Till Rohrmann > >> > wrote: > >> > > >> > > Thanks a lot for this effort Chesnay! The improvements sound really > >> good. > >> > > > >> > > Cheers, > >> > > Till > >> > > > >> > > On Mon, Oct 11, 2021 at 8:46 AM David Morávek > wrote: > >> > > > >> > > > Nice! Thanks for the effort Chesnay, this is really a huge step > >> > forward! > >> > > > > >> > > > Best, > >> > > > D. > >> > > > > >> > > > On Mon, Oct 11, 2021 at 6:02 AM Xintong Song < > tonysong...@gmail.com> > >> > > > wrote: > >> > > > > >> > > > > Thanks for the effort, @Chesnay. This is super helpful. > >> > > > > > >> > > > > @Jing, > >> > > > > Every push to the PR branch should automatically trigger an > entire > >> > new > >> > > > > build. `@flinkbot run azure` should only be used when you want > to > >> > > re-run > >> > > > > the failed stages without changing the PR. E.g., when running > into > >> > > known > >> > > > > unstable cases that are unrelated to the PR. > >> > > > > > >> > > > > Thank you~ > >> > > > > > >> > > > > Xintong Song > >> > > > > > >> > > > > > >> > > > > > >> > > > > On Mon, Oct 11, 2021 at 11:45 AM JING ZHANG < > beyond1...@gmail.com> > >> > > > wrote: > >> > > > > > >> > > > > > Hi Chesnay, > >> > > > > > Thanks for the effort. It is a very useful improvement. > >> > > > > > I have a minor question. Please forgive me if the question is > too > >> > > > naive. > >> > > > > > Since '@flinkbot run azure' now behaves like "Rerun failed > jobs", > >> > is > >> > > > > there > >> > > > > > any way to trigger an entirely new build? Because some times > I'm > >> > not > >> > > > sure > >> > > > > > the passed cases in the last build could still success in the > new > >> > > build > >> > > > > > because of introduced updates in new commit. > >> > > > > > > >> > > > > > Best, > >> > > > > > JING ZHANG > >> > > > > > > >> > > > > > > >> > > > > > Yangze Guo 于2021年10月11日周一 上午10:31写道： > >> > > > > > > >> > > > > > > Thanks for that great job, Chesnay! "Rerun failed jobs" will > >> > help a > >> > > > > lot. > >> > > > > > > > >> > > > > > > Best, > >> > > > > > > Yangze Guo > >> > > > > > > > >> > > > > > > On Sun, Oct 10, 2021 at 4:56 PM Chesnay Schepler < > >> > > ches...@apache.org > >> > > > > > >> > > > > > > wrote: > >> > > > > > > > > >> > > > > > > > I made a number of changes to the CiBot over the weekend. > >> > > > > > > > > >> > > > > > > > - '@flinkbot run azure' previously triggered an entirely > new > >> > > build > >> > > > > > based > >> > > > > > > > on the last completed one. It now instead retries the last > >> > > > completed > >> > > > > > > > build, only running the jobs that actually failed. It > >> basically > >> > > > > behaves > >> > > > > > > > like the "Rerun failed jobs" button in the Azure UI. > >> > > > > > > > - various optimizations to increase responsiveness > (primarily > >> > by > >> > > > > doing > >> > > > > > > > significantly less unnecessary work / requests to GH) > >> > > > > > > > - removed TravisCI support (since we no longer support a > >> > release > >> > > > that > >> > > > > > > > used Travis) > >> > > > > > > > > >> > > > > > > > Please ping me if you spot anything weird. > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > >> >
