Re: [DISCUSS] Have foreground processes also create log files
Just for clarifications and as Yang already pointed out: The discussion here is about also creating the log, out and err files as well as keeping writing to STDOUT and STDERR. Hence, there should be no regression for K8s users. The main problem, as Chesnay pointed out, could be the increased disk usage by creating these files. Cheers, Till On Wed, May 6, 2020 at 5:10 AM Yang Wang wrote: > Hi Thomas Weise, > > I am not sure why this change will break the existing K8s/container-based > setup. > Since it will output the logs to stdout and log files at the same time, you > could still > use the `kubectl logs` to view the logs. And log collection could just work > as before. > > What we could benefit from this is the logs could also be accessed via > Flink web > dashboard. I think it is more convenient when the users do not have the > permissions > to execute `kubectl`. > > > @Till Rohrmann , it will be great if we could use > tee > to side output logs to file and avoid > duplication. I will have a look. > > > Best, > Yang > > Thomas Weise 于2020年5月6日周三 上午1:40写道： > > > The downside of change to the default behavior is the breakage of > existing > > k8s or other container-based production setups. > > > > Logs that are emitted to stdout are visible through kubectl logs and > > infrastructure is setup for log collection. > > > > Thanks, > > Thomas > > > > > > On Tue, May 5, 2020 at 6:31 AM Arvid Heise wrote: > > > > > From my previous experience with K8s, I'd assume that the cluster > itself > > > already has some ELK attached to it and all stdout/err is collected > > > automatically. > > > > > > So if you want to also add logs, I'd make that configurable and I'm > torn > > > what the default should be as both make sense. > > > > > > However, thinking a bit further, it sounds as if the current way is > > already > > > very inconsistent. We should probably either have log files for > > everything > > > or log everything to stdout/err and let K8s deal with it. And then we > > > should have some configuration to toggle between the modes. > > > > > > On Tue, May 5, 2020 at 2:50 PM Stephan Ewen wrote: > > > > > > > @Patrick could you chime in? > > > > > > > > We should at least understand the original motivation before simply > > > > changing the way it works. > > > > > > > > On Tue, May 5, 2020 at 2:49 PM Stephan Ewen > wrote: > > > > > > > > > I think Patrick originally introduced the foreground mode, and I > > > believe > > > > > it had indeed something to do with container use and logging. > > > > > > > > > > IIRC the default assumption in docker and Kubernetes is that the > logs > > > > come > > > > > on stdout (or stderr) so after "principle of least astonishment" > the > > > idea > > > > > was to give a similar experience with Flink. > > > > > > > > > > On Tue, May 5, 2020 at 10:49 AM David Anderson < > > da...@alpinegizmo.com> > > > > > wrote: > > > > > > > > > >> I like this idea because it should improve the experience (and > > reduce > > > > >> confusion) for folks having their first flink experience via one > of > > > the > > > > >> docker playgrounds. Right now it gives the impression that > something > > > is > > > > >> broken out-of-the-box. > > > > >> > > > > >> Regards, > > > > >> David > > > > >> > > > > >> On Mon, May 4, 2020 at 6:01 PM Till Rohrmann < > trohrm...@apache.org> > > > > >> wrote: > > > > >> > > > > >> > Hi everyone, > > > > >> > > > > > >> > thanks for starting this discussion Chesnay. > > > > >> > > > > > >> > I think it would be nice if we also displayed the logs when > > starting > > > > the > > > > >> > process in the foreground. > > > > >> > > > > > >> > The repercussions could be mitigated if the default logger > > > > >> configurations > > > > >> > would contain file rolling with a max log file size. > > > > >> > > > > > >> > @Yang I think there are solutions how to redirect stdout and > > stderr > > > > into > > > > >> > separate files using tee without duplication [1]. > > > > >> > > > > > >> > [1] http://www.softpanorama.org/Tools/tee.shtml > > > > >> > > > > > >> > Cheers, > > > > >> > Till > > > > >> > > > > > >> > On Wed, Apr 29, 2020 at 4:28 AM Yang Wang < > danrtsey...@gmail.com> > > > > >> wrote: > > > > >> > > > > > >> > > Thanks for Chesnay starting this discussion. > > > > >> > > > > > > >> > > In FLINK-17166 implementation[1], we are trying to use "tee" > > > instead > > > > >> of > > > > >> > > introducing the stream redirection(redirect the out/err to > > files). > > > > >> > However, > > > > >> > > a side effect is that the logging will be duplicated both in > > .log > > > > and > > > > >> > .out > > > > >> > > files. > > > > >> > > Then it may consume more disk space. However it is not a very > > > > critical > > > > >> > > problem since we could use log4j/logback configuration to > > control > > > > the > > > > >> > > rolling > > > > >> > > files and max size. > > > > >> > > > > > > >> > > Also, it only happens in docker/K8s deployment. For YARN/Mesos >
Re: [DISCUSS] Have foreground processes also create log files
Thanks for clarifying, that was not clear to me. That sounds fine to me, given that it just adds extra information, not changes existing one. On Wed, May 6, 2020 at 9:06 AM Till Rohrmann wrote: > Just for clarifications and as Yang already pointed out: The discussion > here is about also creating the log, out and err files as well as keeping > writing to STDOUT and STDERR. > > Hence, there should be no regression for K8s users. The main problem, as > Chesnay pointed out, could be the increased disk usage by creating these > files. > > Cheers, > Till > > On Wed, May 6, 2020 at 5:10 AM Yang Wang wrote: > > > Hi Thomas Weise, > > > > I am not sure why this change will break the existing K8s/container-based > > setup. > > Since it will output the logs to stdout and log files at the same time, > you > > could still > > use the `kubectl logs` to view the logs. And log collection could just > work > > as before. > > > > What we could benefit from this is the logs could also be accessed via > > Flink web > > dashboard. I think it is more convenient when the users do not have the > > permissions > > to execute `kubectl`. > > > > > > @Till Rohrmann , it will be great if we could use > > tee > > to side output logs to file and avoid > > duplication. I will have a look. > > > > > > Best, > > Yang > > > > Thomas Weise 于2020年5月6日周三 上午1:40写道： > > > > > The downside of change to the default behavior is the breakage of > > existing > > > k8s or other container-based production setups. > > > > > > Logs that are emitted to stdout are visible through kubectl logs and > > > infrastructure is setup for log collection. > > > > > > Thanks, > > > Thomas > > > > > > > > > On Tue, May 5, 2020 at 6:31 AM Arvid Heise > wrote: > > > > > > > From my previous experience with K8s, I'd assume that the cluster > > itself > > > > already has some ELK attached to it and all stdout/err is collected > > > > automatically. > > > > > > > > So if you want to also add logs, I'd make that configurable and I'm > > torn > > > > what the default should be as both make sense. > > > > > > > > However, thinking a bit further, it sounds as if the current way is > > > already > > > > very inconsistent. We should probably either have log files for > > > everything > > > > or log everything to stdout/err and let K8s deal with it. And then we > > > > should have some configuration to toggle between the modes. > > > > > > > > On Tue, May 5, 2020 at 2:50 PM Stephan Ewen > wrote: > > > > > > > > > @Patrick could you chime in? > > > > > > > > > > We should at least understand the original motivation before simply > > > > > changing the way it works. > > > > > > > > > > On Tue, May 5, 2020 at 2:49 PM Stephan Ewen > > wrote: > > > > > > > > > > > I think Patrick originally introduced the foreground mode, and I > > > > believe > > > > > > it had indeed something to do with container use and logging. > > > > > > > > > > > > IIRC the default assumption in docker and Kubernetes is that the > > logs > > > > > come > > > > > > on stdout (or stderr) so after "principle of least astonishment" > > the > > > > idea > > > > > > was to give a similar experience with Flink. > > > > > > > > > > > > On Tue, May 5, 2020 at 10:49 AM David Anderson < > > > da...@alpinegizmo.com> > > > > > > wrote: > > > > > > > > > > > >> I like this idea because it should improve the experience (and > > > reduce > > > > > >> confusion) for folks having their first flink experience via one > > of > > > > the > > > > > >> docker playgrounds. Right now it gives the impression that > > something > > > > is > > > > > >> broken out-of-the-box. > > > > > >> > > > > > >> Regards, > > > > > >> David > > > > > >> > > > > > >> On Mon, May 4, 2020 at 6:01 PM Till Rohrmann < > > trohrm...@apache.org> > > > > > >> wrote: > > > > > >> > > > > > >> > Hi everyone, > > > > > >> > > > > > > >> > thanks for starting this discussion Chesnay. > > > > > >> > > > > > > >> > I think it would be nice if we also displayed the logs when > > > starting > > > > > the > > > > > >> > process in the foreground. > > > > > >> > > > > > > >> > The repercussions could be mitigated if the default logger > > > > > >> configurations > > > > > >> > would contain file rolling with a max log file size. > > > > > >> > > > > > > >> > @Yang I think there are solutions how to redirect stdout and > > > stderr > > > > > into > > > > > >> > separate files using tee without duplication [1]. > > > > > >> > > > > > > >> > [1] http://www.softpanorama.org/Tools/tee.shtml > > > > > >> > > > > > > >> > Cheers, > > > > > >> > Till > > > > > >> > > > > > > >> > On Wed, Apr 29, 2020 at 4:28 AM Yang Wang < > > danrtsey...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > > >> > > Thanks for Chesnay starting this discussion. > > > > > >> > > > > > > > >> > > In FLINK-17166 implementation[1], we are trying to use "tee" > > > > instead > > > > > >> of > > > > > >> > > introducing the stream redirection(redirect the out/err to > > > files). > > > > >
Re: [VOTE] Release 1.10.1, release candidate #2
Hi all, I wonder if we could also include FLINK-17313 which I backported into 1.10 branch yesterday. Best, Dawid On 06/05/2020 07:26, Yu Li wrote: > Thanks Till and Thomas, will include fix for both FLINK-17496 and > FLINK-17514 in the next RC. > > Best Regards, > Yu > > > On Tue, 5 May 2020 at 22:10, Thomas Weise wrote: > >> I opened a PR to backport the Kinesis fix - it would be nice to include if >> there is another RC: >> >> https://github.com/apache/flink/pull/11998 >> >> >> On Tue, May 5, 2020 at 4:50 AM Till Rohrmann wrote: >> >>> I've opened a PR for FLINK-17514. >>> >>> Cheers, >>> Till >>> >>> On Tue, May 5, 2020 at 11:46 AM Yu Li wrote: >>> Thanks all for the efforts. I'm hereby canceling the vote due to FLINK-17514, will prepare another >> RC after the issue is fixed (hopefully soon). Best Regards, Yu On Tue, 5 May 2020 at 17:28, Till Rohrmann >> wrote: > I agree with Aljoscha. This is something we should fix since this is >>> very > important for Flink's stability. I will prepare a fix for the >> problem. > Cheers, > Till > > On Tue, May 5, 2020 at 10:30 AM Congxian Qiu wrote: > >> +1 （no-binding) >> - sha and gpg, ok >> - all pom files point to same version, ok >> - build from souce, ok >> - LICENCE, ok >> - run demo in standalone cluster, ok >> >> Best, >> Congxian >> >> >> Aljoscha Krettek 于2020年5月5日周二 下午3:50写道： >> >>> Unfortunately, I found this bug which prevents the TaskCancelerWatchdog >>> [sic] from working: https://issues.apache.org/jira/browse/FLINK-17514. > I >>> think it's quite crucial that this failsafe mechanism works correctly. >>> We should cancel the release and fix it. >>> >>> Best, >>> Aljoscha >>> >>> On 05.05.20 05:55, Hequn Cheng wrote: Thanks a lot for managing the release! +1 (binding) - Go through all new commits for 1.10.1 and spot no new license >> problems. - Built from source archive successfully. - Signatures and hash are correct. - Run SocketWindowWordCount on the local cluster. - Install Python package and run Python WordCount example. - Reviewed website PR Best, Hequn On Sun, May 3, 2020 at 9:10 PM Robert Metzger < >>> rmetz...@apache.org >>> wrote: > Thanks a lot for addressing the issues from the last release > candidate >>> and > creating this one! > > +1 (binding) > > - Started Flink on YARN on Google Cloud DataProc by setting > HADOOP_CLASSPATH > - checked staging repo > > > > On Sat, May 2, 2020 at 6:57 PM Thomas Weise wrote: >> +1 (binding) >> >> Checked signatures and hashes. >> >> Run internal benchmark applications. >> >> I found a regression that was actually introduced with >> 1.10.0, > hence >>> not > a >> blocker for this release: >> >> https://github.com/apache/flink/pull/11975 >> >> Thanks, >> Thomas >> >> >> On Fri, May 1, 2020 at 5:37 AM Yu Li >> wrote: >>> Hi everyone, >>> >>> Please review and vote on the release candidate #2 for >> version >> 1.10.1, > as >>> follows: >>> [ ] +1, Approve the release >>> [ ] -1, Do not approve the release (please provide specific >> comments) >>> >>> The complete staging area is available for your review, >> which >>> includes: >>> * JIRA release notes [1], >>> * the official Apache source release and binary convenience > releases >>> to >> be >>> deployed to dist.apache.org [2], which are signed with the >>> key > with >>> fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3], >>> * all artifacts to be deployed to the Maven Central >> Repository > [4], >>> * source code tag "release-1.10.1-rc2" [5], >>> * website pull request listing the new release and adding >> announcement >> blog >>> post [6]. >>> >>> The vote will be open for at least 72 hours. It is adopted >> by >> majority >>> approval, with at least 3 PMC affirmative votes. >>> >>> Thanks, >>> Yu >>> >>> [1] >>> >>> >> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12346891 >>> [2] > https://dist.apache.org/repos/dist/dev/flink/flink-1.10.1-rc2/ >>> [3] https://dist.apache.org/repos/dist/release/flink/KEYS >>> [4] >>> >> https://repository.apache.org/content/repositories/orgapacheflink-1363/ >>> [5] >>> >>>
The Questions about StreamTableEnvironmentImpl
Hi all: why StreamTableEnvironmentImpl#isEagerOperationTranslation() always return true?
[jira] [Created] (FLINK-17541) Support inline structured types
Timo Walther created FLINK-17541: Summary: Support inline structured types Key: FLINK-17541 URL: https://issues.apache.org/jira/browse/FLINK-17541 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Reporter: Timo Walther Assignee: Timo Walther Many locations in the code base already support structured types. The runtime treats them as row types. However, some final work is needed to support structured types though the stack. We start with inline structured types. Registered structured types in catalog are covered in a different issue. Inline structured types are a prerequisite to enable aggregate functions in FLIP-65 again. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Hierarchies in ConfigOption
Hi, everyone ~ Allows me to share some thoughts here. Personally i think for SQL, "format" is obviously better than "format.name", it is more concise and straight-forward, similar with Presto FORMAT[2] and KSQL VALUE_FORMAT[1]; i think we move from "connector.type" to "connector" for the same reason, the "type" or "name" suffix is implicit, SQL syntax like the DDL is a top-level user API, so from my side keeping good user-friendly syntax is more important. @Timo I'm big +1 for the a good code style guide, but that does not mean we should go for a json-style table options in the DDL, the DDL could have its own contract. Can we move "represent these config options in YAML" to another topic ? Otherwise, how should we handle the "connector" key, should we prefix all the table options with "connector" ? The original inention of FLIP-122 is to remove some redundant prefix/suffix of the table options because they are obviously implicit there, and the "connector." prefix and the ".type" or ".name" suffix are the ones we most want to delete. @Dawid Although ".type" is just another 4 characters, but we force the SQL users to do the thing that is obvious reduadant, i know serialize catalog table to YAML or use the options in DataStream has similar keys request, but they are different use cases that i believe many SQL user would not encounter, that means we force many users to obey rules for cases they would never have. [1] https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/ [2] https://prestodb.io/docs/current/sql/create-table.html Best, Danny Chan 在 2020年5月4日 +0800 PM11:34，Till Rohrmann ，写道： > Hi everyone, > > I like Timo's proposal to organize our configuration more hierarchical > since this is what the coding guide specifies. The benefit I see is that > config options belonging to the same concept will be found in the same > nested object. Moreover, it will be possible to split the configuration > into unrelated parts which are fed to the respective components. That way > one has a much better separation of concern since component A cannot read > the configuration of component B. > > Concerning Timo's last two proposals: > > If fail-on-missing-field is a common configuration shared by all formats, > then I would go with the first option: > > format.kind: json > format.fail-on-missing-field: true > > If fail-on-missing-field is specific for json, then one could go with > > format: json > json.fail-on-missing-field: true > > or > > format.kind: json > format.json.fail-on-missing-field: true > > Cheers, > Till > > > On Fri, May 1, 2020 at 11:55 AM Timo Walther wrote: > > > Hi Jark, > > > > yes, in theory every connector can design options as they like. But for > > user experience and good coding style we should be consistent in Flink > > connectors and configuration. Because implementers of new connectors > > will copy the design of existing ones. > > > > Furthermore, I could image that people in the DataStream API would also > > like to configure their connector based on options in the near future. > > It might be the case that Flink DataStream API connectors will reuse the > > ConfigOptions from Table API for consistency. > > > > I'm favoring either: > > > > format.kind = json > > format.fail-on-missing-field: true > > > > Or: > > > > format = json > > json.fail-on-missing-field: true > > > > Both are valid hierarchies. > > > > Regards, > > Timo > > > > > > On 30.04.20 17:57, Jark Wu wrote: > > > Hi Dawid, > > > > > > I just want to mention one of your response, > > > > > > > What you described with > > > > 'format' = 'csv', > > > > 'csv.allow-comments' = 'true', > > > > 'csv.ignore-parse-errors' = 'true' > > > > would not work though as the `format` prefix is mandatory in the sources > > > as only the properties with format > > > > will be passed to the format factory in majority of cases. We already > > > have some implicit contracts. > > > > > > IIUC, in FLIP-95 and FLIP-122, the property key style are totally decided > > > by connectors, not the framework. > > > So I custom connector can define above properties, and extract the value > > of > > > 'format', i.e. 'csv', to find the format factory. > > > And extract the properties with `csv.` prefix and remove the prefix, and > > > pass the properties (e.g. 'allow-comments' = 'true') > > > into the format factory to create format. > > > > > > So there is no a strict guarantee to have a "nested JSON style" > > properties. > > > Users can still develop a custom connector with this > > > un-hierarchy properties and works well. > > > > > > 'format' = 'json', > > > 'format.fail-on-missing-field' = 'false' > > > > > > Best, > > > Jark > > > > > > > > > On Thu, 30 Apr 2020 at 14:29, Dawid Wysakowicz > > > wrote: > > > > > > > Hi all, > > > > > > > > I'd like to start with a comment that I am ok with the current state of > > > > the FLIP-122 if there is a strong preference for it. Nevertheless I > > still > > > > like the idea of addi
Re: [DISCUSS] Update our Roadmap
Maybe we could say that the new release manager will do this (or at least make sure that updating the roadmap will happen). The community usually tries to find a release manager at the beginning of the release cycle. Cheers, Till On Tue, May 5, 2020 at 5:42 PM Marta Paes Moreira wrote: > I agree with Till — this would make more sense somewhere during the > kick-off of the release cycle. > > > On Tue, May 5, 2020 at 4:44 PM Till Rohrmann wrote: > > > I don't think that a post-release step makes sense since the release > > manager of release X won't necessarily be involved in the high level > > planning of the release X+1. > > > > Cheers, > > Till > > > > On Tue, May 5, 2020 at 2:01 PM Robert Metzger > wrote: > > > > > Making the release managers responsible for keeping the roadmap up to > > date > > > sounds like a good idea to me! > > > > > > We could add this to the "Creating a Flink Release" Wiki page as a > > > post-release step. > > > > > > On Tue, May 5, 2020 at 10:15 AM Marta Paes Moreira < > ma...@ververica.com> > > > wrote: > > > > > > > Thanks for bringing this up, Robert. > > > > > > > > More than updating the roadmap, it would be valuable to discuss how > to > > > make > > > > this update systematic. I took care of the last update back in > > September > > > > and - to be honest - the effort for someone who is not involved in > > > release > > > > planning (or at least for me) feels unnecessarily high. > > > > > > > > One way to ensure that the roadmap is kept fresh could be to make the > > > > update part of the release kick-off (for example, as part of the > > > > responsibilities of the release managers). > > > > > > > > What do you think? > > > > > > > > Marta > > > > > > > > On Mon, May 4, 2020 at 10:35 PM Robert Metzger > > > > wrote: > > > > > > > > > Hey all, > > > > > The roadmap has been updated last in September 2019. There is > quite a > > > bit > > > > > of outdated information on that page. > > > > > > > > > > Is there a volunteer who wants to create a first draft for an > updated > > > > > version? > > > > > > > > > > On Mon, Aug 19, 2019 at 12:14 PM Robert Metzger < > rmetz...@apache.org > > > > > > > > wrote: > > > > > > > > > > > Nice, thanks! Looking forward to the PR > > > > > > > > > > > > On Mon, Aug 19, 2019 at 11:08 AM Marta Paes Moreira < > > > > ma...@ververica.com > > > > > > > > > > > > wrote: > > > > > > > > > > > >> Hey, Robert. > > > > > >> > > > > > >> Updating the roadmap is something I can work on (and also have > on > > my > > > > > >> radar, > > > > > >> moving forward). Already had a quick word with Stephan and he's > > > > > available > > > > > >> to provide support, if needed. > > > > > >> > > > > > >> Marta > > > > > >> > > > > > >> On Sun, Aug 18, 2019 at 4:46 PM Stephan Ewen > > > > wrote: > > > > > >> > > > > > >> > I could help with that. > > > > > >> > > > > > > >> > On Fri, Aug 16, 2019 at 2:36 PM Robert Metzger < > > > rmetz...@apache.org > > > > > > > > > > >> > wrote: > > > > > >> > > > > > > >> > > Flink 1.9 is feature freezed and almost released. > > > > > >> > > I guess it makes sense to update the roadmap on the website > > > again. > > > > > >> > > > > > > > >> > > Who feels like having a good overview of what's coming up? > > > > > >> > > > > > > > >> > > On Tue, May 7, 2019 at 4:33 PM Fabian Hueske < > > fhue...@gmail.com > > > > > > > > > >> wrote: > > > > > >> > > > > > > > >> > > > Yes, that's a very good proposal Jark. > > > > > >> > > > +1 > > > > > >> > > > > > > > > >> > > > Best, Fabian > > > > > >> > > > > > > > > >> > > > Am Mo., 6. Mai 2019 um 16:33 Uhr schrieb Till Rohrmann < > > > > > >> > > > trohrm...@apache.org > > > > > >> > > > >: > > > > > >> > > > > > > > > >> > > > > I think this is a good idea Jark. Putting the last > update > > > date > > > > > on > > > > > >> the > > > > > >> > > > > roadmap would also force us to regularly update it. > > > > > >> > > > > > > > > > >> > > > > Cheers, > > > > > >> > > > > Till > > > > > >> > > > > > > > > > >> > > > > On Mon, May 6, 2019 at 4:14 AM Jark Wu < > imj...@gmail.com> > > > > > wrote: > > > > > >> > > > > > > > > > >> > > > > > Hi, > > > > > >> > > > > > > > > > > >> > > > > > One suggestion for the roadmap: > > > > > >> > > > > > > > > > > >> > > > > > Shall we add a `latest-update-time` to the top of > > Roadmap > > > > > page? > > > > > >> So > > > > > >> > > that > > > > > >> > > > > > users can know this is a up-to-date Roadmap. > > > > > >> > > > > > > > > > > >> > > > > > On Thu, 2 May 2019 at 04:49, Bowen Li < > > > bowenl...@gmail.com> > > > > > >> wrote: > > > > > >> > > > > > > > > > > >> > > > > > > +1 > > > > > >> > > > > > > > > > > > >> > > > > > > On Mon, Apr 29, 2019 at 11:41 PM jincheng sun < > > > > > >> > > > > sunjincheng...@gmail.com> > > > > > >> > > > > > > wrote: > > > > > >> > > > > > > > > > > > >> > > > > > > > Hi Jeff&Fabian, > > > > > >> > > > > > > > > > > > > >> > > > > > > > I have open the PR about add Python Table API > > section >
Re: What is the RocksDB local directory in flink checkpointing?
Hi LakeShen, `state.backend.rocksdb.localdir` defines the directory in which RocksDB will store its local files. Local files are RocksDB's SST and metadata files for example. This directory does not need to be persisted. If the config option is not configured, then it will use the nodes temporary file directory. Cheers, Till On Wed, May 6, 2020 at 6:07 AM LakeShen wrote: > Hi community, > > Now I have a question about flink checkpoint local directory , our flink > version is 1.6, job mode is > > flink on yarn per job . I saw the flink source code , and I find the flink > checkpoint local directory is > > /tmp when you didn't config the "state.backend.rocksdb.localdir". But I go > into the /tmp dir ,I > > couldn't find the flink checkpoint state local directory. > > What is the RocksDB local directory in flink checkpointing? I am looking > forward to your reply. > > Best, > LakeShen >
Re: [VOTE] Release 1.10.1, release candidate #2
I've merged the fix for FLINK-17514. Cheers, Till On Wed, May 6, 2020 at 10:53 AM Dawid Wysakowicz wrote: > Hi all, > > I wonder if we could also include FLINK-17313 which I backported into > 1.10 branch yesterday. > > Best, > > Dawid > > On 06/05/2020 07:26, Yu Li wrote: > > Thanks Till and Thomas, will include fix for both FLINK-17496 and > > FLINK-17514 in the next RC. > > > > Best Regards, > > Yu > > > > > > On Tue, 5 May 2020 at 22:10, Thomas Weise wrote: > > > >> I opened a PR to backport the Kinesis fix - it would be nice to include > if > >> there is another RC: > >> > >> https://github.com/apache/flink/pull/11998 > >> > >> > >> On Tue, May 5, 2020 at 4:50 AM Till Rohrmann > wrote: > >> > >>> I've opened a PR for FLINK-17514. > >>> > >>> Cheers, > >>> Till > >>> > >>> On Tue, May 5, 2020 at 11:46 AM Yu Li wrote: > >>> > Thanks all for the efforts. > > I'm hereby canceling the vote due to FLINK-17514, will prepare another > >> RC > after the issue is fixed (hopefully soon). > > Best Regards, > Yu > > > On Tue, 5 May 2020 at 17:28, Till Rohrmann > >> wrote: > > I agree with Aljoscha. This is something we should fix since this is > >>> very > > important for Flink's stability. I will prepare a fix for the > >> problem. > > Cheers, > > Till > > > > On Tue, May 5, 2020 at 10:30 AM Congxian Qiu > wrote: > > > >> +1 （no-binding) > >> - sha and gpg, ok > >> - all pom files point to same version, ok > >> - build from souce, ok > >> - LICENCE, ok > >> - run demo in standalone cluster, ok > >> > >> Best, > >> Congxian > >> > >> > >> Aljoscha Krettek 于2020年5月5日周二 下午3:50写道： > >> > >>> Unfortunately, I found this bug which prevents the > TaskCancelerWatchdog > >>> [sic] from working: > https://issues.apache.org/jira/browse/FLINK-17514. > > I > >>> think it's quite crucial that this failsafe mechanism works > correctly. > >>> We should cancel the release and fix it. > >>> > >>> Best, > >>> Aljoscha > >>> > >>> On 05.05.20 05:55, Hequn Cheng wrote: > Thanks a lot for managing the release! > > +1 (binding) > > - Go through all new commits for 1.10.1 and spot no new license > >> problems. > - Built from source archive successfully. > - Signatures and hash are correct. > - Run SocketWindowWordCount on the local cluster. > - Install Python package and run Python WordCount example. > - Reviewed website PR > > Best, > Hequn > > On Sun, May 3, 2020 at 9:10 PM Robert Metzger < > >>> rmetz...@apache.org > >>> wrote: > > Thanks a lot for addressing the issues from the last release > > candidate > >>> and > > creating this one! > > > > +1 (binding) > > > > - Started Flink on YARN on Google Cloud DataProc by setting > > HADOOP_CLASSPATH > > - checked staging repo > > > > > > > > On Sat, May 2, 2020 at 6:57 PM Thomas Weise > wrote: > >> +1 (binding) > >> > >> Checked signatures and hashes. > >> > >> Run internal benchmark applications. > >> > >> I found a regression that was actually introduced with > >> 1.10.0, > > hence > >>> not > > a > >> blocker for this release: > >> > >> https://github.com/apache/flink/pull/11975 > >> > >> Thanks, > >> Thomas > >> > >> > >> On Fri, May 1, 2020 at 5:37 AM Yu Li > >> wrote: > >>> Hi everyone, > >>> > >>> Please review and vote on the release candidate #2 for > >> version > >> 1.10.1, > > as > >>> follows: > >>> [ ] +1, Approve the release > >>> [ ] -1, Do not approve the release (please provide specific > >> comments) > >>> > >>> The complete staging area is available for your review, > >> which > >>> includes: > >>> * JIRA release notes [1], > >>> * the official Apache source release and binary convenience > > releases > >>> to > >> be > >>> deployed to dist.apache.org [2], which are signed with the > >>> key > > with > >>> fingerprint D8D3D42E84C753CA5F170BDF93C07902771AB743 [3], > >>> * all artifacts to be deployed to the Maven Central > >> Repository > > [4], > >>> * source code tag "release-1.10.1-rc2" [5], > >>> * website pull request listing the new release and adding > >> announcement > >> blog > >>> post [6]. > >>> > >>> The vote will be open for at least 72 hours. It is adopted > >> by > >> majority > >>> approval, with at least 3 PMC affirmative votes. > >>> > >>> Than
Fw: Re: Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation
-原始邮件- 发件人:"刘大龙" 发送时间:2020-05-06 17:55:25 (星期三) 收件人: "Jark Wu" 抄送: 主题: Re: Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation Thanks for your tuning ideas, I will test it later. Just to emphasize, I use non-mini batch deduplication for tests. -原始邮件- 发件人:"Jark Wu" 发送时间:2020-05-05 10:48:27 (星期二) 收件人: dev 抄送: "刘大龙" , "Yu Li" , "Yun Tang" 主题: Re: Re: The use of state ttl incremental cleanup strategy in sql deduplication resulting in significant performance degradation Hi Andrey, Thanks for the tuning ideas. I will explain the design of deduplication. The mini-batch implementation of deduplication buffers a bundle of input data in heap (Java Map), when the bundle size hit the trigger size or trigger time, the buffered data will be processed together. So we only need to access the state once per key. This is designed for rocksdb statebackend to reduce the frequently accessing, (de)serialization. And yes, this may slow down the checkpoint, but the suggested mini-batch timeout is <= 10s. From our production experience, it doesn't have much impact on checkpoint. Best, Jark On Tue, 5 May 2020 at 06:48, Andrey Zagrebin wrote: Hi lsyldliu, You can try to tune the StateTtlConfig. As the documentation suggests [1] the TTL incremental cleanup can decrease the per record performance. This is the price of the automatic cleanup. If the only thing, which happens mostly in your operator, is working with state then even checking one additional record to cleanup is two times more actions to do. Timer approach was discussed in TTL feature design. It needs an additional implementation and keeps more state but performs only one cleanup action exactly when needed so it is a performance/storage trade-off. Anyways, 20x degradation looks indeed a lot. As a first step, I would suggest to configure the incremental cleanup explicitly in `StateTtlConfigUtil#createTtlConfig` with a less entries to check, e.g. 1 because processFirstRow/processLastRow already access the state twice and do cleanup: .cleanupIncrementally(1, false) Also not sure but depending on the input data, finishBundle can happen mostly during the snapshotting which slows down taking the checkpoint. Could this fail the checkpoint accumulating the backpressure and slowing down the pipeline? Not sure why to keep the deduplication data in a Java map and in Flink state at the same time, why not to keep it only in Flink state and deduplicate on each incoming record? Best, Andrey [1] note 2 in https://ci.apache.org/projects/flink/flink-docs-stable/dev/stream/state/state.html#incremental-cleanup On Wed, Apr 29, 2020 at 11:53 AM 刘大龙 wrote: > > > > > -原始邮件- > > 发件人: "Jark Wu" > > 发送时间: 2020-04-29 14:09:44 (星期三) > > 收件人: dev , "Yu Li" , > myas...@live.com > > 抄送: azagre...@apache.org > > 主题: Re: The use of state ttl incremental cleanup strategy in sql > deduplication resulting in significant performance degradation > > > > Hi lsyldliu, > > > > Thanks for investigating this. > > > > First of all, if you are using mini-batch deduplication, it doesn't > support > > state ttl in 1.9. That's why the tps looks the same with 1.11 disable > state > > ttl. > > We just introduce state ttl for mini-batch deduplication recently. > > > > Regarding to the performance regression, it looks very surprise to me. > The > > performance is reduced by 19x when StateTtlConfig is enabled in 1.11. > > I don't have much experience of the underlying of StateTtlConfig. So I > loop > > in @Yu Li @YunTang in CC who may have more insights > on > > this. > > > > For more information, we use the following StateTtlConfig [1] in blink > > planner: > > > > StateTtlConfig > > .newBuilder(Time.milliseconds(retentionTime)) > > .setUpdateType(StateTtlConfig.UpdateType.OnCreateAndWrite) > > .setStateVisibility(StateTtlConfig.StateVisibility.NeverReturnExpired) > > .build(); > > > > > > Best, > > Jark > > > > > > [1]: > > > https://github.com/apache/flink/blob/master/flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/runtime/util/StateTtlConfigUtil.java#L27 > > > > > > > > > > > > On Wed, 29 Apr 2020 at 11:53, 刘大龙 wrote: > > > > > Hi, all! > > > > > > At flink master branch, we have supported state ttl for sql mini-batch > > > deduplication using incremental cleanup strategy on heap backend, > refer to > > > FLINK-16581. Because I want to test the performance of this feature, > so I > > > compile master branch code and deploy the jar to production > > > environment，then run three types of tests, respectively: > > > > > > > > > > > > > > > flink 1.9.0 release version enable state ttl > > > flink 1.11-snapshot version disable state ttl > > > flink 1.11-snapshot version enable state ttl > > > > > > > > > > > > > > > The test query sql as follows: > > > > > > select order_date, > > > sum(price * amount - goods_all_fav_amt - virtual_money_amt
Re: [DISCUSS] Hierarchies in ConfigOption
Hi, I think Timo proposed a good idea to make both side happy. That is: format = json json.fail-on-missing-field = true json.ignore-parse-error = true value.format = json value.json.fail-on-missing-field = true value.json.ignore-parse-error = true This is a valid hierarchies. Besides, it's more clear that the option belongs to a specific component (i.e. json). This will be more readable when we introducing more formats, e.g. parquet. format = parquet parquet.compression = ... parquet.block.size = ... parquet.page.size = ... is more readable than current style: format = parquet format.compression = ... format.block.size = ... format.page.size = ... To sum up, I'm +1 to use "format = json", "json.fail-on-missing-field = true". Best, Jark On Wed, 6 May 2020 at 17:12, Danny Chan wrote: > Hi, everyone ~ > > Allows me to share some thoughts here. > > Personally i think for SQL, "format" is obviously better than "format.name", > it is more concise and straight-forward, similar with Presto FORMAT[2] and > KSQL VALUE_FORMAT[1]; i think we move from "connector.type" to "connector" > for the same reason, the "type" or "name" suffix is implicit, SQL syntax > like the DDL is a top-level user API, so from my side keeping good > user-friendly syntax is more important. > > @Timo I'm big +1 for the a good code style guide, but that does not mean > we should go for a json-style table options in the DDL, the DDL could have > its own contract. Can we move "represent these config options in YAML" to > another topic ? Otherwise, how should we handle the "connector" key, should > we prefix all the table options with "connector" ? The original inention of > FLIP-122 is to remove some redundant prefix/suffix of the table options > because they are obviously implicit there, and the "connector." prefix and > the ".type" or ".name" suffix are the ones we most want to delete. > > @Dawid Although ".type" is just another 4 characters, but we force the SQL > users to do the thing that is obvious reduadant, i know serialize catalog > table to YAML or use the options in DataStream has similar keys request, > but they are different use cases that i believe many SQL user would not > encounter, that means we force many users to obey rules for cases they > would never have. > > > [1] https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/ > [2] https://prestodb.io/docs/current/sql/create-table.html > > Best, > Danny Chan > 在 2020年5月4日 +0800 PM11:34，Till Rohrmann ，写道： > > Hi everyone, > > > > I like Timo's proposal to organize our configuration more hierarchical > > since this is what the coding guide specifies. The benefit I see is that > > config options belonging to the same concept will be found in the same > > nested object. Moreover, it will be possible to split the configuration > > into unrelated parts which are fed to the respective components. That way > > one has a much better separation of concern since component A cannot read > > the configuration of component B. > > > > Concerning Timo's last two proposals: > > > > If fail-on-missing-field is a common configuration shared by all formats, > > then I would go with the first option: > > > > format.kind: json > > format.fail-on-missing-field: true > > > > If fail-on-missing-field is specific for json, then one could go with > > > > format: json > > json.fail-on-missing-field: true > > > > or > > > > format.kind: json > > format.json.fail-on-missing-field: true > > > > Cheers, > > Till > > > > > > On Fri, May 1, 2020 at 11:55 AM Timo Walther wrote: > > > > > Hi Jark, > > > > > > yes, in theory every connector can design options as they like. But for > > > user experience and good coding style we should be consistent in Flink > > > connectors and configuration. Because implementers of new connectors > > > will copy the design of existing ones. > > > > > > Furthermore, I could image that people in the DataStream API would also > > > like to configure their connector based on options in the near future. > > > It might be the case that Flink DataStream API connectors will reuse > the > > > ConfigOptions from Table API for consistency. > > > > > > I'm favoring either: > > > > > > format.kind = json > > > format.fail-on-missing-field: true > > > > > > Or: > > > > > > format = json > > > json.fail-on-missing-field: true > > > > > > Both are valid hierarchies. > > > > > > Regards, > > > Timo > > > > > > > > > On 30.04.20 17:57, Jark Wu wrote: > > > > Hi Dawid, > > > > > > > > I just want to mention one of your response, > > > > > > > > > What you described with > > > > > 'format' = 'csv', > > > > > 'csv.allow-comments' = 'true', > > > > > 'csv.ignore-parse-errors' = 'true' > > > > > would not work though as the `format` prefix is mandatory in the > sources > > > > as only the properties with format > > > > > will be passed to the format factory in majority of cases. We > already > > > > have some implicit contracts. > > > > > > > > IIUC, in FLIP-95 and FLIP-122, the prop
Re: [DISCUSS] Hierarchies in ConfigOption
Hi, +1 to: format = parquet parquet.compression = ... parquet.block.size = ... parquet.page.size = ... For the formats like parquet and orc, Not just Flink itself, but this way also let Flink keys compatible with the property keys of Hadoop / Hive / Spark. And like Jark said, this way works for Kafka key value too. Best, Jingsong Lee On Wed, May 6, 2020 at 6:19 PM Jark Wu wrote: > Hi, > > I think Timo proposed a good idea to make both side happy. That is: > > format = json > json.fail-on-missing-field = true > json.ignore-parse-error = true > > value.format = json > value.json.fail-on-missing-field = true > value.json.ignore-parse-error = true > > This is a valid hierarchies. Besides, it's more clear that the option > belongs to a specific component (i.e. json). > This will be more readable when we introducing more formats, e.g. parquet. > > format = parquet > parquet.compression = ... > parquet.block.size = ... > parquet.page.size = ... > > is more readable than current style: > > format = parquet > format.compression = ... > format.block.size = ... > format.page.size = ... > > To sum up, I'm +1 to use "format = json", "json.fail-on-missing-field = > true". > > Best, > Jark > > On Wed, 6 May 2020 at 17:12, Danny Chan wrote: > > > Hi, everyone ~ > > > > Allows me to share some thoughts here. > > > > Personally i think for SQL, "format" is obviously better than " > format.name", > > it is more concise and straight-forward, similar with Presto FORMAT[2] > and > > KSQL VALUE_FORMAT[1]; i think we move from "connector.type" to > "connector" > > for the same reason, the "type" or "name" suffix is implicit, SQL syntax > > like the DDL is a top-level user API, so from my side keeping good > > user-friendly syntax is more important. > > > > @Timo I'm big +1 for the a good code style guide, but that does not mean > > we should go for a json-style table options in the DDL, the DDL could > have > > its own contract. Can we move "represent these config options in YAML" to > > another topic ? Otherwise, how should we handle the "connector" key, > should > > we prefix all the table options with "connector" ? The original inention > of > > FLIP-122 is to remove some redundant prefix/suffix of the table options > > because they are obviously implicit there, and the "connector." prefix > and > > the ".type" or ".name" suffix are the ones we most want to delete. > > > > @Dawid Although ".type" is just another 4 characters, but we force the > SQL > > users to do the thing that is obvious reduadant, i know serialize catalog > > table to YAML or use the options in DataStream has similar keys request, > > but they are different use cases that i believe many SQL user would not > > encounter, that means we force many users to obey rules for cases they > > would never have. > > > > > > [1] https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/ > > [2] https://prestodb.io/docs/current/sql/create-table.html > > > > Best, > > Danny Chan > > 在 2020年5月4日 +0800 PM11:34，Till Rohrmann ，写道： > > > Hi everyone, > > > > > > I like Timo's proposal to organize our configuration more hierarchical > > > since this is what the coding guide specifies. The benefit I see is > that > > > config options belonging to the same concept will be found in the same > > > nested object. Moreover, it will be possible to split the configuration > > > into unrelated parts which are fed to the respective components. That > way > > > one has a much better separation of concern since component A cannot > read > > > the configuration of component B. > > > > > > Concerning Timo's last two proposals: > > > > > > If fail-on-missing-field is a common configuration shared by all > formats, > > > then I would go with the first option: > > > > > > format.kind: json > > > format.fail-on-missing-field: true > > > > > > If fail-on-missing-field is specific for json, then one could go with > > > > > > format: json > > > json.fail-on-missing-field: true > > > > > > or > > > > > > format.kind: json > > > format.json.fail-on-missing-field: true > > > > > > Cheers, > > > Till > > > > > > > > > On Fri, May 1, 2020 at 11:55 AM Timo Walther > wrote: > > > > > > > Hi Jark, > > > > > > > > yes, in theory every connector can design options as they like. But > for > > > > user experience and good coding style we should be consistent in > Flink > > > > connectors and configuration. Because implementers of new connectors > > > > will copy the design of existing ones. > > > > > > > > Furthermore, I could image that people in the DataStream API would > also > > > > like to configure their connector based on options in the near > future. > > > > It might be the case that Flink DataStream API connectors will reuse > > the > > > > ConfigOptions from Table API for consistency. > > > > > > > > I'm favoring either: > > > > > > > > format.kind = json > > > > format.fail-on-missing-field: true > > > > > > > > Or: > > > > > > > > format = json > > > > json.fail-on-missing-field: tr
Re: [DISCUSS] Hierarchies in ConfigOption
Cool, so let's go with: format = json json.fail-on-missing-field = true json.ignore-parse-error = true value.format = json value.json.fail-on-missing-field = true value.json.ignore-parse-error = true Regards, Timo On 06.05.20 12:39, Jingsong Li wrote: Hi, +1 to: format = parquet parquet.compression = ... parquet.block.size = ... parquet.page.size = ... For the formats like parquet and orc, Not just Flink itself, but this way also let Flink keys compatible with the property keys of Hadoop / Hive / Spark. And like Jark said, this way works for Kafka key value too. Best, Jingsong Lee On Wed, May 6, 2020 at 6:19 PM Jark Wu wrote: Hi, I think Timo proposed a good idea to make both side happy. That is: format = json json.fail-on-missing-field = true json.ignore-parse-error = true value.format = json value.json.fail-on-missing-field = true value.json.ignore-parse-error = true This is a valid hierarchies. Besides, it's more clear that the option belongs to a specific component (i.e. json). This will be more readable when we introducing more formats, e.g. parquet. format = parquet parquet.compression = ... parquet.block.size = ... parquet.page.size = ... is more readable than current style: format = parquet format.compression = ... format.block.size = ... format.page.size = ... To sum up, I'm +1 to use "format = json", "json.fail-on-missing-field = true". Best, Jark On Wed, 6 May 2020 at 17:12, Danny Chan wrote: Hi, everyone ~ Allows me to share some thoughts here. Personally i think for SQL, "format" is obviously better than " format.name", it is more concise and straight-forward, similar with Presto FORMAT[2] and KSQL VALUE_FORMAT[1]; i think we move from "connector.type" to "connector" for the same reason, the "type" or "name" suffix is implicit, SQL syntax like the DDL is a top-level user API, so from my side keeping good user-friendly syntax is more important. @Timo I'm big +1 for the a good code style guide, but that does not mean we should go for a json-style table options in the DDL, the DDL could have its own contract. Can we move "represent these config options in YAML" to another topic ? Otherwise, how should we handle the "connector" key, should we prefix all the table options with "connector" ? The original inention of FLIP-122 is to remove some redundant prefix/suffix of the table options because they are obviously implicit there, and the "connector." prefix and the ".type" or ".name" suffix are the ones we most want to delete. @Dawid Although ".type" is just another 4 characters, but we force the SQL users to do the thing that is obvious reduadant, i know serialize catalog table to YAML or use the options in DataStream has similar keys request, but they are different use cases that i believe many SQL user would not encounter, that means we force many users to obey rules for cases they would never have. [1] https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/ [2] https://prestodb.io/docs/current/sql/create-table.html Best, Danny Chan 在 2020年5月4日 +0800 PM11:34，Till Rohrmann ，写道： Hi everyone, I like Timo's proposal to organize our configuration more hierarchical since this is what the coding guide specifies. The benefit I see is that config options belonging to the same concept will be found in the same nested object. Moreover, it will be possible to split the configuration into unrelated parts which are fed to the respective components. That way one has a much better separation of concern since component A cannot read the configuration of component B. Concerning Timo's last two proposals: If fail-on-missing-field is a common configuration shared by all formats, then I would go with the first option: format.kind: json format.fail-on-missing-field: true If fail-on-missing-field is specific for json, then one could go with format: json json.fail-on-missing-field: true or format.kind: json format.json.fail-on-missing-field: true Cheers, Till On Fri, May 1, 2020 at 11:55 AM Timo Walther wrote: Hi Jark, yes, in theory every connector can design options as they like. But for user experience and good coding style we should be consistent in Flink connectors and configuration. Because implementers of new connectors will copy the design of existing ones. Furthermore, I could image that people in the DataStream API would also like to configure their connector based on options in the near future. It might be the case that Flink DataStream API connectors will reuse the ConfigOptions from Table API for consistency. I'm favoring either: format.kind = json format.fail-on-missing-field: true Or: format = json json.fail-on-missing-field: true Both are valid hierarchies. Regards, Timo On 30.04.20 17:57, Jark Wu wrote: Hi Dawid, I just want to mention one of your response, What you described with 'format' = 'csv', 'csv.allow-comments' = 'true', 'csv.ignore-parse-errors' = 'true' would not work though as the `for
UpsertStreamTableSink vs OverwritableTableSink
Hi all! While working on a Table Sink implementation for Kudu (key-value store) , we got a bit confused about the expected semantics of UpsertStreamTableSink vs OverwritableTableSink and statements like INSERT vs INSERT OVERWRITE I am wondering what external operation should each incoming record corresponds to? - Insert (fail on duplicate row) - Delete - Upsert Purely looking at the UpsertStreamTableSink we either upsert or delete so every INSERT is pretty much an INSERT OVERWRITE. Still we cannot use INSERT OVERWRITE if we don't implement OverwritableTableSink. So I wonder what implementing OverwritableTableSink is expected to do for an UpsertStreamTableSink. Can someone please help me clarify this so we can get it right? Thanks Gyula
[jira] [Created] (FLINK-17542) Unify slot request timeout handling for streaming and batch tasks
Zhu Zhu created FLINK-17542: --- Summary: Unify slot request timeout handling for streaming and batch tasks Key: FLINK-17542 URL: https://issues.apache.org/jira/browse/FLINK-17542 Project: Flink Issue Type: Sub-task Components: Runtime / Coordination Affects Versions: 1.11.0 Reporter: Zhu Zhu Assignee: Zhu Zhu Fix For: 1.11.0 There are 2 different slot request timeout handling mechanism for batch and streaming tasks. For streaming tasks, the slot request will fail if it is not fulfilled within slotRequestTimeout. For batch tasks, the slot request will be checked periodically to see whether it is fulfillable, and only fails if it has been unfulfillable for a certain period(slotRequestTimeout). With slot marked with whether they will be occupied indefinitely, we can unify these handling. See [FLIP-119|https://cwiki.apache.org/confluence/display/FLINK/FLIP-119+Pipelined+Region+Scheduling#FLIP-119PipelinedRegionScheduling-ExtendedSlotProviderInterface] for more details. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17543) Rerunning failed azure jobs fails when uploading logs
Chesnay Schepler created FLINK-17543:
Summary: Rerunning failed azure jobs fails when uploading logs
Key: FLINK-17543
URL: https://issues.apache.org/jira/browse/FLINK-17543
Project: Flink
Issue Type: Bug
Components: Build System / Azure Pipelines
Reporter: Chesnay Schepler
{code}
No LastRequestResponse on exception ArtifactExistsException: Artifact
logs-ci-tests already exists for build 677.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: UpsertStreamTableSink vs OverwritableTableSink
Hi, Insert overwrite comes from Batch SQL in Hive. It means overwriting whole table/partition instead of overwriting per key. So if "insert overwrite kudu_table", should delete whole table in kudu first, and then insert new data to the table in kudu. The same semantics should be used in streaming jobs, but I don't know if there are any requirements. UpsertStreamTableSink is a way to deal with changelog in streaming jobs, the upsert messages and delete messages are produced by keyed and retracted stream, this is not related to "insert overwrite" grammar. And FYI, upsert stream sink will be unified to "DynamicTableSink" with primary keys in DDL.[1] [1] https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces#FLIP-95:NewTableSourceandTableSinkinterfaces-SinkInterfaces Best, Jingsong Lee On Wed, May 6, 2020 at 8:48 PM Gyula Fóra wrote: > Hi all! > > While working on a Table Sink implementation for Kudu (key-value store) , > we got a bit confused about the expected semantics of UpsertStreamTableSink > vs OverwritableTableSink and statements like INSERT vs INSERT OVERWRITE > > I am wondering what external operation should each incoming record > corresponds to? > - Insert (fail on duplicate row) > - Delete > - Upsert > > Purely looking at the UpsertStreamTableSink we either upsert or delete so > every INSERT is pretty much an INSERT OVERWRITE. Still we cannot use INSERT > OVERWRITE if we don't implement OverwritableTableSink. > > So I wonder what implementing OverwritableTableSink is expected to do for > an UpsertStreamTableSink. > > Can someone please help me clarify this so we can get it right? > > Thanks > Gyula > -- Best, Jingsong Lee
Re: UpsertStreamTableSink vs OverwritableTableSink
Thanks a lot for the detailed explanation, makes complete sense. Gyula On Wed, May 6, 2020 at 3:53 PM Jingsong Li wrote: > Hi, > > Insert overwrite comes from Batch SQL in Hive. > It means overwriting whole table/partition instead of overwriting per key. > So if "insert overwrite kudu_table", should delete whole table in kudu > first, and then insert new data to the table in kudu. > > The same semantics should be used in streaming jobs, but I don't know if > there are any requirements. > > UpsertStreamTableSink is a way to deal with changelog in streaming jobs, > the upsert messages and delete messages are produced by keyed and retracted > stream, this is not related to "insert overwrite" grammar. > And FYI, upsert stream sink will be unified to "DynamicTableSink" with > primary keys in DDL.[1] > > [1] > > https://cwiki.apache.org/confluence/display/FLINK/FLIP-95%3A+New+TableSource+and+TableSink+interfaces#FLIP-95:NewTableSourceandTableSinkinterfaces-SinkInterfaces > > > Best, > Jingsong Lee > > On Wed, May 6, 2020 at 8:48 PM Gyula Fóra wrote: > > > Hi all! > > > > While working on a Table Sink implementation for Kudu (key-value store) , > > we got a bit confused about the expected semantics of > UpsertStreamTableSink > > vs OverwritableTableSink and statements like INSERT vs INSERT OVERWRITE > > > > I am wondering what external operation should each incoming record > > corresponds to? > > - Insert (fail on duplicate row) > > - Delete > > - Upsert > > > > Purely looking at the UpsertStreamTableSink we either upsert or delete so > > every INSERT is pretty much an INSERT OVERWRITE. Still we cannot use > INSERT > > OVERWRITE if we don't implement OverwritableTableSink. > > > > So I wonder what implementing OverwritableTableSink is expected to do for > > an UpsertStreamTableSink. > > > > Can someone please help me clarify this so we can get it right? > > > > Thanks > > Gyula > > > > > -- > Best, Jingsong Lee >
[jira] [Created] (FLINK-17544) NPE JDBCUpsertOutputFormat
John Lonergan created FLINK-17544:
-
Summary: NPE JDBCUpsertOutputFormat
Key: FLINK-17544
URL: https://issues.apache.org/jira/browse/FLINK-17544
Project: Flink
Issue Type: Improvement
Components: Connectors / JDBC
Affects Versions: 1.10.0
Reporter: John Lonergan
Encountered a situation where I get an NPE from JDBCUpsertOutputFormat.
This occurs when close is called before open.
This happened because I had a sink where it had a final field of type
JDBCUpsertOutputFormat.
The open operation of my sink was slow (blocked on something else) and open on
the JDBCUpsertOutputFormat had not yet been called.
In the mean time the job was cancelled, which caused close on my sink to be
called, which then
called close on the JDBCUpsertOutputFormat .
This throws an NPE due to a lack of a guard on an internal field that is only
initialised in the JDBCUpsertOutputFormat open operation.
The close method already guards one potentially null value ..
{code:java}
if (this.scheduledFuture != null) {
{code}
But needs the additional guard below ...
{code:java}
if (jdbcWriter != null) // << THIS LINE NEEDED TO GUARD UNINITIALISE VAR
try {
jdbcWriter.close();
} catch (SQLException e) {
LOG.warn("Close JDBC writer failed.", e);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17545) CLONE - NPE JDBCUpsertOutputFormat
John Lonergan created FLINK-17545:
-
Summary: CLONE - NPE JDBCUpsertOutputFormat
Key: FLINK-17545
URL: https://issues.apache.org/jira/browse/FLINK-17545
Project: Flink
Issue Type: Improvement
Components: Connectors / JDBC
Affects Versions: 1.10.0
Reporter: John Lonergan
Encountered a situation where I get an NPE from JDBCUpsertOutputFormat.
This occurs when close is called before open.
This happened because I had a sink where it had a final field of type
JDBCUpsertOutputFormat.
The open operation of my sink was slow (blocked on something else) and open on
the JDBCUpsertOutputFormat had not yet been called.
In the mean time the job was cancelled, which caused close on my sink to be
called, which then
called close on the JDBCUpsertOutputFormat .
This throws an NPE due to a lack of a guard on an internal field that is only
initialised in the JDBCUpsertOutputFormat open operation.
The close method already guards one potentially null value ..
{code:java}
if (this.scheduledFuture != null) {
{code}
But needs the additional guard below ...
{code:java}
if (jdbcWriter != null) // << THIS LINE NEEDED TO GUARD UNINITIALISE VAR
try {
jdbcWriter.close();
} catch (SQLException e) {
LOG.warn("Close JDBC writer failed.", e);
}
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [DISCUSS] FLIP-84 Feedback Summary
Hi @fhue...@gmail.com @Timo Walther @Dawid Wysakowicz What do you think we limit watermark must be defined on top-level column ? if we do that, we can add an expression column to represent watermark like compute column, An example of all cases: create table MyTable ( f0 BIGINT NOT NULL, f1 ROW, f2 VARCHAR<256>, f3 AS f0 + 1, f4 TIMESTAMP(3) NOT NULL, PRIMARY KEY (f0), UNIQUE (f3, f2), WATERMARK f4 AS f4 - INTERVAL '3' SECOND ) with (...) ++--+---+---+---+--+ | name | type | null | key| compute column | watermark | ++--+---+---+---+--+ | f0 | BIGINT | false | PRI | (NULL) | (NULL) | ++--+---+---+---+--+ | f1 | ROW | true | (NULL) | (NULL) | (NULL) | ++--+---+---+---+--+ | f2 | VARCHAR<256> | true | UNQ | (NULL) | (NULL) | ++--+---+---+---+--+ | f3 | BIGINT | false | UNQ | f0 + 1 | (NULL) | ++--+---+---+---+--+ | f4 | TIMESTAMP(3)| false | (NULL) | (NULL)| f4 - INTERVAL '3' SECOND | ++--+---+---+---+--+ WDYT ? Best, Godfrey godfrey he 于2020年4月30日周四 下午11:57写道： > Hi Fabian, > > the broken example is: > > create table MyTable ( > > f0 BIGINT NOT NULL, > > f1 ROW, > > f2 VARCHAR<256>, > > f3 AS f0 + 1, > > PRIMARY KEY (f0), > > UNIQUE (f3, f2), > > WATERMARK f1.q2 AS (`f1.q2` - INTERVAL '3' SECOND) > > ) with (...) > > > name > > type > > key > > compute column > > watermark > > f0 > > BIGINT NOT NULL > > PRI > > (NULL) > > f1 > > ROW<`q1` STRING, `q2` TIMESTAMP(3)> > > UNQ > > (NULL) > > f1.q2 AS (`f1.q2` - INTERVAL '3' SECOND) > > f2 > > VARCHAR<256> > > (NULL) > > NULL > > f3 > > BIGINT NOT NULL > > UNQ > > f0 + 1 > > > or we add a column to represent nullability. > > name > > type > > null > > key > > compute column > > watermark > > f0 > > BIGINT > > false > > PRI > > (NULL) > > f1 > > ROW<`q1` STRING, `q2` TIMESTAMP(3)> > > true > > UNQ > > (NULL) > > f1.q2 AS (`f1.q2` - INTERVAL '3' SECOND) > > f2 > > VARCHAR<256> > > true > > (NULL) > > NULL > > f3 > > BIGINT > > false > > UNQ > > f0 + 1 > > > > > Hi Jark, > If we can limit watermark must be defined on top-level column, > this will become more simple. > > Best, > Godfrey > > Jark Wu 于2020年4月30日周四 下午11:38写道： > >> Hi, >> >> I'm in favor of Fabian's proposal. >> First, watermark is not a column, but a metadata just like primary key, so >> shouldn't stand with columns. >> Second, AFAIK, primary key can only be defined on top-level columns. >> Third, I think watermark can also follow primary key than only allow to >> define on top-level columns. >> >> I have to admit that in FLIP-66, watermark can define on nested fields. >> However, during implementation, I found that it's too complicated to do >> that. We have refactor time-based physical nodes, >> we have to use code generation to access event-time, we have to refactor >> FlinkTypeFactory to support a complex nested rowtime. >> There is not much value of this feature, but introduce a lot of complexity >> in code base. >> So I think we can force watermark define on top-level columns. If user >> want >> to define on nested columns, >> he/she can use computed column to be a top-level column. >> >> Best, >> Jark >> >> >> On Thu, 30 Apr 2020 at 17:55, Fabian Hueske wrote: >> >> > Hi Godfrey, >> > >> > The formatting of your example seems to be broken. >> > Could you send them again please? >> > >> > Regarding your points >> > > because watermark express can be a sub-column, just like `f1.q2` in >> above >> > example I give. >> > >> > I would put the watermark information in the row of the top-level field >> and >> > indicate to which nested field the watermark refers. >> > Don't we have to solve the same issue for primary keys th
Re: [DISCUSS] FLIP-84 Feedback Summary
Hi Godfrey, This looks good to me. One side note, indicating unique constraints with "UNQ" is probably not enough. There might be multiple unique constraints and users would like to know which field combinations are unique. So in your example above, "UNQ(f2, f3)" might be a better marker. Just as a thought, if we would later add support for watermark on nested columns, we could add a row just for the nested field (in addition to the top-level field) like this: ++---+---+---+-+---+ | f4.nested.rowtime | TIMESTAMP(3)| false | (NULL) | (NULL) | f4.nested.rowtime - INTERVAL '3' SECOND | ++---+---+---+-+---+ Thanks, Fabian Am Mi., 6. Mai 2020 um 17:51 Uhr schrieb godfrey he : > Hi @fhue...@gmail.com @Timo Walther @Dawid > Wysakowicz > What do you think we limit watermark must be defined on top-level column ? > > if we do that, we can add an expression column to represent watermark like > compute column, > An example of all cases: > create table MyTable ( > f0 BIGINT NOT NULL, > f1 ROW, > f2 VARCHAR<256>, > f3 AS f0 + 1, > f4 TIMESTAMP(3) NOT NULL, > PRIMARY KEY (f0), > UNIQUE (f3, f2), > WATERMARK f4 AS f4 - INTERVAL '3' SECOND > ) with (...) > > > ++--+---+---+---+--+ > | name | type | > null | key| compute column | watermark > | > > ++--+---+---+---+--+ > | f0 | BIGINT | > false | PRI | (NULL) | (NULL) > | > > ++--+---+---+---+--+ > | f1 | ROW | true | (NULL) | (NULL) > | (NULL) | > > ++--+---+---+---+--+ > | f2 | VARCHAR<256> | true | > UNQ | (NULL) | (NULL) | > > ++--+---+---+---+--+ > | f3 | BIGINT | > false | UNQ | f0 + 1 | (NULL) > | > > ++--+---+---+---+--+ > | f4 | TIMESTAMP(3)| false | > (NULL) | (NULL)| f4 - INTERVAL '3' SECOND | > > ++--+---+---+---+--+ > > WDYT ? > > Best, > Godfrey > > > > godfrey he 于2020年4月30日周四 下午11:57写道： > >> Hi Fabian, >> >> the broken example is: >> >> create table MyTable ( >> >> f0 BIGINT NOT NULL, >> >> f1 ROW, >> >> f2 VARCHAR<256>, >> >> f3 AS f0 + 1, >> >> PRIMARY KEY (f0), >> >> UNIQUE (f3, f2), >> >> WATERMARK f1.q2 AS (`f1.q2` - INTERVAL '3' SECOND) >> >> ) with (...) >> >> >> name >> >> type >> >> key >> >> compute column >> >> watermark >> >> f0 >> >> BIGINT NOT NULL >> >> PRI >> >> (NULL) >> >> f1 >> >> ROW<`q1` STRING, `q2` TIMESTAMP(3)> >> >> UNQ >> >> (NULL) >> >> f1.q2 AS (`f1.q2` - INTERVAL '3' SECOND) >> >> f2 >> >> VARCHAR<256> >> >> (NULL) >> >> NULL >> >> f3 >> >> BIGINT NOT NULL >> >> UNQ >> >> f0 + 1 >> >> >> or we add a column to represent nullability. >> >> name >> >> type >> >> null >> >> key >> >> compute column >> >> watermark >> >> f0 >> >> BIGINT >> >> false >> >> PRI >> >> (NULL) >> >> f1 >> >> ROW<`q1` STRING, `q2` TIMESTAMP(3)> >> >> true >> >> UNQ >> >> (NULL) >> >> f1.q2 AS (`f1.q2` - INTERVAL '3' SECOND) >> >> f2 >> >> VARCHAR<256> >> >> true >> >> (NULL) >> >> NULL >> >> f3 >> >> BIGINT >> >> false >> >> UNQ >> >> f0 + 1 >> >> >> >> >> Hi Jark, >> If we can limit watermark must be defined on top-level column, >> this will become more simple. >> >> Best, >> Godfrey >> >> Jark Wu 于2020年4月30日周四 下午11:38写道： >> >>> Hi, >>> >>> I'm in favor of Fabian's proposal. >>> First, watermark is not a column, but a metadata just like primary key, >>> so >>> shouldn't stand with columns. >>> Second, AFAIK, primary key can only be defined on top-level columns. >>> Third, I think watermark can also follow primary key
[jira] [Created] (FLINK-17546) Consider setting the number of TM CPU cores to the actual number of cores
Andrey Zagrebin created FLINK-17546: --- Summary: Consider setting the number of TM CPU cores to the actual number of cores Key: FLINK-17546 URL: https://issues.apache.org/jira/browse/FLINK-17546 Project: Flink Issue Type: Improvement Components: Runtime / Configuration Reporter: Andrey Zagrebin So far we do not use CPU cores resource in TaskExecutorResourceSpec. It was a preparation for dynamic slot/resource allocation (FLINK-14187). It is not fully clear how Flink or users would define the number of cores. We could consider setting the number of TM CPU cores to the actual number of cores by default, e.g. got somehow from OS in standalone or container configuration. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17547) Support unaligned checkpoints for records spilled to files
Roman Khachatryan created FLINK-17547: - Summary: Support unaligned checkpoints for records spilled to files Key: FLINK-17547 URL: https://issues.apache.org/jira/browse/FLINK-17547 Project: Flink Issue Type: New Feature Components: Runtime / Task Affects Versions: 1.11.0 Reporter: Roman Khachatryan Assignee: Roman Khachatryan Fix For: 1.11.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17548) Table API Schema from not working
Rafael Wicht created FLINK-17548:
Summary: Table API Schema from not working
Key: FLINK-17548
URL: https://issues.apache.org/jira/browse/FLINK-17548
Project: Flink
Issue Type: Bug
Components: Table SQL / API
Affects Versions: 1.10.0
Reporter: Rafael Wicht
The method 'from' of org.apache.flink.table.descriptors.Schema do not work for
me.
Maybe I miss something.
h3. Code:
{{.withFormat({color:#cc7832}new {color}Json()}}
{{ .failOnMissingField({color:#cc7832}false{color}))}}
{{.withSchema({color:#cc7832}new {color}Schema()}}
{{ .field({color:#6a8759}"user_server"{color}{color:#cc7832},
{color}DataTypes.STRING()).from({color:#6a8759}"user"{color}))}}
h3. Error:
Caused by: org.apache.flink.table.api.ValidationException: Could not map the
schema field 'user_server' to a field from source. Please specify the source
field from which it can be derived.
h3. Workaround:
This code works!
{{.withFormat({color:#cc7832}new {color}Json()}}
{{ .failOnMissingField({color:#cc7832}false{color}))}}
{{.withSchema({color:#cc7832}new {color}Schema()}}
{{ .field({color:#6a8759}"user"{color}{color:#cc7832},
{color}DataTypes.STRING())}}
h3.
h3. From the
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17549) Support running Stateful Functions on native Kubernetes setup
Canbin Zheng created FLINK-17549: Summary: Support running Stateful Functions on native Kubernetes setup Key: FLINK-17549 URL: https://issues.apache.org/jira/browse/FLINK-17549 Project: Flink Issue Type: New Feature Components: Build System / Stateful Functions, Deployment / Kubernetes Reporter: Canbin Zheng This is the umbrella issue for running Stateful Functions on Kubernetes in native mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
set a retract switch
Hi: During the execution of flink, especially the sql API, many operations in DataStream are not available. In many cases, we don't care about the DELETE record when retracting. Is it possible to set a switch so that the DELETE record when retracting is not processed? In other words, the downstream only receives a value after UPDATE, and does not need to receive the value before UPDATE. In some programming modes, processing DELETE records actually makes the logic more complicated. Best Lec Ssmi
[jira] [Created] (FLINK-17550) set a retract switch
Cheng Shi created FLINK-17550: - Summary: set a retract switch Key: FLINK-17550 URL: https://issues.apache.org/jira/browse/FLINK-17550 Project: Flink Issue Type: New Feature Reporter: Cheng Shi During the execution of flink, especially the sql API, many operations in DataStream are not available. In many cases, we don't care about the DELETE record when retracting. Is it possible to set a switch so that the DELETE record when retracting is not processed? In other words, the downstream only receives a value after UPDATE, and does not need to receive the value before UPDATE. In some programming modes, processing DELETE records actually makes the logic more complicated. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17551) Documentation of stable releases are actually built on top of snapshot code bases.
Xintong Song created FLINK-17551: Summary: Documentation of stable releases are actually built on top of snapshot code bases. Key: FLINK-17551 URL: https://issues.apache.org/jira/browse/FLINK-17551 Project: Flink Issue Type: Bug Components: Project Website Affects Versions: 1.10.0 Reporter: Xintong Song When browsing Flink's documentation on the project website, we can choose from both the latest snapshot version and the stable release versions. However, it seems the documentation of stable release version is actually built on top of the snapshot version of the release branch. E.g., currently the latest stable release is 1.10.0, but the documentation described as "Flink 1.10 (Latest stable release)" is actually built with 1.10-SNAPSHOT. As a consequence, users might be confused when they use release 1.10.0 and some latest documentation changes meant for 1.10.1. [This comment|https://github.com/apache/flink/pull/11300#issuecomment-624776199] shows one of such confusions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: [DISCUSS] Hierarchies in ConfigOption
Thanks all for the discussion, I have updated FLIP-105 and FLIP-122 to use the new format option keys. FLIP-105: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=147427289 FLIP-122: https://cwiki.apache.org/confluence/display/FLINK/FLIP-122%3A+New+Connector+Property+Keys+for+New+Factory Best, Jark On Wed, 6 May 2020 at 20:37, Timo Walther wrote: > Cool, so let's go with: > > format = json > json.fail-on-missing-field = true > json.ignore-parse-error = true > > value.format = json > value.json.fail-on-missing-field = true > value.json.ignore-parse-error = true > > Regards, > Timo > > > On 06.05.20 12:39, Jingsong Li wrote: > > Hi, > > > > +1 to: > > format = parquet > > parquet.compression = ... > > parquet.block.size = ... > > parquet.page.size = ... > > > > For the formats like parquet and orc, > > Not just Flink itself, but this way also let Flink keys compatible with > the > > property keys of Hadoop / Hive / Spark. > > > > And like Jark said, this way works for Kafka key value too. > > > > Best, > > Jingsong Lee > > > > On Wed, May 6, 2020 at 6:19 PM Jark Wu wrote: > > > >> Hi, > >> > >> I think Timo proposed a good idea to make both side happy. That is: > >> > >> format = json > >> json.fail-on-missing-field = true > >> json.ignore-parse-error = true > >> > >> value.format = json > >> value.json.fail-on-missing-field = true > >> value.json.ignore-parse-error = true > >> > >> This is a valid hierarchies. Besides, it's more clear that the option > >> belongs to a specific component (i.e. json). > >> This will be more readable when we introducing more formats, e.g. > parquet. > >> > >> format = parquet > >> parquet.compression = ... > >> parquet.block.size = ... > >> parquet.page.size = ... > >> > >> is more readable than current style: > >> > >> format = parquet > >> format.compression = ... > >> format.block.size = ... > >> format.page.size = ... > >> > >> To sum up, I'm +1 to use "format = json", "json.fail-on-missing-field = > >> true". > >> > >> Best, > >> Jark > >> > >> On Wed, 6 May 2020 at 17:12, Danny Chan wrote: > >> > >>> Hi, everyone ~ > >>> > >>> Allows me to share some thoughts here. > >>> > >>> Personally i think for SQL, "format" is obviously better than " > >> format.name", > >>> it is more concise and straight-forward, similar with Presto FORMAT[2] > >> and > >>> KSQL VALUE_FORMAT[1]; i think we move from "connector.type" to > >> "connector" > >>> for the same reason, the "type" or "name" suffix is implicit, SQL > syntax > >>> like the DDL is a top-level user API, so from my side keeping good > >>> user-friendly syntax is more important. > >>> > >>> @Timo I'm big +1 for the a good code style guide, but that does not > mean > >>> we should go for a json-style table options in the DDL, the DDL could > >> have > >>> its own contract. Can we move "represent these config options in YAML" > to > >>> another topic ? Otherwise, how should we handle the "connector" key, > >> should > >>> we prefix all the table options with "connector" ? The original > inention > >> of > >>> FLIP-122 is to remove some redundant prefix/suffix of the table options > >>> because they are obviously implicit there, and the "connector." prefix > >> and > >>> the ".type" or ".name" suffix are the ones we most want to delete. > >>> > >>> @Dawid Although ".type" is just another 4 characters, but we force the > >> SQL > >>> users to do the thing that is obvious reduadant, i know serialize > catalog > >>> table to YAML or use the options in DataStream has similar keys > request, > >>> but they are different use cases that i believe many SQL user would not > >>> encounter, that means we force many users to obey rules for cases they > >>> would never have. > >>> > >>> > >>> [1] https://docs.ksqldb.io/en/latest/developer-guide/create-a-table/ > >>> [2] https://prestodb.io/docs/current/sql/create-table.html > >>> > >>> Best, > >>> Danny Chan > >>> 在 2020年5月4日 +0800 PM11:34，Till Rohrmann ，写道： > Hi everyone, > > I like Timo's proposal to organize our configuration more hierarchical > since this is what the coding guide specifies. The benefit I see is > >> that > config options belonging to the same concept will be found in the same > nested object. Moreover, it will be possible to split the > configuration > into unrelated parts which are fed to the respective components. That > >> way > one has a much better separation of concern since component A cannot > >> read > the configuration of component B. > > Concerning Timo's last two proposals: > > If fail-on-missing-field is a common configuration shared by all > >> formats, > then I would go with the first option: > > format.kind: json > format.fail-on-missing-field: true > > If fail-on-missing-field is specific for json, then one could go with > > format: json > json.fail-on-missing-field: true > > or > > format.kind: json > >>
Re: set a retract switch
Hi Lec, You can use `StreamTableEnvironment#toRetractStream(table, Row.class)` to get a `DataStream>`. The true Boolean flag indicates an add message, a false flag indicates a retract (delete) message. So you can just simply apply a flatmap function after this to ignore the false messages. Then you can get a pure UPSERT stream. Btw, such question should be posted in u...@flink.apache.org, not the dev mailing list. Best, Jark On Thu, 7 May 2020 at 10:07, lec ssmi wrote: > Hi: > During the execution of flink, especially the sql API, many operations > in DataStream are not available. In many cases, we don't care about the > DELETE record when retracting. Is it possible to set a switch so that the > DELETE record when retracting is not processed? In other words, the > downstream only receives a value after UPDATE, and does not need to receive > the value before UPDATE. In some programming modes, processing DELETE > records actually makes the logic more complicated. > > Best > Lec Ssmi >
Re: [VOTE] Release 1.10.1, release candidate #2
@Dawid Yes we will include it into RC3. Thanks for the note. @Till Thanks for the quick fix and the note. I've checked and confirmed there are no open issues under 1.10.1, neither any resolved/closed ones under 1.10.2, so will start to prepare RC3. Best Regards, Yu On Wed, 6 May 2020 at 17:36, Till Rohrmann wrote: > I've merged the fix for FLINK-17514. > > Cheers, > Till > > On Wed, May 6, 2020 at 10:53 AM Dawid Wysakowicz > wrote: > > > Hi all, > > > > I wonder if we could also include FLINK-17313 which I backported into > > 1.10 branch yesterday. > > > > Best, > > > > Dawid > > > > On 06/05/2020 07:26, Yu Li wrote: > > > Thanks Till and Thomas, will include fix for both FLINK-17496 and > > > FLINK-17514 in the next RC. > > > > > > Best Regards, > > > Yu > > > > > > > > > On Tue, 5 May 2020 at 22:10, Thomas Weise wrote: > > > > > >> I opened a PR to backport the Kinesis fix - it would be nice to > include > > if > > >> there is another RC: > > >> > > >> https://github.com/apache/flink/pull/11998 > > >> > > >> > > >> On Tue, May 5, 2020 at 4:50 AM Till Rohrmann > > wrote: > > >> > > >>> I've opened a PR for FLINK-17514. > > >>> > > >>> Cheers, > > >>> Till > > >>> > > >>> On Tue, May 5, 2020 at 11:46 AM Yu Li wrote: > > >>> > > Thanks all for the efforts. > > > > I'm hereby canceling the vote due to FLINK-17514, will prepare > another > > >> RC > > after the issue is fixed (hopefully soon). > > > > Best Regards, > > Yu > > > > > > On Tue, 5 May 2020 at 17:28, Till Rohrmann > > >> wrote: > > > I agree with Aljoscha. This is something we should fix since this > is > > >>> very > > > important for Flink's stability. I will prepare a fix for the > > >> problem. > > > Cheers, > > > Till > > > > > > On Tue, May 5, 2020 at 10:30 AM Congxian Qiu < > qcx978132...@gmail.com > > > wrote: > > > > > >> +1 （no-binding) > > >> - sha and gpg, ok > > >> - all pom files point to same version, ok > > >> - build from souce, ok > > >> - LICENCE, ok > > >> - run demo in standalone cluster, ok > > >> > > >> Best, > > >> Congxian > > >> > > >> > > >> Aljoscha Krettek 于2020年5月5日周二 下午3:50写道： > > >> > > >>> Unfortunately, I found this bug which prevents the > > TaskCancelerWatchdog > > >>> [sic] from working: > > https://issues.apache.org/jira/browse/FLINK-17514. > > > I > > >>> think it's quite crucial that this failsafe mechanism works > > correctly. > > >>> We should cancel the release and fix it. > > >>> > > >>> Best, > > >>> Aljoscha > > >>> > > >>> On 05.05.20 05:55, Hequn Cheng wrote: > > Thanks a lot for managing the release! > > > > +1 (binding) > > > > - Go through all new commits for 1.10.1 and spot no new license > > >> problems. > > - Built from source archive successfully. > > - Signatures and hash are correct. > > - Run SocketWindowWordCount on the local cluster. > > - Install Python package and run Python WordCount example. > > - Reviewed website PR > > > > Best, > > Hequn > > > > On Sun, May 3, 2020 at 9:10 PM Robert Metzger < > > >>> rmetz...@apache.org > > >>> wrote: > > > Thanks a lot for addressing the issues from the last release > > > candidate > > >>> and > > > creating this one! > > > > > > +1 (binding) > > > > > > - Started Flink on YARN on Google Cloud DataProc by setting > > > HADOOP_CLASSPATH > > > - checked staging repo > > > > > > > > > > > > On Sat, May 2, 2020 at 6:57 PM Thomas Weise > > wrote: > > >> +1 (binding) > > >> > > >> Checked signatures and hashes. > > >> > > >> Run internal benchmark applications. > > >> > > >> I found a regression that was actually introduced with > > >> 1.10.0, > > > hence > > >>> not > > > a > > >> blocker for this release: > > >> > > >> https://github.com/apache/flink/pull/11975 > > >> > > >> Thanks, > > >> Thomas > > >> > > >> > > >> On Fri, May 1, 2020 at 5:37 AM Yu Li > > >> wrote: > > >>> Hi everyone, > > >>> > > >>> Please review and vote on the release candidate #2 for > > >> version > > >> 1.10.1, > > > as > > >>> follows: > > >>> [ ] +1, Approve the release > > >>> [ ] -1, Do not approve the release (please provide specific > > >> comments) > > >>> > > >>> The complete staging area is available for your review, > > >> which > > >>> includes: > > >>> * JIRA release notes [1], > > >>> * the official Apache source release and binary convenience > > > releases > > >>> to
Re: [DISCUSS] Send issue and pull request notifications for flink-web and flink-shaded to iss...@flink.apache.org
+1 Thanks, Zhu Zhu Congxian Qiu 于2020年5月6日周三 下午12:02写道： > +1 for this > > Best, > Congxian > > > Benchao Li 于2020年5月5日周二 下午5:22写道： > > > belated +1 > > > > Till Rohrmann 于2020年5月5日周二 下午2:24写道： > > > > > Thanks for all the feedback. I will open the PRs now which change the > > > notification settings of the above-mentioned repositories. > > > > > > Cheers, > > > Till > > > > > > On Tue, May 5, 2020 at 6:26 AM Hequn Cheng wrote: > > > > > > > +1, thanks a lot for driving this. > > > > > > > > Best, Hequn > > > > > > > > On Tue, May 5, 2020 at 11:56 AM Dian Fu > wrote: > > > > > > > > > +1 > > > > > > > > > > Regards, > > > > > Dian > > > > > > > > > > > 在 2020年5月5日，上午9:58，Yangze Guo 写道： > > > > > > > > > > > > +1 > > > > > > > > > > > > Best, > > > > > > Yangze Guo > > > > > > > > > > > > On Tue, May 5, 2020 at 6:14 AM Thomas Weise > > wrote: > > > > > >> > > > > > >> +1 > > > > > >> > > > > > >> > > > > > >> On Mon, May 4, 2020 at 10:02 AM Marta Paes Moreira < > > > > ma...@ververica.com > > > > > > > > > > > >> wrote: > > > > > >> > > > > > >>> +1, this is quite annoying and distracting. > > > > > >>> > > > > > >>> Marta > > > > > >>> > > > > > >>> On Mon, May 4, 2020 at 6:27 PM Yu Li wrote: > > > > > >>> > > > > > +1 > > > > > > > > > > Best Regards, > > > > > Yu > > > > > > > > > > > > > > > On Tue, 5 May 2020 at 00:21, Konstantin Knauf < > > kna...@apache.org> > > > > > wrote: > > > > > > > > > > > Yes, please. > > > > > > > > > > > > On Mon, May 4, 2020 at 5:50 PM Dawid Wysakowicz < > > > > > >>> dwysakow...@apache.org> > > > > > > wrote: > > > > > > > > > > > >> +1 > > > > > >> > > > > > >> Yes, please. I've also observed a lot of noise in the past > > days. > > > > > >> > > > > > >> Best, > > > > > >> > > > > > >> Dawid > > > > > >> > > > > > >> On 04/05/2020 17:48, Tzu-Li (Gordon) Tai wrote: > > > > > >>> +1 > > > > > >>> > > > > > >>> All the recent new repos, flink-statefun / > > > flink-statefun-docker > > > > / > > > > > >>> flink-training etc. are also sending notifications to > issues@ > > . > > > > > >>> > > > > > >>> Gordon > > > > > >>> > > > > > >>> > > > > > >>> On Mon, May 4, 2020, 11:44 PM Till Rohrmann < > > > > trohrm...@apache.org> > > > > > >> wrote: > > > > > >>> > > > > > Hi everyone, > > > > > > > > > > due to some changes on the ASF side, we are now seeing > issue > > > and > > > > > pull > > > > > request notifications for the flink-web [1] and > flink-shaded > > > [2] > > > > > repo > > > > > > on > > > > > dev@flink.apache.org. I think this is not ideal since the > > dev > > > > ML > > > > > >>> is > > > > > >> much > > > > > more noisy now. > > > > > > > > > > I would propose to send these notifications to > > > > > > iss...@flink.apache.org > > > > > >> as > > > > > we are currently doing it for the Flink main repo [3]. > > > > > > > > > > What do you think? > > > > > > > > > > [1] https://github.com/apache/flink-web > > > > > [2] https://github.com/apache/flink-shaded > > > > > [3] https://gitbox.apache.org/schemes.cgi?flink > > > > > > > > > > Cheers, > > > > > Till > > > > > > > > > > >> > > > > > >> > > > > > > > > > > > > -- > > > > > > > > > > > > Konstantin Knauf > > > > > > > > > > > > https://twitter.com/snntrable > > > > > > > > > > > > https://github.com/knaufk > > > > > > > > > > > > > > > > >>> > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Benchao Li > > School of Electronics Engineering and Computer Science, Peking University > > Tel:+86-15650713730 > > Email: libenc...@gmail.com; libenc...@pku.edu.cn > > >
