Hi, I agree with Vino, that you should check if the watermark is progressing for all subtasks, if you are using event time semantics. If this is not the problem it would help if you could share the code of your job. By the way have you tried reproducing the problem with collection source?
Best, Dawid On 10/10/18 08:44, 潘 功森 wrote: > Hi， > > > > Cause default state size in one hour is too small，and the max window > size is 24 hours, so I used 500M. > > > > MemoryStateBackend stateBackend = new > MemoryStateBackend(/MAX_STATE_SIZE/);//500M env.setStateBackend(stateBackend); > > > > And I found Irrespective of the configured maximal state size, the > state cannot be larger than the akka frame size. > > So I add a config in flink-comf.yaml: > > akka.framesize: 524288000b > > > > What else do I have to pay attention to? > > > > Yours, > > September > > > > ------------------------------------------------------------------------ > *发件人:* vino yang <yanghua1...@gmail.com> > *发送时间:* Wednesday, October 10, 2018 11:45:31 AM > *收件人:* pangong...@hotmail.com > *抄送:* user > *主题:* Re: No data issued by flink window after a few hours > > Hi, > > I saw the exception image you provided. Based on the exception > message, it seems you used the default max state size (5MB). > > You can specify the max state size to override the default value. Try : > > /MemoryStateBackend stateBackend = new > MemoryStateBackend(*theSizeOfBytes*);/ > / > / > Please note that you need to reserve enough memory for Flink. > > Thanks, vino. > > 潘 功森 <pangong...@hotmail.com <mailto:pangong...@hotmail.com>> > 于2018年10月10日周三 上午11:36写道： > > Please have a look about my last mail. > > > > When the cached window data is too large, how? > > > > Yours, > > September > > > > ------------------------------------------------------------------------ > *发件人:* vino yang <yanghua1...@gmail.com > <mailto:yanghua1...@gmail.com>> > *发送时间:* Wednesday, October 10, 2018 11:33:48 AM > *收件人:* pangong...@hotmail.com <mailto:pangong...@hotmail.com> > *抄送:* user > *主题:* Re: No data issued by flink window after a few hours > > Hi, > > Did you mean "computer momery" referring to Memory Statebackend? > The Flink window mechanism is internally based on State, and this > is done for fault tolerance. > If you introduce external storage, it will break its design and > bring other problems. > > Thanks, vino. > > 潘 功森 <pangong...@hotmail.com <mailto:pangong...@hotmail.com>> > 于2018年10月10日周三 上午11:02写道： > > Hi, > > "ram to cache the distinct data about sliding window" means I > used computer momery not the third part db to cache the data > need used in window. > > “the data need used in window” means ：such as the sliding > window is 1 hour, and I need to count the distinct users, I > need to cache the user id about one hour. > > Cause there’re no related errors. > > Yours, > > September > > > > > ------------------------------------------------------------------------ > *发件人:* vino yang <yanghua1...@gmail.com > <mailto:yanghua1...@gmail.com>> > *发送时间:* Wednesday, October 10, 2018 10:49:43 AM > *抄送:* user > *主题:* Re: No data issued by flink window after a few hours > > Hi, > > Can you explain what "ram to cache the distinct data about > sliding window" mean? > The information you provide is too small and will not help > others to help you analyze the problem and provide advice. > > In addition, regarding the usage of Flink related issues, > please only send mail to the user mailing list. > The dev mailing list is mainly used to discuss development > related issues. > > Thanks vino. > > ? ?? <pangong...@hotmail.com <mailto:pangong...@hotmail.com>> > 于2018年10月10日周三 上午10:37写道： > > Hi all, > I used flink window, and when the job begins, we could > get the results of windiow.But there’re no results issued > after a few hours. > I found the job is still running and no errors, and > the data not used window all can be issued. > By the way, I used Flink 1.3.2 and ram to cache the > distinct data about sliding window. > > Yours, > September >
signature.asc
Description: OpenPGP digital signature
