Hi Ziv, if I mark it as transient, then its value erased and I get null pointer exception. on the other hand, leave it without 'transient', it's a compilation error - implementation of function is not serializable. For unserializable class fields that you need to use for processing, you need to make it transient, and only initialize it in the open() method.
To override the open() method you will need to extend the Rich variant of the operator you are implementing. For example, the RichFlatMapFunction is the Rich variant of the FlatMapFunction. Does that answer your question? Cheers, Gordon On 26 September 2017 at 11:58:43 AM, Meri Ziv (ziv.m...@elbitsystems.com) wrote: hi Tzu-Li, sorry I have to reopen it but I'm still having troubles with using data from previous records. you said: “If that’s the case, and you don’t care about losing state on failures and don’t care about at-least-once / exactly-once, then yes you don’t have to use the managed state APIs in Flink. You can just have ordinary fields to achieve this since Flink’s streaming operators are basically long-running processes that continuously process events in the stream and manipulates its state.” look that if one of my fields is of unserializable type, then: if I mark it as transient, then its value erased and I get null pointer exception. on the other hand, leave it without 'transient', it's a compilation error - implementation of function is not serializable. how you explain that? Best, Ziv. From: Tzu-Li (Gordon) Tai [mailto:tzuli...@apache.org] Sent: 28 July 2017 09:12 To: dev@flink.apache.org; Meri Ziv Subject: RE: is flink' states functionality futile? Here it is (sorry about that): [1] https://ci.apache.org/projects/flink/flink-docs-release-1.3/internals/stream_checkpointing.html On 26 July 2017 at 4:15:54 PM, ziv (ziv.m...@elbitsystems.com<mailto:ziv.m...@elbitsystems.com>) wrote: Hi, what is the reference you wanted to link in [1]? From: Tzu-Li (Gordon) Tai [via Apache Flink Mailing List archive.] [mailto:ml+s1008284n18936...@n3.nabble.com]<mailto:[mailto:ml+s1008284n18936...@n3.nabble.com]> Sent: 26 July 2017 05:44 To: Meri Ziv Subject: Re: is flink' states functionality futile? <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html> it's a program that sinks messages only after enough items accumulated in the buffe Yes, that’s another important aspect of how Flink’s checkpointing is used to achieve at-least-once / exactly-once delivery to external systems. For more details on how this works, I recommend taking at the docs here [1]. Now, assume I'm not bothered of recovering failures and only want the simplest way to implement a program that remembers data from the last run in the stream By “the last run in the stream”, you mean the history of the stream so far, correct? If that’s the case, and you don’t care about losing state on failures and don’t care about at-least-once / exactly-once, then yes you don’t have to use the managed state APIs in Flink. You can just have ordinary fields to achieve this since Flink’s streaming operators are basically long-running processes that continuously process events in the stream and manipulates its state. Cheers, Gordon On 23 July 2017 at 10:28:37 PM, ziv ([hidden email]</user/SendEmail.jtp?type=node&node=18936&i=0>) wrote: Ok, Let me see if I understand you correctly. You actually state that flink' states functionality is introduced only to handle recovering from failures. Let's take the example given in 1.3 documentary - https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html <https://ci.apache.org/projects/flink/flink-docs-release-1.3/dev/stream/state.html> it's a program that sinks messages only after enough items accumulated in the buffer. Now, assume I'm not bothered of recovering failures and only want the simplest way to implement a program that remembers data from the last run in the stream, then, according to you, I may not use none of the elements associated with flink' states - ListState snapshotState initializeState restoreState and the program still functions correctly? -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18879.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com. ________________________________ If you reply to this email, your message will be added to the discussion below: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18936.html To start a new topic under Apache Flink Mailing List archive., email ml+s1008284n1...@n3.nabble.com<mailto:ml+s1008284n1...@n3.nabble.com<mailto:ml+s1008284n1...@n3.nabble.com%3cmailto:ml+s1008284n1...@n3.nabble.com>> To unsubscribe from is flink' states functionality futile?, click here<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=18867&code=eml2Lm1lcmlAZWxiaXRzeXN0ZW1zLmNvbXwxODg2N3wxNzE4MzA5NDcz>. NAML<http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml> The information in this e-mail transmission contains proprietary and business sensitive information. Unauthorized interception of this e-mail may constitute a violation of law. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. You are also asked to contact the sender by reply email and immediately destroy all copies of the original message. -- View this message in context: http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/is-flink-states-functionality-futile-tp18867p18939.html Sent from the Apache Flink Mailing List archive. mailing list archive at Nabble.com. The information in this e-mail transmission contains proprietary and business sensitive information. Unauthorized interception of this e-mail may constitute a violation of law. If you are not the intended recipient, you are hereby notified that any review, dissemination, distribution or duplication of this communication is strictly prohibited. You are also asked to contact the sender by reply email and immediately destroy all copies of the original message.
