Re: [DISCUSS] SPIP: Native support of session window

2018-10-06 Thread Yuanjian Li
Cool, thanks! Sorry for the late reply, we'll check out the UT and your design doc ASAP when we back from National Day holiday. Thanks, Yuanjian Li Jungtaek Lim 于2018年9月29日周六 上午5:21写道: > Btw, just wrote up detailed design doc on existing patch: > > https://docs.google.com/document/d/1tUO29BDXb9

Re: [DISCUSS] SPIP: Native support of session window

2018-09-28 Thread Jungtaek Lim
Btw, just wrote up detailed design doc on existing patch: https://docs.google.com/document/d/1tUO29BDXb9127RiivUS7Hv324dC0YHuokYvyQRpurDY/edit?usp=sharing This doc is a wall of text, since I guess we already imagine how session window works (and I showed a simple example in SPIP doc), so try to av

Re: [DISCUSS] SPIP: Native support of session window

2018-09-28 Thread Jungtaek Lim
Thanks for sharing your proposal as well as implementation. Looks like your proposal is more like focused to design details: I may be better to write one more for design details and share it as well. Stay tuned! Btw, I'm trying out your patch to see whether it passes the tests I've added, and look

Re: [DISCUSS] SPIP: Native support of session window

2018-09-28 Thread Yuanjian Li
Hi Jungtaek: We also meet this problem during migration of streaming application to Structure Streaming in Baidu practice, we solved this in our folk and just steady running in product. As the initial plan we are doing the code clean work and preparing give a SPIP in Oct, happy to see you

[DISCUSS] SPIP: Native support of session window

2018-09-27 Thread Jungtaek Lim
Hi all, I would like to initiate discussion thread to discuss "Native support of session window". Origin issue is filed to SPARK-10816 [1] but I can file another one for representing SPIP if necessary. WIP but working PR is available as well, so we can even test it directly or see the difference i