.On Mon, Oct 9, 2023 at 1:11 PM Robert Burke <rob...@frantil.com> wrote:
> I'll note that the file "Writes" in the Go SDK are currently an unscalable > antipattern, because of this exact question. > > Aside from carefully examining other SDKs it's not clear how one authors > a reliable, automatically shardable, window and pane aware in an arbitrary > SDK, simply by referring to common beam constructs. > > Closely examining how other SDKs do it is time consuming and an > antipattern, and doesn't lend itself to educating arbitrary beam end users > on good patterns and why they work, because they tend not to have that sort > of commentary (for all the complexity you mention.) > Yeah, I agree there's implementation complexity here. Hopefully it only has to be done once per SDK (and all other file formats can just fill in the "write this batch of records to a filestream" part), and with multi-langauge not even all SDKs need to go there (initially at least). > But it's just as likely I missed a document somewhere. It has been a while > since I last searched for this, let alone have time to do the deep dives > required to produce it. > > Robert Burke > Beam Go Busybody > >> >> >>
