Hi Eduardo,
These differences are described by the link I sent (
https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L465-L666)
- it documents what kind of things it's best to do in each method. Please
let me know if something is still unclear.
On Tue, Jul 3, 2018 at 1:35 PM eduardo.mora...@gmail.com <
eduardo.mora...@gmail.com> wrote:
> FinishBundle() does the job.
>
> Should I keep using Setup()? What is the difference between Setup() and
> StartBundle()?
>
> Thanks again.
>
> On 2018/07/03 20:10:21, Henning Rohde <hero...@google.com> wrote:
> > Teardown has very loose guarantees on when it's called and you
> essentially
> > can't rely on it. Currently, for Go on non-direct runners, we hang on to
> > the bundle descriptors forever and never destroy them (and in turn never
> > call Teardown). Even if we didn't, failures/restarts could cause Teardown
> > to not be called.
> >
> > If something _must_ happen, FinishBundle is the right method.
> >
> > Thanks,
> > Henning
> >
> > On Tue, Jul 3, 2018 at 10:37 AM eduardo.mora...@gmail.com <
> > eduardo.mora...@gmail.com> wrote:
> >
> > > Essentially I have the following code:
> > >
> > > type Writer struct {
> > > Pool WriterPool
> > > }
> > >
> > > func (w *Writer) Setup() {
> > > w.Pool = Init()
> > > }
> > >
> > > func (w* Writer) ProcessElement(ctx, elem Elem) {
> > > w.Pool.Add(elem)
> > > }
> > >
> > > func (w* Writer) Teardown() {
> > > w.Pool.Write()
> > > w.Pool.Close()
> > > }
> > >
> > > beam.ParDo0(scope, &Writer{}, elemCollection)
> > >
> > > The above code runs fine with the direct runner but not with dataflow.
> > >
> > > I added log lines to the above methods, and the ones in Teardown()
> never
> > > appear in the logs.
> > > If I change my code as follows:
> > >
> > > func (w* Writer) ProcessElement(ctx, elem Elem) {
> > > w.Pool.Add(elem)
> > > w.Pool.Write()
> > > }
> > >
> > > Then I see the data being written, but I lose the ability to pool,
> plus I
> > > am leaking connections.
> > >
> > > Is this a known issue, or I am going something wrong?
> > >
> > > Thanks again for the help.
> > >
> >
>
