Re: ExpansionService...as a service?

2021-10-06 Thread Steve Niemitz
> I was imagining a long running expansion service would just serve up a fixed number of transforms, and changing this set would be done via a new release of the service. This is how I was thinking about it as well, I don't think we'd have any requirement to dynamically add/remove transforms at ru

Re: ExpansionService...as a service?

2021-10-06 Thread Steve Niemitz
I noticed that Environments.getDeferredArtifacts will add a function to DefaultArtifactResolver.INSTANCE which is never removed, so those will accumulate forever. That's the only one I noticed so far from code review, but it's possible there's more. On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw

Re: ExpansionService...as a service?

2021-10-06 Thread Steve Niemitz
cool, thanks for the info. I might be the first to try then :) On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik wrote: > I believe that was one of the ideas behind an expansion service but to my > knowledge I don't know of anyone who has a long running expansion service. > I was thinking that the Apac

ExpansionService...as a service?

2021-10-06 Thread Steve Niemitz
Has anyone ever tried hosting a long-running expansion service as a real "service", the intent being that users don't need to run it locally, and can instead connect to the shared one when expanding pipelines? Looking around the code I already see a few assumptions that it will only live for a sho

Re: File sink with windows and side inputs

2021-10-06 Thread Pavel Solomin
Hello! After some experimentation I was able to make it half-broken instead of completely broken: - I am using Combine.globally(new Combiner()).asSingletonView() to reduce PCollection into a Map - I made the main input to have smaller windows, and the PCollectionView window to be GlobalWindows wi