Re: Compatibility Matrix vs Runners in the code base

2018-09-21 Thread John MacMillan
As someone who works on an active runner that's not in the core codebase (the IBM Streams one), we'd like to stay in the matrix and if that means requiring a link to an external page that's easy enough to provide.   Thanks, John   - Original message -From: Robert Bradshaw To:

Re: DoFn thread constraint.

2018-04-26 Thread John MacMillan
apache/beam-site? Better to fix an issue by the person getting the issue in general in that area ;)   Romain Manni-Bucau@rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book  2018-04-26 15:01 GMT+02:00 John MacMillan <john...@ca.ibm.com>: Thank you for your clarification!   M

Re: DoFn thread constraint.

2018-04-26 Thread John MacMillan
ibucau |  Blog | Old Blog | Github | LinkedIn | Book  2018-04-26 14:37 GMT+02:00 John MacMillan <john...@ca.ibm.com>: I'm trying to understand how restrictive this sentence from 4.3.2 of the Programming Guide is for our runner:   Each instance of your function object is accessed by a s

DoFn thread constraint.

2018-04-26 Thread John MacMillan
I'm trying to understand how restrictive this sentence from 4.3.2 of the Programming Guide is for our runner:   Each instance of your function object is accessed by a single thread on a worker instance, unless you explicitly create your own threads.   Does this imply that a runner may not ever use

Re: Configuring file-based transforms with different options

2018-03-09 Thread John MacMillan
AWS may not be the only provider, and if you use a different endpoint the API requires a region.   CreateOptions are probably a better place if the pipeline needs to access multiple endpoints or regions, but I suspect the user application is likely to still end up with pipeline options of its own