Re: Python profiling

2018-11-16 Thread Ahmet Altay
On Fri, Nov 16, 2018 at 10:12 AM, Thomas Weise wrote: > Since it is for users, it should eventually go to the web site. > > How about a new section under: https://beam.apache. > org/documentation/sdks/python/ > > "Troubleshooting and Tuning" ? > That is a good idea. > > > On Fri, Nov 16, 2018

Re: Python profiling

2018-11-16 Thread Thomas Weise
Since it is for users, it should eventually go to the web site. How about a new section under: https://beam.apache.org/documentation/sdks/python/ "Troubleshooting and Tuning" ? On Fri, Nov 16, 2018 at 10:08 AM Ahmet Altay wrote: > > > On Fri, Nov 16, 2018 at 2:12 AM, Robert Bradshaw > wrote:

Re: Python profiling

2018-11-16 Thread Ahmet Altay
On Fri, Nov 16, 2018 at 2:12 AM, Robert Bradshaw wrote: > One needs to ensure that gprof2dot is importable (i.e. installed via pip > into your Python environment). > > As for specifying the FnApiRunner via the runner argument, --runner can > take fully qualified names (if it's not in the short li

Re: Python profiling

2018-11-16 Thread Robert Bradshaw
One needs to ensure that gprof2dot is importable (i.e. installed via pip into your Python environment). As for specifying the FnApiRunner via the runner argument, --runner can take fully qualified names (if it's not in the short list of known runners). However, the FnApiRunner is the DirectRunner

Re: Python profiling

2018-11-15 Thread Thomas Weise
Hi Robert, This is great. It should be added to our Python documentation because users will like need this! After I installed gprof2dot I'm still prompted to install: "Please install gprof2dot and dot for profile renderings." Also is there a way to run a pipeline unmodified with fn_api_runner?

Re: Python profiling

2018-11-05 Thread Ankur Goenka
All containers are destroyed by default on termination so to analyze profiling data for portable runners, either disable container cleanup (using --retainDockerContainers=true) or use remote distributed file system path. On Mon, Nov 5, 2018 at 1:05 AM Robert Bradshaw wrote: > Any portable runner

Re: Python profiling

2018-11-05 Thread Robert Bradshaw
Any portable runner should pick it up automatically. On Tue, Oct 30, 2018 at 3:32 AM Manu Zhang wrote: > > Cool ! Can we document it somewhere such that other Runners could pick it up > later ? > > Thanks, > Manu Zhang > On Oct 29, 2018, 5:54 PM +0800, Maximilian Michels , wrote: > > This looks v

Re: Python profiling

2018-10-29 Thread Manu Zhang
Cool ! Can we document it somewhere such that other Runners could pick it up later ? Thanks, Manu Zhang On Oct 29, 2018, 5:54 PM +0800, Maximilian Michels , wrote: > This looks very helpful for debugging performance of portable pipelines. > Great work! > > Enabling local directories for Flink or

Re: Python profiling

2018-10-29 Thread Maximilian Michels
This looks very helpful for debugging performance of portable pipelines. Great work! Enabling local directories for Flink or other portable Runners would be useful for debugging, e.g. per https://issues.apache.org/jira/browse/BEAM-5440 On 26.10.18 18:08, Robert Bradshaw wrote: Now that we'v