Re: Hiding logging for beam playground examples
+1 to at least setting the log level to higher than info. Some runner logging (e.g. job started/done) may be useful. On Tue, Nov 14, 2023 at 9:37 AM Joey Tran wrote: > > Hi all, > > I just had a workshop to demo beam for people at my company and there was a > bit of confusion about whether the beam python playground examples were even > working and it turned out they just got confused by all the runner logging > that is output. > > Is this worth keeping? It seems like it'd be a common source of confusion for > new users > > Cheers, > Joey
Re: Hiding logging for beam playground examples
Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default. On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev wrote: > +1 to at least setting the log level to higher than info. Some runner > logging (e.g. job started/done) may be useful. > > On Tue, Nov 14, 2023 at 9:37 AM Joey Tran > wrote: > > > > Hi all, > > > > I just had a workshop to demo beam for people at my company and there > was a bit of confusion about whether the beam python playground examples > were even working and it turned out they just got confused by all the > runner logging that is output. > > > > Is this worth keeping? It seems like it'd be a common source of > confusion for new users > > > > Cheers, > > Joey >
Re: Hiding logging for beam playground examples
The motivating example does not use LogElements, just Map(print) https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function Some examples of the extraneous logging: ``` 2023-09-08 22:46:37,334 [INFO] 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600 2023-09-08 22:46:37,338 [INFO] Created Worker handler for environment ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'') ``` The example code itself doesn't set the log level in some playground code. Does anyone have a pointer to where? I'm not familiar On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev < dev@beam.apache.org> wrote: > Are the examples using LogElements? > https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 > > Note that LogElements by default prints to stdout, but can be configured > to use a different logger. We could also change the default. > > On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev < > dev@beam.apache.org> wrote: > >> +1 to at least setting the log level to higher than info. Some runner >> logging (e.g. job started/done) may be useful. >> >> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran >> wrote: >> > >> > Hi all, >> > >> > I just had a workshop to demo beam for people at my company and there >> was a bit of confusion about whether the beam python playground examples >> were even working and it turned out they just got confused by all the >> runner logging that is output. >> > >> > Is this worth keeping? It seems like it'd be a common source of >> confusion for new users >> > >> > Cheers, >> > Joey >> >
Re: Hiding logging for beam playground examples
I am also not familiar with Playground. I suspect you could try to make it crash and maybe find a stacktrace? Setting logging could like like so: https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110 On Wed, Nov 15, 2023 at 12:06 PM Joey Tran wrote: > The motivating example does not use LogElements, just Map(print) > > https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function > > Some examples of the extraneous logging: > ``` > 2023-09-08 22:46:37,334 [INFO] populate_data_channel_coders at 0x7ff2665e1a20> > 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600 > 2023-09-08 22:46:37,338 [INFO] Created Worker handler > object at 0x7ff2664c9870> for environment > ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'') > ``` > > The example code itself doesn't set the log level in some playground code. > Does anyone have a pointer to where? I'm not familiar > > On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev < > dev@beam.apache.org> wrote: > >> Are the examples using LogElements? >> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 >> >> Note that LogElements by default prints to stdout, but can be configured >> to use a different logger. We could also change the default. >> >> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev < >> dev@beam.apache.org> wrote: >> >>> +1 to at least setting the log level to higher than info. Some runner >>> logging (e.g. job started/done) may be useful. >>> >>> On Tue, Nov 14, 2023 at 9:37 AM Joey Tran >>> wrote: >>> > >>> > Hi all, >>> > >>> > I just had a workshop to demo beam for people at my company and there >>> was a bit of confusion about whether the beam python playground examples >>> were even working and it turned out they just got confused by all the >>> runner logging that is output. >>> > >>> > Is this worth keeping? It seems like it'd be a common source of >>> confusion for new users >>> > >>> > Cheers, >>> > Joey >>> >>
Re: Hiding logging for beam playground examples
Good idea though it ended up being a shallow trace ``` File "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", line 57, in assert False ``` I think I found where the log level is set anyways ``` https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39 ``` When I have some time, I'll try doing a local deployment of playground and modifying those log levels On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev wrote: > I am also not familiar with Playground. I suspect you could try to make it > crash and maybe find a stacktrace? Setting logging could like like so: > https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110 > > On Wed, Nov 15, 2023 at 12:06 PM Joey Tran > wrote: > >> The motivating example does not use LogElements, just Map(print) >> >> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function >> >> Some examples of the extraneous logging: >> ``` >> 2023-09-08 22:46:37,334 [INFO] > populate_data_channel_coders at 0x7ff2665e1a20> >> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600 >> 2023-09-08 22:46:37,338 [INFO] Created Worker handler >> > object at 0x7ff2664c9870> for environment >> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'') >> ``` >> >> The example code itself doesn't set the log level in some playground >> code. Does anyone have a pointer to where? I'm not familiar >> >> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev < >> dev@beam.apache.org> wrote: >> >>> Are the examples using LogElements? >>> https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 >>> >>> Note that LogElements by default prints to stdout, but can be configured >>> to use a different logger. We could also change the default. >>> >>> On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev < >>> dev@beam.apache.org> wrote: >>> +1 to at least setting the log level to higher than info. Some runner logging (e.g. job started/done) may be useful. On Tue, Nov 14, 2023 at 9:37 AM Joey Tran wrote: > > Hi all, > > I just had a workshop to demo beam for people at my company and there was a bit of confusion about whether the beam python playground examples were even working and it turned out they just got confused by all the runner logging that is output. > > Is this worth keeping? It seems like it'd be a common source of confusion for new users > > Cheers, > Joey >>>
Re: Hiding logging for beam playground examples
I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up. [1]https://github.com/apache/beam/pull/29948 "ghnt [1] https://github.com/apache/beam/pull/29948#event-11504996254 On Thu, Nov 16, 2023 at 7:42 AM Joey Tran wrote: > Good idea though it ended up being a shallow trace > ``` > File > "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", > line 57, in > assert False > ``` > I think I found where the log level is set anyways > ``` > > https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39 > ``` > > When I have some time, I'll try doing a local deployment of playground and > modifying those log levels > > On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev > wrote: > >> I am also not familiar with Playground. I suspect you could try to make >> it crash and maybe find a stacktrace? Setting logging could like like so: >> https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110 >> >> On Wed, Nov 15, 2023 at 12:06 PM Joey Tran >> wrote: >> >>> The motivating example does not use LogElements, just Map(print) >>> >>> https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function >>> >>> Some examples of the extraneous logging: >>> ``` >>> 2023-09-08 22:46:37,334 [INFO] >> populate_data_channel_coders at 0x7ff2665e1a20> >>> 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600 >>> 2023-09-08 22:46:37,338 [INFO] Created Worker handler >>> >> object at 0x7ff2664c9870> for environment >>> ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'') >>> ``` >>> >>> The example code itself doesn't set the log level in some playground >>> code. Does anyone have a pointer to where? I'm not familiar >>> >>> On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev < >>> dev@beam.apache.org> wrote: >>> Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default. On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev < dev@beam.apache.org> wrote: > +1 to at least setting the log level to higher than info. Some runner > logging (e.g. job started/done) may be useful. > > On Tue, Nov 14, 2023 at 9:37 AM Joey Tran > wrote: > > > > Hi all, > > > > I just had a workshop to demo beam for people at my company and > there was a bit of confusion about whether the beam python playground > examples were even working and it turned out they just got confused by all > the runner logging that is output. > > > > Is this worth keeping? It seems like it'd be a common source of > confusion for new users > > > > Cheers, > > Joey >
Re: Hiding logging for beam playground examples
Hi Joey, Thanks for reaching out! I see that your changes haven't been deploed yet, so I've triggered the corresponding job and Playground will be updated soon. Thanks, Andrey From: Joey Tran Reply to: "dev@beam.apache.org" Date: Thursday, 7 March 2024 at 22:33 To: Valentyn Tymofieiev Subject: [EXTERNAL] Re: Hiding logging for beam playground examples I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up. [1]https://github.com/apache/beam/pull/29948 """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt [1] https://github.com/apache/beam/pull/29948#event-11504996254 On Thu, Nov 16, 2023 at 7:42 AM Joey Tran mailto:joey.t...@schrodinger.com>> wrote: Good idea though it ended up being a shallow trace ``` File "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", line 57, in assert False ``` I think I found where the log level is set anyways ``` https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39 ``` When I have some time, I'll try doing a local deployment of playground and modifying those log levels On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev mailto:valen...@google.com>> wrote: I am also not familiar with Playground. I suspect you could try to make it crash and maybe find a stacktrace? Setting logging could like like so: https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110 On Wed, Nov 15, 2023 at 12:06 PM Joey Tran mailto:joey.t...@schrodinger.com>> wrote: The motivating example does not use LogElements, just Map(print) https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function Some examples of the extraneous logging: ``` 2023-09-08 22:46:37,334 [INFO] 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600 2023-09-08 22:46:37,338 [INFO] Created Worker handler for environment ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'') ``` The example code itself doesn't set the log level in some playground code. Does anyone have a pointer to where? I'm not familiar On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev mailto:dev@beam.apache.org>> wrote: Are the examples using LogElements? https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 Note that LogElements by default prints to stdout, but can be configured to use a different logger. We could also change the default. On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev mailto:dev@beam.apache.org>> wrote: +1 to at least setting the log level to higher than info. Some runner logging (e.g. job started/done) may be useful. On Tue, Nov 14, 2023 at 9:37 AM Joey Tran mailto:joey.t...@schrodinger.com>> wrote: > > Hi all, > > I just had a workshop to demo beam for people at my company and there was a > bit of confusion about whether the beam python playground examples were even > working and it turned out they just got confused by all the runner logging > that is output. > > Is this worth keeping? It seems like it'd be a common source of confusion for > new users > > Cheers, > Joey
Re: Hiding logging for beam playground examples
Ah thanks Andrey! I had assumed that since my change was in a release that went out a few weeks ago that Playground would've been redeployed by now. I'll check playground again in an hour or two to see if my changes did anything :) On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin < andrey.devyat...@akvelon.com> wrote: > Hi Joey, > > Thanks for reaching out! I see that your changes haven't been deploed yet, > so I've triggered the corresponding job and Playground will be updated soon. > > Thanks, > Andrey > > > > *From: *Joey Tran > *Reply to: *"dev@beam.apache.org" > *Date: *Thursday, 7 March 2024 at 22:33 > *To: *Valentyn Tymofieiev > *Subject: *[EXTERNAL] Re: Hiding logging for beam playground examples > > > > I ended up making this change[1] but it looks like Playground is still > outputting all the log statements. Could someone help me with this? I > suppose it's best to try and actually test this, but I had a hard time > getting the environment set up. > > [1]https://github.com/apache/beam/pull/29948 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > """""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""ghnt > > [1] https://github.com/apache/beam/pull/29948#event-11504996254 > > > > On Thu, Nov 16, 2023 at 7:42 AM Joey Tran > wrote: > > Good idea though it ended up being a shallow trace > ``` > File > "/opt/playground/backend/executable_files/91e3e49b-7197-4252-a8bd-93c5b252ed55/91e3e49b-7197-4252-a8bd-93c5b252ed55.py", > line 57, in > assert False > ``` > I think I found where the log level is set anyways > ``` > > https://github.com/apache/beam/blob/master/playground/infrastructure/logger.py#L39 > ``` > > When I have some time, I'll try doing a local deployment of playground and > modifying those log levels > > > > On Wed, Nov 15, 2023 at 10:52 PM Valentyn Tymofieiev > wrote: > > I am also not familiar with Playground. I suspect you could try to make it > crash and maybe find a stacktrace? Setting logging could like like so: > https://github.com/apache/beam/blob/729c4de416b8252ec99f0a1253ac7af3023733df/sdks/python/apache_beam/examples/wordcount.py#L110 > > > > On Wed, Nov 15, 2023 at 12:06 PM Joey Tran > wrote: > > The motivating example does not use LogElements, just Map(print) > > https://beam.apache.org/documentation/transforms/python/aggregation/combineglobally/#example-2-combining-with-a-lambda-function > > Some examples of the extraneous logging: > ``` > 2023-09-08 22:46:37,334 [INFO] populate_data_channel_coders at 0x7ff2665e1a20> > 2023-09-08 22:46:37,336 [INFO] Creating state cache with size 104857600 > 2023-09-08 22:46:37,338 [INFO] Created Worker handler > object at 0x7ff2664c9870> for environment > ref_Environment_default_environment_2 (beam:env:embedded_python:v1, b'') > ``` > > The example code itself doesn't set the log level in some playground code. > Does anyone have a pointer to where? I'm not familiar > > > > On Wed, Nov 15, 2023 at 2:10 PM Valentyn Tymofieiev via dev < > dev@beam.apache.org> wrote: > > Are the examples using LogElements? > https://github.com/apache/beam/blob/2012107a0fa2bb3fedf1b5aedcb49445534b2dad/sdks/python/apache_beam/transforms/util.py#L1271 > > > > Note that LogElements by default prints to stdout, but can be configured > to use a different logger. We could also change the default. > > > > On Tue, Nov 14, 2023 at 9:48 AM Robert Bradshaw via dev < > dev@beam.apache.org> wrote: > > +1 to at least setting the log level to higher than info. Some runner > logging (e.g. job started/done) may be useful. > > On Tue, Nov 14, 2023 at 9:37 AM Joey Tran > wrote: > > > > Hi all, > > > > I just had a workshop to demo beam for people at my company and there > was a bit of confusion about whether the beam python playground examples > were even working and it turned out they just got confused by all the > runner logging that is output. > > > > Is this worth keeping? It seems like it'd be a common source of > confusion for new users > > > > Cheers, > > Joey > >
Re: [EXTERNAL] Re: Hiding logging for beam playground examples
Hi Joey, We just deployed the 2.54 changes on playground. Can you now check? Thanks, Vlado From: Joey Tran Sent: Friday, March 8, 2024 2:45:32 PM To: Andrey Devyatkin Subject: [EXTERNAL] Re: Hiding logging for beam playground examples Ah thanks Andrey! I had assumed that since my change was in a release that went out a few weeks ago that Playground would've been redeployed by now. I'll check playground again in an hour or two to see if my changes did anything :) On Fri, Mar 8, 2024 at 8:28 AM Andrey Devyatkin mailto:andrey.devyat...@akvelon.com>> wrote: Hi Joey, Thanks for reaching out! I see that your changes haven't been deploed yet, so I've triggered the corresponding job and Playground will be updated soon. Thanks, Andrey From: Joey Tran mailto:joey.t...@schrodinger.com>> Reply to: "dev@beam.apache.org<mailto:dev@beam.apache.org>" mailto:dev@beam.apache.org>> Date: Thursday, 7 March 2024 at 22:33 To: Valentyn Tymofieiev mailto:valen...@google.com>> Subject: [EXTERNAL] Re: Hiding logging for beam playground examples I ended up making this change[1] but it looks like Playground is still outputting all the log statements. Could someone help me with this? I suppose it's best to try and actually test this, but I had a hard time getting the environment set up. [1]https://github.com/apache/beam/pull/29948 """"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""&qu
