+chad...@gmail.com <chad...@gmail.com> is this consistent with behavior that you observed?
+Udi Meiri <eh...@google.com> sorry about the short notice on this. I know you are working on RC1. But do you think this should be a blocker for 2.18.0? I have https://issues.apache.org/jira/browse/BEAM-8976 to track that. On Mon, Dec 16, 2019 at 5:49 PM Pablo Estrada <pabl...@google.com> wrote: > Hi all, > So there are two places where we log: > 1. At pipeline construction. > 2. At worker startup, to execute user code in workers. > > The workers have always set up their logging handlers properly, but > pipeline construction setup is up to the user. > > If the user doesn't set it up, we could have a default configuration - and > I propose doing it when we create the Pipeline object. See my proposed fix: > https://github.com/apache/beam/pull/10396 - @Robert Bradshaw > <rober...@google.com> wdyt? > > If this keeps holding out, then I have a quickfix to make all l9ogging > happen in the root logger for now: > https://github.com/apache/beam/pull/10394 > > Best > -P. > > On Fri, Dec 13, 2019 at 7:45 PM Ahmet Altay <al...@google.com> wrote: > >> >> >> On Fri, Dec 13, 2019 at 1:08 PM Robert Bradshaw <rober...@google.com> >> wrote: >> >>> The default behavior of Python logging is >>> >>> (1) Logging handlers may get added (usually in main or very close to it). >>> (2) Logging goes to those handlers, or a default handler if none were >>> added. >>> >>> With this proposal, we would have >>> >>> (0) A default handler gets added. >>> (1) Logging handlers may get added (usually in main or very close to >>> it) in addition to the default. >>> (2) Logging goes to these handlers and the default. >>> >>> On Fri, Dec 13, 2019 at 12:46 PM Pablo Estrada <pabl...@google.com> >>> wrote: >>> > >>> > I looked at the documentation for basicConfig: >>> https://docs.python.org/3/library/logging.html#logging.basicConfig >>> > >>> > Specifically, the following line: >>> > >>> > > This function does nothing if the root logger already has handlers >>> configured, unless the keyword argument force is set to True. >>> > >>> > That would mean that anyone can override the handling later on - which >>> the workers do? >>> > Best >>> > -P. >>> > >>> > On Fri, Dec 13, 2019 at 10:55 AM Robert Bradshaw <rober...@google.com> >>> wrote: >>> >> >>> >> Thanks for looking into this. >>> >> >>> >> I'm not sure unconditionally calling logging.basicConfig() on module >>> >> import is the correct solution--this prevents modules that wish to set >>> >> up handlers in place of the default handler from being able to do so. >>> >> (This is why logging.basicConfig is lazily done at the first logging >>> >> statement for the root logger, rather than earlier.) >>> >> >>> >> On Thu, Dec 12, 2019 at 4:34 PM Pablo Estrada <pabl...@google.com> >>> wrote: >>> >> > >>> >> > Hello all, >>> >> > It has been pointed out to me by Chad, and also by others, that my >>> logging changes have caused logs to start getting lost. >>> >> >> Does this include losing user logs? Is there a JIRA, and should we fix it >> in Beam 2.18 branch? >> >> >>> >> > >>> >> > It seems that by never logging on the root logger, initialization >>> for a root handler is skipped; and that's what causes the failures. >>> >> > >>> >> > I will work on a fix for this. I am thinking of providing a very >>> simple apache_beam.utils.get_logger function that does something like this: >>> >> > >>> >> > def get_logger(name): >>> >> > logging.basicConfig() >>> >> > return logging.getLogger(name) >>> >> > >>> >> > And specific paths that need special handling of the logs should >>> override this config by adding their own handlers (e.g. sdk_worker, >>> fn_api_runner, etc). >>> >> > >>> >> > I hope I can have a fix for this by tomorrow. >>> >> > Best >>> >> > -P. >>> >>
