[jira] [Commented] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow
[ https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020087#comment-16020087 ] Daniel Halperin commented on BEAM-2326: --- [~kenn], maybe you should take a look at this one? > Verbose INFO logging with stateful DoFns and Dataflow > -- > > Key: BEAM-2326 > URL: https://issues.apache.org/jira/browse/BEAM-2326 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 0.6.0 >Reporter: peay >Assignee: Kenneth Knowles > Labels: log > > I am seeing a lot of INFO level logging: > {code} > jsonPayload: { > logger: > "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" > message: "Processing timers for key {} for stateful DoFn" > } > jsonPayload: { > message: "Processing key KV{one of my keys} for stateful DoFn" > logger: > "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" > > } > {code} > out of one of my stateful DoFn. There is one such group of logs for each key > I process, which leads to a very large amount of logs and possibly to a > significant slowdown. > Also, not sure if the {{Processing timers}} log message is missing some > string interpolation or if the empty key is on purpose. > At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given > the large volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow
[ https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16043784#comment-16043784 ] Kenneth Knowles commented on BEAM-2326: --- This was fixed right when it was reported, so should be g2g for the released version. > Verbose INFO logging with stateful DoFns and Dataflow > -- > > Key: BEAM-2326 > URL: https://issues.apache.org/jira/browse/BEAM-2326 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 0.6.0 >Reporter: peay >Assignee: Kenneth Knowles > Labels: log > Fix For: 2.0.0 > > > I am seeing a lot of INFO level logging: > {code} > jsonPayload: { > logger: > "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" > message: "Processing timers for key {} for stateful DoFn" > } > jsonPayload: { > message: "Processing key KV{one of my keys} for stateful DoFn" > logger: > "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" > > } > {code} > out of one of my stateful DoFn. There is one such group of logs for each key > I process, which leads to a very large amount of logs and possibly to a > significant slowdown. > Also, not sure if the {{Processing timers}} log message is missing some > string interpolation or if the empty key is on purpose. > At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given > the large volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (BEAM-2326) Verbose INFO logging with stateful DoFns and Dataflow
[ https://issues.apache.org/jira/browse/BEAM-2326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16044935#comment-16044935 ] peay commented on BEAM-2326: Thanks [~kenn]! > Verbose INFO logging with stateful DoFns and Dataflow > -- > > Key: BEAM-2326 > URL: https://issues.apache.org/jira/browse/BEAM-2326 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 0.6.0 >Reporter: peay >Assignee: Kenneth Knowles > Labels: log > Fix For: 2.0.0 > > > I am seeing a lot of INFO level logging: > {code} > jsonPayload: { > logger: > "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" > message: "Processing timers for key {} for stateful DoFn" > } > jsonPayload: { > message: "Processing key KV{one of my keys} for stateful DoFn" > logger: > "com.google.cloud.dataflow.worker.runners.worker.BatchModeUngroupingParDoFn" > > } > {code} > out of one of my stateful DoFn. There is one such group of logs for each key > I process, which leads to a very large amount of logs and possibly to a > significant slowdown. > Also, not sure if the {{Processing timers}} log message is missing some > string interpolation or if the empty key is on purpose. > At any rate, this seems more like something for {{DEBUG}} than {{INFO}} given > the large volume. -- This message was sent by Atlassian JIRA (v6.3.15#6346)