[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=778004&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-778004 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 03/Jun/22 09:47 Start Date: 03/Jun/22 09:47 Worklog Time Spent: 10m Work Description: tvalentyn merged PR #1: URL: https://github.com/apache/beam/pull/1 Issue Time Tracking --- Worklog Id: (was: 778004) Time Spent: 3h 40m (was: 3.5h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 3h 40m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=777916&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777916 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 03/Jun/22 03:24 Start Date: 03/Jun/22 03:24 Worklog Time Spent: 10m Work Description: lukecwik commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1145547276 Run Python PreCommit Issue Time Tracking --- Worklog Id: (was: 777916) Time Spent: 3.5h (was: 3h 20m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 3.5h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=777843&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777843 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 02/Jun/22 21:29 Start Date: 02/Jun/22 21:29 Worklog Time Spent: 10m Work Description: lukecwik commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1145360321 Run Python PreCommit Issue Time Tracking --- Worklog Id: (was: 777843) Time Spent: 3h 20m (was: 3h 10m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 3h 20m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=777653&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777653 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 02/Jun/22 15:40 Start Date: 02/Jun/22 15:40 Worklog Time Spent: 10m Work Description: lukecwik commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1145010005 Run Python PreCommit Issue Time Tracking --- Worklog Id: (was: 777653) Time Spent: 3h 10m (was: 3h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=777527&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-777527 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 02/Jun/22 13:30 Start Date: 02/Jun/22 13:30 Worklog Time Spent: 10m Work Description: lukecwik commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1144868593 Run Python PreCommit Issue Time Tracking --- Worklog Id: (was: 777527) Time Spent: 3h (was: 2h 50m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 3h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > f
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=775563&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775563 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 27/May/22 19:10 Start Date: 27/May/22 19:10 Worklog Time Spent: 10m Work Description: codecov[bot] commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1139954786 # [Codecov](https://codecov.io/gh/apache/beam/pull/1?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report > Merging [#1](https://codecov.io/gh/apache/beam/pull/1?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (49d8d8a) into [master](https://codecov.io/gh/apache/beam/commit/64c61e99236533bd92596983ea2d4e9bcbecdb9d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (64c61e9) will **decrease** coverage by `0.00%`. > The diff coverage is `n/a`. ```diff @@Coverage Diff @@ ## master #1 +/- ## == - Coverage 73.99% 73.99% -0.01% == Files 696 696 Lines 9185191861 +10 == + Hits6796467968 +4 - Misses 2263822644 +6 Partials 1249 1249 ``` | Flag | Coverage Δ | | |---|---|---| | python | `83.73% <ø> (-0.01%)` | :arrow_down: | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more. | [Impacted Files](https://codecov.io/gh/apache/beam/pull/1?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | | |---|---|---| | [...hon/apache\_beam/runners/direct/test\_stream\_impl.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvdGVzdF9zdHJlYW1faW1wbC5weQ==) | `93.28% <0.00%> (-0.75%)` | :arrow_down: | | [sdks/python/apache\_beam/runners/direct/executor.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy9kaXJlY3QvZXhlY3V0b3IucHk=) | `96.46% <0.00%> (-0.55%)` | :arrow_down: | | [sdks/python/apache\_beam/transforms/combiners.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vdHJhbnNmb3Jtcy9jb21iaW5lcnMucHk=) | `93.05% <0.00%> (-0.39%)` | :arrow_down: | | [...hon/apache\_beam/runners/worker/bundle\_processor.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvYnVuZGxlX3Byb2Nlc3Nvci5weQ==) | `93.17% <0.00%> (-0.26%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/sdk\_worker.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvc2RrX3dvcmtlci5weQ==) | `88.94% <0.00%> (-0.16%)` | :arrow_down: | | [...ks/python/apache\_beam/runners/worker/operations.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vcnVubmVycy93b3JrZXIvb3BlcmF0aW9ucy5weQ==) | `74.02% <0.00%> (-0.07%)` | :arrow_down: | | [sdks/python/apache\_beam/io/gcp/bigquery.py](https://codecov.io/gh/apache/beam/pull/1/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9weXRob24vYXBhY2hlX2JlYW0vaW8vZ2NwL2
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=775562&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775562 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 27/May/22 19:08 Start Date: 27/May/22 19:08 Worklog Time Spent: 10m Work Description: tvalentyn commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1139953501 LGTM Issue Time Tracking --- Worklog Id: (was: 775562) Time Spent: 2h 40m (was: 2.5h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2h 40m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elemen
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=775559&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775559 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 27/May/22 18:52 Start Date: 27/May/22 18:52 Worklog Time Spent: 10m Work Description: lukecwik commented on PR #1: URL: https://github.com/apache/beam/pull/1#issuecomment-1139941895 R: @damccorm CC: @tvalentyn @angoenka Issue Time Tracking --- Worklog Id: (was: 775559) Time Spent: 2.5h (was: 2h 20m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=775558&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-775558 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 27/May/22 18:50 Start Date: 27/May/22 18:50 Worklog Time Spent: 10m Work Description: lukecwik opened a new pull request, #1: URL: https://github.com/apache/beam/pull/1 The underlying issue was fixed in gRPC c-core and a min version update to 1.33.1 will ensure that this no longer happens since it will contain https://github.com/grpc/grpc/commit/6e1655447ab2146a643114687d7916249bfdf018 which is the fix for https://github.com/grpc/grpc-java/issues/5188 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] Update `CHANGES.md` with noteworthy changes. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md) GitHub Actions Tests Status (on master branch) [](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule) [](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule) See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI. Issue Time Tracking --- Worklog Id: (was: 775558) Time Spent: 2h 20m (was: 2h 10m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Priority: P3 > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execut
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=185500&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185500 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 16/Jan/19 00:06 Start Date: 16/Jan/19 00:06 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#issuecomment-454599766 Thanks mxm! Merging it. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185500) Time Spent: 2h (was: 1h 50m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=185501&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185501 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 16/Jan/19 00:06 Start Date: 16/Jan/19 00:06 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185501) Time Spent: 2h 10m (was: 2h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Attachments: d44b7eda9e4c_java_server_logs.logs.gz, > d44b7eda9e4c_python_client_logs.log.bz2 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/r
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=185408&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-185408 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 15/Jan/19 19:36 Start Date: 15/Jan/19 19:36 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r248034504 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: Updated the PR to set 20sec keep alive time in python and Java. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 185408) Time Spent: 1h 50m (was: 1h 40m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/wo
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=184832&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184832 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 14/Jan/19 14:57 Start Date: 14/Jan/19 14:57 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r247522399 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: If the runs were successful could you push the changes here? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184832) Time Spent: 1h 40m (was: 1.5h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_w
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=184288&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-184288 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 11/Jan/19 18:29 Start Date: 11/Jan/19 18:29 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r247212176 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: Setting the defaults work. I am just making sure with a few runs that this does work consistently. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 184288) Time Spent: 1.5h (was: 1h 20m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/s
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=183130&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-183130 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 09/Jan/19 16:17 Start Date: 09/Jan/19 16:17 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r246445169 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: Let me know if that is feasible. Otherwise let's merge the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 183130) Time Spent: 1h 20m (was: 1h 10m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/s
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=182639&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-182639 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 08/Jan/19 19:40 Start Date: 08/Jan/19 19:40 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r246126311 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: I will try it and see if it works. The root cause is still unknown so I am not sure setting explicit config is a fix or just another work around. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 182639) Time Spent: 1h 10m (was: 1h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h 10m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent ca
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=181997&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181997 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 07/Jan/19 19:48 Start Date: 07/Jan/19 19:48 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r245776335 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: What if we set the keep alive time explicitly on both ends? Just wondering whether it made sense to set the values instead of using this workaround. https://github.com/grpc/grpc/blob/master/doc/keepalive.md#defaults-values This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181997) Time Spent: 1h (was: 50m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 1h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=181971&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181971 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 07/Jan/19 19:13 Start Date: 07/Jan/19 19:13 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r245763524 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: We are not setting the client configuration anywhere. However, I also feel that this might be a server/client configuration issues. This is just a work around till we find the root cause. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181971) Time Spent: 50m (was: 40m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=181966&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181966 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 07/Jan/19 19:05 Start Date: 07/Jan/19 19:05 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r245763524 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: We are not checking the client configuration anywhere. However, I also feel that this might be a server/client configuration issues. This is just a work around till we find the root cause. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181966) Time Spent: 40m (was: 0.5h) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAW
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=181800&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-181800 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 07/Jan/19 14:39 Start Date: 07/Jan/19 14:39 Worklog Time Spent: 10m Work Description: mxm commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#discussion_r245673708 ## File path: runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/ServerFactory.java ## @@ -144,7 +145,9 @@ private static Server createServer(List services, InetSocketAdd NettyServerBuilder.forPort(socket.getPort()) // Set the message size to max value here. The actual size is governed by the // buffer size in the layers above. - .maxMessageSize(Integer.MAX_VALUE); + .maxMessageSize(Integer.MAX_VALUE) + .permitKeepAliveTime(1, TimeUnit.SECONDS) + .permitKeepAliveWithoutCalls(true); Review comment: It seems to me, we're doing something wrong in the client configuration. This is ok as a workaround if this is a bug, but perhaps we should check the client configuration? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 181800) Time Spent: 0.5h (was: 20m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=180931&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180931 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 04/Jan/19 02:16 Start Date: 04/Jan/19 02:16 Worklog Time Spent: 10m Work Description: angoenka commented on issue #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415#issuecomment-451336926 R: @robertwb @mxm This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180931) Time Spent: 20m (was: 10m) > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam > Issue Type: Bug > Components: sdk-py-harness >Reporter: Ankur Goenka >Assignee: Ankur Goenka >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Data channel and logging channel are failing after some time with 1GB input > data for chicago taxi. > > E1218 02:44:02.837680206 72 chttp2_transport.cc:1148] Received a GOAWAY with > error code ENHANCE_YOUR_CALM and debug data equal to "too_many_pings" > Exception in thread read_grpc_client_inputs: > Traceback (most recent call last): > File "/usr/local/lib/python2.7/threading.py", line 801, in __bootstrap_inner > self.run() > File "/usr/local/lib/python2.7/threading.py", line 754, in run > self.__target(*self.__args, **self.__kwargs) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 273, in > target=lambda: self._read_inputs(elements_iterator), > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruction_id, expected_targets): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 209, in input_elements > raise_(t, v, tb) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/data_plane.py", > line 260, in _read_inputs > for elements in elements_iterator: > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 347, in > next > return self._next() > File "/usr/local/lib/python2.7/site-packages/grpc/_channel.py", line 338, in > _next > raise self > _Rendezvous: <_Rendezvous of RPC that terminated with > (StatusCode.RESOURCE_EXHAUSTED, GOAWAY received)> > Traceback (most recent call last): > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 145, in _execute > response = task() > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 180, in > self._execute(lambda: worker.do_instruction(work), work) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 253, in do_instruction > request.instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py", > line 269, in process_bundle > bundle_processor.process_bundle(instruction_id) > File > "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py", > line 481, in process_bundle > instruct
[jira] [Work logged] (BEAM-6258) Data channel failing after some time for 1G data input
[ https://issues.apache.org/jira/browse/BEAM-6258?focusedWorklogId=180930&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-180930 ] ASF GitHub Bot logged work on BEAM-6258: Author: ASF GitHub Bot Created on: 04/Jan/19 02:15 Start Date: 04/Jan/19 02:15 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #7415: [BEAM-6258] Set grpc keep alive on server creation URL: https://github.com/apache/beam/pull/7415 Reference upstream issues https://github.com/grpc/grpc-java/issues/5188 Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | --- | --- | --- | --- Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | --- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 180930) Time Spent: 10m Remaining Estimate: 0h > Data channel failing after some time for 1G data input > -- > > Key: BEAM-6258 > URL: https://issues.apache.org/jira/browse/BEAM-6258 > Project: Beam >
