[ https://issues.apache.org/jira/browse/BEAM-7949?focusedWorklogId=364948&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-364948 ]
ASF GitHub Bot logged work on BEAM-7949: ---------------------------------------- Author: ASF GitHub Bot Created on: 31/Dec/19 10:14 Start Date: 31/Dec/19 10:14 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10246: [BEAM-7949] Add time-based cache threshold support in the data service of the Python SDK harness URL: https://github.com/apache/beam/pull/10246#discussion_r362187710 ########## File path: sdks/python/apache_beam/runners/worker/sdk_worker_main.py ########## @@ -200,6 +203,29 @@ def _get_state_cache_size(pipeline_options): return 0 +def _get_data_buffer_time_limit_ms(pipeline_options): + """Defines the time limt of the outbound data buffering. + + Note: data_buffer_time_limit_ms is an experimental flag and might + not be available in future releases. + + Returns: + an int indicating the time limit in milliseconds of the the outbound + data buffering. Default is 0 (disabled) + """ + experiments = pipeline_options.view_as(DebugOptions).experiments + experiments = experiments if experiments else [] + + for experiment in experiments: + # There should only be 1 match so returning from the loop + if re.match(r'data_buffer_time_limit_ms=', experiment): + return int( + re.match( + r'data_buffer_time_limit_ms=(?P<data_buffer_time_limit_ms>.*)', Review comment: I have also thought about this question when preparing this PR. The reason I have not done that in this PR is because I found that most config keys in the Java SDK harness starts with "beam_fn_api_" and it's not the same case for the config keys in the Python SDK harness. I'm fine to unify the config key if you don't think that's a problem. What's your thought? @lukecwik @mxm ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 364948) Time Spent: 4h (was: 3h 50m) > Add time-based cache threshold support in the data service of the Python SDK > harness > ------------------------------------------------------------------------------------ > > Key: BEAM-7949 > URL: https://issues.apache.org/jira/browse/BEAM-7949 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness > Reporter: sunjincheng > Assignee: sunjincheng > Priority: Major > Fix For: 2.19.0 > > Time Spent: 4h > Remaining Estimate: 0h > > Currently only size-based cache threshold is supported in the data service of > Python SDK harness. It should also support the time-based cache threshold. > This is very important, especially for streaming jobs which are sensitive to > the delay. -- This message was sent by Atlassian Jira (v8.3.4#803005)