[ https://issues.apache.org/jira/browse/BEAM-1507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16013831#comment-16013831 ]
ASF GitHub Bot commented on BEAM-1507: -------------------------------------- GitHub user hepeknet opened a pull request: https://github.com/apache/beam/pull/3167 [BEAM-1507] adding TTL check for staging location Be sure to do all of the following to help us incorporate your contribution quickly and easily: - [ ] Make sure the PR title is formatted like: `[BEAM-1507] adding TTL check for staging location` - [ ] Make sure tests pass via `mvn clean verify`. - [ ] Replace `<Jira issue #>` in the title with the actual Jira issue number, if there is one. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). @dhalperi you like this solution? --- You can merge this pull request into a Git repository by running: $ git pull https://github.com/hepeknet/beam BEAM-1507 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/beam/pull/3167.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #3167 ---- commit 01f2c894de0f80e5f7830f1301e8d5133e5bfd7e Author: Borisa Zivkovic <borisha.zivko...@gmail.com> Date: 2017-05-17T10:37:03Z adding TTL check ---- > java DataflowRunner should warn if the stagingLocation has a TTL > ---------------------------------------------------------------- > > Key: BEAM-1507 > URL: https://issues.apache.org/jira/browse/BEAM-1507 > Project: Beam > Issue Type: Improvement > Components: runner-dataflow > Reporter: Daniel Halperin > Priority: Minor > Labels: newbie, starter > > We have seen a few customers run into a hard-to-track-down bug where the > staging bucket has a TTL, but files get TTL-deleted when they are still > needed. > This might be because of: > 1. Long lived batch jobs / streaming jobs can reference staged files > arbitrarily later and will fail in bad ways if they have been deleted. > 2. Some customers even hit issues where the "check file already exists" > succeeds when starting a job, but then the file is TTL-deleted before the job > actually starts. (This sounds crazy, but may happen if TTL is 7 days and jobs > run every 7 days, for example. Race condition.) > I'm hoping it's not hard to check that files would have TTLs and warn if so. -- This message was sent by Atlassian JIRA (v6.3.15#6346)