AndyJiang99 opened a new pull request, #3760: URL: https://github.com/apache/gobblin/pull/3760
Dear Gobblin maintainers, Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below! ### JIRA - [ ] My PR addresses the following [Gobblin JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR" - https://issues.apache.org/jira/browse/GOBBLIN-1896 ### Description - [ ] Here are some details about my PR, including screenshots (if applicable): Problem: There is a scenario where an audit map returned from Kafka Audit URL is not empty, as when there are counts in at least one of the Kafka tiers. However, it is possible that all or some of the source tiers and all or some of the reference tiers have 0 count. If there is a comparison of two counts where the source tier is 0 and the reference tier is also 0, it will result in a 0/0 calculation when calculating the percentage completeness and this evaluates to NaN (not a number). In this scenario as the determined percentage is NaN, the completeness check for that type of completeness will be returned as false and thus, the watermark will not progress for that hour. Fix: If the source tier reports 0 and the corresponding reference tier also reports 0, the completeness for that hour can be marked as complete as no records were expected and no records were consumed. Thus, the completeness check for the completeness type for that hour can be marked as true and the watermark can be updated and moved forward. ### Tests - [ ] My PR adds the following unit tests __OR__ does not need testing for this extremely good reason: Adds in the test scenario of when source tier count is 0 and reference tier count is also 0. The expected result is that the completeness check will return true. ### Commits - [ ] My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "[How to write a good git commit message](http://chris.beams.io/posts/git-commit/)": 1. Subject is separated from body by a blank line 2. Subject is limited to 50 characters 3. Subject does not end with a period 4. Subject uses the imperative mood ("add", not "adding") 5. Body wraps at 72 characters 6. Body explains "what" and "why", not "how" -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
