> On June 1, 2017, 10:46 p.m., David McLaughlin wrote: > > src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java > > Line 376 (original), 377 (patched) > > <https://reviews.apache.org/r/59699/diff/3/?file=1739138#file1739138line379> > > > > I wonder if it's using the taskIds from the event? Since they have been > > selected from the database and might not match the taskIds we passed in. > > This logic was there in the previous patch too. This would ensure > > consistency between taskStore.deleteTasks and the taskIds we broadcast as > > deleted.
Previously taskStore.deleteTasks() is not using the taskIds from the event. But it doesn't hurt to ensure the consistency here. - Kai ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/59699/#review176694 ----------------------------------------------------------- On June 1, 2017, 10:41 p.m., Kai Huang wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/59699/ > ----------------------------------------------------------- > > (Updated June 1, 2017, 10:41 p.m.) > > > Review request for Aurora, David McLaughlin and Santhosh Kumar. > > > Bugs: AURORA-1929 > https://issues.apache.org/jira/browse/AURORA-1929 > > > Repository: aurora > > > Description > ------- > > Improve task history pruning by batch deleting tasks. > > The `'aurora_admin prune_tasks'` endpoint seems to be very slow when the > cluster has a large number of inactive tasks. > > This CR batches all removeTasks operations and execute them all at once to > avoid additional cost of coalescing. The fix will also benefit implicit task > history pruning since it has similar underlying implementation. See > https://issues.apache.org/jira/browse/AURORA-1929 for more information and > details. > > > Diffs > ----- > > src/main/java/org/apache/aurora/scheduler/state/StateManagerImpl.java > 73878210f9028901fda3b08e66c6a63c24260d35 > > > Diff: https://reviews.apache.org/r/59699/diff/3/ > > > Testing > ------- > > __unit_tests:__ > > ./build-support/jenkins/build.sh > > No unit tests were created for this patch since it does not add new > functionalities or alter the interface, but improves the efficiency of the > existing code. > > __e2e tests:__ > > Attached was a screenshot of the task history pruning benchmark obtained from > a scale test in Twitter's test cluster. > > - Before applying this patch, the task history pruning takes ~30 minutes on > 130K tasks. > > - After applying the patch, the pruning takes ~1 minute. > > > File Attachments > ---------------- > > task_history_pruning_benchmark.png > > https://reviews.apache.org/media/uploaded/files/2017/06/01/74eb5104-d338-4530-abd2-b82fbdc6bf84__task_history_pruning_benchmark.png > > > Thanks, > > Kai Huang > >
