keith-turner opened a new issue, #5660: URL: https://github.com/apache/accumulo/issues/5660
**Is your feature request related to a problem? Please describe.** Main has a similar issue to #5543 except that instead of creating a batch writer it will create a conditional writer [here](https://github.com/apache/accumulo/blob/439ea97c7b12487c62f04c57eb8cc362e8bcf5a7/server/manager/src/main/java/org/apache/accumulo/manager/compaction/coordinator/CompactionCoordinator.java#L794) when committing an external compaction. This conditional writer will write to the fate table and it will creates threads, so this could lead to the same problem that #5543 solved which is creating an excessive number of threads when completing lots of small external compactions. **Describe the solution you'd like** The solution to #5543 introduced a new SharedBatchWriter class. This was done because the batch writer is designed to be used by a single thread writing lots of data. The ConditionalWriter is designed to support many threads using a single ConditionalWriter object to write lots of data. So do not need to create a shared conditional writer, multiple threads can directly use a shared conditinal writer The user fate store could be modified to use a single conditional writer instance for all of its writes. This may be generally beneficial outside of the compaction commit use case. A single shared conditional writer for fate operations would allow RPC batching of conditional mutations for all threads in the manger doing fate table updates at around the same time. Hopefully the following could be done to avoid creating lots of conditional writer threads in the manager. * Create a conditional writer in the UserFateStore constructor. Add a new property to configure how many threads this conditional writer has. * Chang the places in FateMutatorImpl and UserFateStore that call AccumuloClient.createConditionalWriter to use the conditional writer created in the construcor. * When fate is closed, close the conditional writer. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
