[ https://issues.apache.org/jira/browse/SOLR-11277?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16345951#comment-16345951 ]
Rupa Shankar commented on SOLR-11277: ------------------------------------- Thanks [~anshumg] for the feedback! Just uploaded a new patch incorporating your suggestions. As for the benchmarking, I wrote a quick script that has 5 threads concurrently adding X documents to a collection. Here were the results: *MASTER* *5 threads, each adding 1000 docs* Iteration #0: 5232 ms Iteration #1: 4238 ms Iteration #2: 3798 ms Iteration #3: 3144 ms Iteration #4: 2774 ms Iteration #5: 2589 ms Iteration #6: 2513 ms Iteration #7: 2452 ms Iteration #8: 2452 ms Iteration #9: 2482 ms AVG: 3167.4 ms *5 threads, each adding 5000 docs* Iteration #0: 12720 ms Iteration #1: 12191 ms Iteration #2: 12340 ms Iteration #3: 12124 ms Iteration #4: 12170 ms Iteration #5: 12058 ms Iteration #6: 11491 ms Iteration #7: 12274 ms Iteration #8: 12208 ms Iteration #9: 12100 ms AVG: 12167.6 ms *HARD COMMIT FEATURE BRANCH* *5 threads, each adding 1000 docs* Iteration #0: 2758 ms Iteration #1: 2576 ms Iteration #2: 2440 ms Iteration #3: 2416 ms Iteration #4: 2443 ms Iteration #5: 2406 ms Iteration #6: 2419 ms Iteration #7: 2430 ms Iteration #8: 2421 ms Iteration #9: 2448 ms AVG: 2475.7 ms *5 threads, each adding 5000 docs* Iteration #0: 12683 ms Iteration #1: 12520 ms Iteration #2: 12264 ms Iteration #3: 12223 ms Iteration #4: 12148 ms Iteration #5: 12161 ms Iteration #6: 11810 ms Iteration #7: 12398 ms Iteration #8: 12168 ms Iteration #9: 12522 ms AVG: 12289.7 ms Seems like the impact is minimal. > Add auto hard commit setting based on tlog size > ----------------------------------------------- > > Key: SOLR-11277 > URL: https://issues.apache.org/jira/browse/SOLR-11277 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Reporter: Rupa Shankar > Assignee: Anshum Gupta > Priority: Major > Attachments: max_size_auto_commit.patch, max_size_auto_commit.patch > > > When indexing documents of variable sizes and at variable schedules, it can > be hard to estimate the optimal auto hard commit maxDocs or maxTime settings. > We’ve had some occurrences of really huge tlogs, resulting in serious issues, > so in an attempt to avoid this, it would be great to have a “maxSize” setting > based on the tlog size on disk. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org