----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/73430/ -----------------------------------------------------------
Review request for atlas, Ashutosh Mestry, Jayendra Parab, Madhan Neethiraj, Nikhil Bonte, and Pinal Shah. Bugs: ATLAS-4340 https://issues.apache.org/jira/browse/ATLAS-4340 Repository: atlas Description ------- In Atlas when a transaction is committed, the entries are committed to HBase (primary storage) and Solr (indexing storage). A transaction is rolled-back if the primary storage commit fails, on the other hand when the secondary commit fails (solr), the transaction is not-rolled back and logged as warning and it is recommended to use reindex to repair the missing index documents. This behavior is due to the fact that the primary storage is the source of truth and indexes can be rebuild. In Janusgraph, there is a property for Solr to make solr commits async. This is set to true in Atlas making every commit to wait until the solr commit is successful. This will have a negative impact on performance and is recommended to be false by default. Property: index.[X].solr.wait-searcher When mutating - wait for the index to reflect new mutations before returning. This can have a negative impact on performance. This Jira is about setting the default value for above property to FALSE and can be overridden if need arises. Diffs ----- intg/src/main/java/org/apache/atlas/ApplicationProperties.java bf97ab146 repository/src/test/java/org/apache/atlas/BasicTestSetup.java a1d7b62a5 repository/src/test/java/org/apache/atlas/discovery/AtlasDiscoveryServiceTest.java a9fbd435a repository/src/test/java/org/apache/atlas/discovery/ClassificationSearchProcessorTest.java 121dca9f5 repository/src/test/java/org/apache/atlas/discovery/EntitySearchProcessorTest.java 764568915 repository/src/test/java/org/apache/atlas/repository/store/graph/v2/EntityCorrelationStoreTest.java a3be5f4ed webapp/src/test/java/org/apache/atlas/web/adapters/TypeDefsRESTTest.java 962345b22 Diff: https://reviews.apache.org/r/73430/diff/1/ Testing ------- 1. Precommit Test: https://ci-builds.apache.org/job/Atlas/job/PreCommit-ATLAS-Build-Test/688/ 2. Build Atlas with embedded Hbase/Solr profile and validated basic sanity tests - running quick start, basic search, tag propagation 3. Performance Test details: Run with default settings - Solr wait-searcher property enabled - true (without patch) ---------------------------------------------------------------------- Start Time : Tue Jun 15 22:26:58 PDT 2021 End Time : Fri Jun 18 02:32:34 PDT 2021 Messages Processed : 91,225 Time Taken : 52 hours 5 mins Rate : ~ 29.2 messages/minute Run with disabled Solr wait-searcher property (will improve solr commit time making it async) - with patch ----------------------------------------------------------------------------------------------------------- Start Time : Mon Jun 14 13:30:04 PDT 2021 End Time : Tue Jun 15 17:23:56 PDT 2021 Messages Processed : 91,225 Time Taken : 27 hours 54 mins Rate : ~ 54.5 messages/minute We see almost 50% perf imrpovement with this change. Thanks, Sarath Subramanian