[
https://issues.apache.org/jira/browse/SOLR-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Per Steffensen updated SOLR-3178:
---------------------------------
Attachment: SOLR-3173_3178_3382_3428_plus.patch
SOLR-3173_3178_3382_3428_plus.patch updated to fit on top of revision 1355667
on 4.x branch.
I have tried to describe for (almost) all files in the patch, which problem the
changes in them solve. Please see below. A lot of files has changed, but it is
very small changes in most of the files basically just making sure
SolrRequestInfo-threadlocal is set in tests and other stuff like that. The
solutions to SOLR-3173 and SOLR-3178 are not very big. Actually much more code
has been done to solve SOLR-3382. With those descriptions I hope to convince
you to do the commit. Trust you test-suite - it has not been changed, except
that additional asserts has been added.
I am really not religious about whether the version-control code is in
DistributedUpdateProcessor or DirectUpdateHandler2, but I am religious about
being backward compatible wrt such fundamental changes in semantics, and I am
religious about getting typed errors/exceptions back in responses, so that I
can react properly. I do believe version-control belong in
DirectUpdateHandler2, but if you want it in DistributedUpdateProcessor,
basically just refactor (after commit) and move the changes in
DirectUpdateHandler2 to DistributedUpdateHandler.
* Solved SOLR-3173 and SOLR-3178 (implementation of
fail-on-unique-key-violation and version-check), by introducing 3 "modes" Solr
server can run in (controlled by "semanticsMode" in solrconfig.xml |
updateHandler) - one for backward compatibility, one for a very strict version
control, and a hybrid (default).
** solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java (the new
features are controlled from here, but basically using the rules in
UpdateSemanticsMode. Got rid of duplicate code in constructors - grrrr)
** solr/core/src/java/org/apache/solr/update/UpdateSemanticsMode.java (enum
that encapsulates the essence in the 3 possible "semanticsModes". Used by
DirectUpdateHandler2. Nice "separation of concern")
** solr/core/src/java/org/apache/solr/update/UpdateCommand.java (added
leaderLogic calculated (as isLeader) in DistributedUpdateProcesser to
UpdateCommand so that it is available in DirectUpdateHandler2, where it is used
to know if the new checks must be performed or not. UpdateCommand also carrying
requestVersion, the version sent in the request)
** solr/core/src/java/org/apache/solr/core/SolrConfig.java (control via
configuration which semantics-mode to use)
**
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
(maintains leaderLogic on UpdateCommand)
** solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java (maintains
requestVersion on UpdateCommand)
** solr/core/src/java/org/apache/solr/update/UpdateLog.java (maintains
requestVersion on UpdateCommand)
** solr/core/src/java/org/apache/solr/update/PeerSync.java (maintains
leaderLogic and requestVersion on UpdateCommand)
**
solr/core/src/test-files/solr/collection1/conf/solrconfig-classic-semantics.xml
(default test solrconfig.xml but with "classic" semantics-mode, used to test
that the new features fail-on-unique-key-violation and version-check are not
activated when running "classic" semantics - the backward compatible "mode")
**
solr/core/src/test-files/solr/collection1/conf/solrconfig-classic-consistency-hybrid-semantics.xml
(default test solrconfig.xml but with "classic-consistency-hybrid"
semantics-mode. Actually default solrconfig.xml could have been used since
"classic-consistency-hybrid" semantics is default, but used to test that it
(also) works when this mode is selected explicitly)
** solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java and
solr/test-framework/src/java/org/apache/solr/util/AbstractSolrTestCase.java and
solr/test-framework/src/java/org/apache/solr/util/TestHarness.java (basically
just a lot of changes making testing easier)
* Removed the (temporary and poor IMHO) implementation of
fail-on-unique-key-violation and version-check, but kept the introduced tests,
so you can see that the same functionality is still provided (on default
semantics "classic-consistency-hybrid")
**
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
** solr/core/src/test/org/apache/solr/search/TestRealTimeGet.java (now we do
not only get exceptions in "bad" situations, we get typed exceptions telling us
what is wrong)
* Solved SOLR-3382 - partRef and PartialError(s) introduced in
multi-document-update-requests so that they can be paired up with errors in
responses, so that the client will know that document-updates failed with what
errors and therefore also which documents where updated successfully
** solr/solrj/src/java/org/apache/solr/common/RequestPart.java (new generic
interface to be implemented by all parts of a request that is "just a part",
and where an error that occures during the handling of this particular part
does not mean that handling other parts will also result in error. Just to make
it a little generic from the start, but by now only SolrInputDocuments in
update-requests are such "request-parts")
** solr/solrj/src/java/org/apache/solr/common/RequestPartImpl.java (new default
impl of RequestPart, to be used by classes that really want to be a
"request-part")
** solr/solrj/src/java/org/apache/solr/common/SolrInputDocument.java (a
"request part" implemented using RequestPartImpl)
** solr/solrj/src/java/org/apache/solr/client/solrj/impl/HttpSolrServer.java
(dealing with errors returned in responses. removed a poor initial but never
finished attempt to transport errors in responses, and replaced with my way of
doing it (complete and finished solution). this indicates a problem with the
approach of only accepting small patches - you very easily end up having partly
implemented features that no-one ever finishes)
**
solr/solrj/src/java/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrServer.java
(dealing with errors returned in responses)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/WrongUsage.java
(new error type indicating that the client is using the system in a wrong way -
basically that a request is inconsistent with server rules about how the
request must be)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/PartialError.java
(super-class of all partial-error-types)
**
solr/solrj/src/java/org/apache/solr/common/partialerrors/update/DocumentUpdatePartialError.java
(super-class of all partial-error-types that has to do with updates)
**
solr/solrj/src/java/org/apache/solr/common/partialerrors/update/DocumentAlreadyExists.java
(new error type indicating that the document you tried to create (not update)
already exists)
**
solr/solrj/src/java/org/apache/solr/common/partialerrors/update/DocumentDoesNotExist.java
(new error type indicating that the document you tried to update (not create)
does not already exist (maybe it has been deleted))
**
solr/solrj/src/java/org/apache/solr/common/partialerrors/update/VersionConflict.java
(new error type indicating that there is a version-mismatch in the document
that the client tries to update. Basically the client loaded a version of the
document, made some updates to it and sent it to the server for storage of the
changes, but the document on the server has changed since the document was
loaded by the client, and the client has therefore created his updated document
based on a obsolete version of the document. Nothing to do but reload, change
and attempt update again)
** solr/solrj/src/java/org/apache/solr/common/partialerrors/PartialErrors.java
(new error type able to carry a map between "request parts" and the errors that
occurred when they where handled)
**
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
(catching PartialErrors (defined as errors only related to the document
currently being handled) and continues with next document after adding it to
response linked to document that triggered the error)
** solr/core/src/java/org/apache/solr/servlet/SolrDispatchFilter.java (encodes
errors. If multiple partial-errors occurred (e.g. multi-document-updates) a
body is still needed in the response, telling about the map between errors and
the documents they occurred for. If only single error occurred still just
report that back using HTTP-error-code and reason-phrase. Also set HTTP header
including the error-type (namespace + name = java-package + java-class-name).
The actual encoding/decoding of errors in responses is isolated in
SolrException)
** solr/solrj/src/java/org/apache/solr/common/SolrException.java (encapsulating
the logic to encode/decode errors in responses - also visible for SolrJ clients
so we can decode there in a nice way)
** solr/core/src/java/org/apache/solr/response/SolrQueryResponse.java (able to
carry information about partial-errors and handled "parts" of a request)
** solr/core/src/java/org/apache/solr/handler/ContentStreamLoader.java and
solr/core/src/java/org/apache/solr/handler/loader/ContentStreamLoader.java (a
few constants - strings used to identify partRefs in requests - why two
classes!?!)
** solr/core/src/java/org/apache/solr/handler/loader/JsonLoader.java (partRef
can be sent in JSON requests)
** solr/core/src/java/org/apache/solr/handler/loader/CSVLoaderBase.java
(partRefs can be sent in CSV requests)
** solr/core/src/java/org/apache/solr/handler/loader/XMLLoader.java (partRefs
can be sent in XML requests)
** solr/solrj/src/java/org/apache/solr/client/solrj/util/ClientUtils.java
(partRefs can be sent in XML requests)
** solr/solrj/src/java/org/apache/solr/common/util/JavaBinCodec.java (partRefs
can be sent in binary requests)
** solr/solrj/src/java/org/apache/solr/client/solrj/SolrResponse.java (gives
access to partial-errors, map between "request parts" (essentially documents)
and corresponding partial-errors, number of partial-errors etc.)
**
solr/solrj/src/java/org/apache/solr/client/solrj/response/UpdateResponse.java
(see SolrResponse.java)
** solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java and
solr/test-framework/src/java/org/apache/solr/util/AbstractSolrTestCase.java and
solr/test-framework/src/java/org/apache/solr/util/TestHarness.java (return
errors in helper-assertFailed-methods, so that those errors can be inspected in
further more specific assertions - basically just a lot of changes making
testing easier)
** solr/solrj/src/test/org/apache/solr/client/solrj/SolrExampleTests.java
(nicer asserts including asserting on correct type (class) of exception)
** solr/core/src/test/org/apache/solr/handler/JsonLoaderTest.java (test of it
when using JSON requests and responses)
** solr/core/src/test/org/apache/solr/handler/CSVRequestHandlerTest.java
* Solved SOLR-3428
** solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
* In DirectUpdateHandler2 (solving SOLR-3173 and SOLR-3178) and other places Im
working with the SolrQueryRequest and SolrQueryResponse found in the
threadlocal of SolrRequestInfo, so therefore this threadlocal needs to be set
(and cleared) in many tests
**
solr/core/src/test/org/apache/solr/update/processor/FieldMutatingUpdateProcessorTest.java
**
solr/core/src/test/org/apache/solr/update/processor/SignatureUpdateProcessorFactoryTest.java
**
solr/core/src/test/org/apache/solr/update/processor/UniqFieldsUpdateProcessorFactoryTest.java
** solr/core/src/test/org/apache/solr/update/TestIndexingPerformance.java
** solr/core/src/test/org/apache/solr/request/TestBinaryResponseWriter.java
** solr/core/src/test/org/apache/solr/request/TestFaceting.java
** solr/core/src/test/org/apache/solr/core/TestArbitraryIndexDir.java
** solr/core/src/test/org/apache/solr/TestGroupingSearch.java
** solr/core/src/test/org/apache/solr/SampleTest.java
** solr/core/src/test/org/apache/solr/EchoParamsTest.java
** solr/core/src/test/org/apache/solr/search/function/TestFunctionQuery.java
** solr/core/src/test/org/apache/solr/search/TestValueSourceCache.java
** solr/core/src/test/org/apache/solr/search/TestSearchPerf.java
** solr/core/src/test/org/apache/solr/search/QueryEqualityTest.java
** solr/core/src/test/org/apache/solr/search/TestSort.java
** solr/core/src/test/org/apache/solr/cloud/BasicZkTest.java
** solr/core/src/test/org/apache/solr/highlight/FastVectorHighlighterTest.java
** solr/core/src/test/org/apache/solr/highlight/HighlighterConfigTest.java
** solr/core/src/test/org/apache/solr/highlight/HighlighterTest.java
** solr/core/src/test/org/apache/solr/BasicFunctionalityTest.java
** solr/core/src/test/org/apache/solr/DisMaxRequestHandlerTest.java
** solr/core/src/test/org/apache/solr/handler/XsltUpdateRequestHandlerTest.java
** solr/core/src/test/org/apache/solr/handler/StandardRequestHandlerTest.java
** solr/core/src/test/org/apache/solr/handler/component/StatsComponentTest.java
** solr/core/src/test/org/apache/solr/handler/MoreLikeThisHandlerTest.java
** solr/core/src/java/org/apache/solr/servlet/DirectSolrConnection.java
**
solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/AbstractDataImportHandlerTestCase.java
**
solr/contrib/dataimporthandler/src/test/org/apache/solr/handler/dataimport/TestDocBuilder2.java
**
solr/contrib/uima/src/test/org/apache/solr/uima/processor/UIMAUpdateRequestProcessorTest.java
** solr/test-framework/src/java/org/apache/solr/SolrTestCaseJ4.java
** solr/test-framework/src/java/org/apache/solr/util/AbstractSolrTestCase.java
** solr/test-framework/src/java/org/apache/solr/util/TestHarness.java
* SolrRequest sub-classes has a lot of duplicated code (grrrr). Cleaned up and
put shared code in one place - SolrRequest
** solr/solrj/src/java/org/apache/solr/client/solrj/SolrRequest.java
**
solr/solrj/src/java/org/apache/solr/client/solrj/request/AbstractUpdateRequest.java
**
solr/solrj/src/java/org/apache/solr/client/solrj/request/FieldAnalysisRequest.java
**
solr/solrj/src/java/org/apache/solr/client/solrj/request/DirectXmlRequest.java
** solr/solrj/src/java/org/apache/solr/client/solrj/request/LukeRequest.java
**
solr/solrj/src/java/org/apache/solr/client/solrj/request/CoreAdminRequest.java
**
solr/solrj/src/java/org/apache/solr/client/solrj/request/DocumentAnalysisRequest.java
** solr/solrj/src/java/org/apache/solr/client/solrj/request/SolrPing.java
** solr/solrj/src/java/org/apache/solr/client/solrj/request/QueryRequest.java
* Think we should work with a common (for both server and client side)
"_version_" String constant, but VersionInfo is not available for client side,
so I removed VERSION_FIELD = "_version_" from VersionInfo to SolrInputDocument
in order to make it available for both server and client (SolrJ) side.
** solr/core/src/java/org/apache/solr/update/VersionInfo.java
** solr/solrj/src/java/org/apache/solr/common/SolrInputDocument.java
** solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
**
solr/core/src/java/org/apache/solr/update/processor/DistributedUpdateProcessor.java
** solr/core/src/test/org/apache/solr/update/DirectUpdateHandlerTest.java
** solr/core/src/test/org/apache/solr/cloud/FullSolrCloudDistribCmdsTest.java
* Added some new tests with focus on testing the new version- and
unique-key-based features - ability to fail instead of overwriting on
unique-key violation and ability to report errors on version conflict
**
solr/core/src/test/org/apache/solr/update/ClassicConsistencyHybridUpdateSemanticsTest.java
**
solr/core/src/test/org/apache/solr/update/ClassicConsistencyHybridUpdateSemanticsPartialErrorsTest.java
** solr/core/src/test/org/apache/solr/update/ClassicUpdateSemanticsTest.java
**
solr/core/src/test/org/apache/solr/cloud/ClassicConsistencyHybridUpdateSemanticsSolrCloudTest.java
**
solr/solrj/src/test/org/apache/solr/client/update/ClassicConsistencyHybridUpdateSemanticsTest.java
**
solr/solrj/src/test/org/apache/solr/client/update/ClassicConsistencyHybridUpdateSemanticsConcurrencyTest.java
* Renamed AddUpdateCommand.overwrite to AddUpdateCommand.classicOverwrite,
because this is a field that is now only used when running with "classic"
semantics
** solr/core/src/java/org/apache/solr/update/DirectUpdateHandler2.java
** solr/core/src/java/org/apache/solr/update/SolrCmdDistributor.java
** solr/core/src/java/org/apache/solr/handler/loader/JsonLoader.java
** solr/core/src/java/org/apache/solr/handler/loader/JavabinLoader.java
** solr/core/src/java/org/apache/solr/handler/loader/CSVLoaderBase.java
** solr/core/src/java/org/apache/solr/handler/loader/XMLLoader.java
**
solr/contrib/extraction/src/java/org/apache/solr/handler/extraction/ExtractingDocumentLoader.java
**
solr/solrj/src/java/org/apache/solr/client/solrj/request/UpdateRequestExt.java
** solr/core/src/java/org/apache/solr/update/AddUpdateCommand.java
** solr/core/src/test/org/apache/solr/update/TestIndexingPerformance.java
** solr/core/src/test/org/apache/solr/handler/XmlUpdateRequestHandlerTest.java
**
solr/core/src/test/org/apache/solr/handler/BinaryUpdateRequestHandlerTest.java
** etc
* When putting config-files into ZK you can have them named something different
in there than their file-name on disk. E.g. ./my/folder/my-solrconfig.xml can
be named solrconfig.xml in ZK
** solr/core/src/test/org/apache/solr/cloud/AbstractZkTestCase.java
** solr/core/src/test/org/apache/solr/cloud/AbstractDistributedZkTestCase.java
* Misc unimportant cleanup - removed unneeded imports, etc.
**
solr/core/src/test/org/apache/solr/update/DirectUpdateHandlerOptimizeTest.java
** solr/core/src/test/org/apache/solr/request/TestWriterPerf.java (just a nicer
way to look and afterwards use rsp.getException())
** solr/core/src/java/org/apache/solr/servlet/DirectSolrConnection.java (just a
nicer way to look and afterwards use rsp.getException())
**
solr/core/src/java/org/apache/solr/client/solrj/embedded/EmbeddedSolrServer.java
(just a nicer way to look and afterwards use rsp.getException())
** solr/solrj/src/java/org/apache/solr/client/solrj/impl/LBHttpSolrServer.java
> Versioning - optimistic locking
> -------------------------------
>
> Key: SOLR-3178
> URL: https://issues.apache.org/jira/browse/SOLR-3178
> Project: Solr
> Issue Type: New Feature
> Components: update
> Affects Versions: 3.5
> Environment: All
> Reporter: Per Steffensen
> Assignee: Per Steffensen
> Labels: RDBMS, insert, locking, nosql, optimistic, uniqueKey,
> update, versioning
> Fix For: 4.0
>
> Attachments: SOLR-3173_3178_3382_3428_plus.patch,
> SOLR-3173_3178_3382_3428_plus.patch, SOLR-3178.patch,
> SOLR_3173_3178_3382_plus.patch
>
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> In order increase the ability of Solr to be used as a NoSql database (lots of
> concurrent inserts, updates, deletes and queries in the entire lifetime of
> the index) instead of just a search index (first: everything indexed (in one
> thread), after: only queries), I would like Solr to support versioning to be
> used for optimistic locking.
> When my intent (see SOLR-3173) is to update an existing document, I will need
> to provide a version-number equal to "the version number I got when I fetched
> the existing document for update" plus one. If this provided version-number
> does not correspond to "the newest version-number of that document at the
> time of update" plus one, I will get a VersionConflict error. If it does
> correspond the document will be updated with the new one, so that "the newest
> version-number of that document" is NOW one higher than before the update.
> Correct but efficient concurrency handling.
> When my intent (see SOLR-3173) is to insert a new document, the version
> number provided will not be used - instead a version-number 0 will be used.
> According to SOLR-3173 insert will only succeed if a document with the same
> value on uniqueKey-field does not already exist.
> In general when talking about different versions of "the same document", of
> course we need to be able to identify when a document "is the same" - that,
> per definition, is when the values of the uniqueKey-fields are equal.
> The functionality provided by this issue is only really meaningfull when you
> run with "updateLog" activated.
> This issue might be solved more or less at the same time as SOLR-3173, and
> only one single SVN patch might be given to cover both issues.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
