http://svn.apache.org/viewvc?view=revision&revision=1390274
SOLR-3883: Distributed indexing forwards non-applicable request params.
> distributed indexing forwards non-applicable request params
> ---
>
>
http://svn.apache.org/viewvc?view=revision&revision=1390274
SOLR-3883: Distributed indexing forwards non-applicable request params.
> distributed indexing forwards non-applicable request params
> ---
>
>
open a new issue to consider how we want to handle NOW values with distrib
indexing.
> distributed indexing forwards non-applicable request params
> ---
>
> Key: SOLR-3883
>
[
https://issues.apache.org/jira/browse/SOLR-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Miller updated SOLR-3883:
--
Fix Version/s: 5.0
> distributed indexing forwards non-applicable request par
rk so far.
> distributed indexing forwards non-applicable request params
> ---
>
> Key: SOLR-3883
> URL: https://issues.apache.org/jira/browse/SOLR-3883
> Project: Solr
x27;s own separate issue. I think we
should get this change in. Anything I still may be missing?
> distributed indexing forwards non-applicable request params
> ---
>
> Key: SOLR-3883
>
[
https://issues.apache.org/jira/browse/SOLR-3883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mark Miller reassigned SOLR-3883:
-
Assignee: Mark Miller
> distributed indexing forwards non-applicable request par
king further, it seems all ways commitWithin can come in,
it gets set on the update command, and our update command distributor then
respects sending them on.
> distributed indexing forwards non-applicable request
adds, deletes, deleteByQuery?
I suppose not - this just covers the commit case. It also covers commitWithin
from xml or whatever, but it seems we do also have to worry about the case
where it's just a request params on an add or delete.
> distributed indexing forwards no
dled within the command - no need
for request params.
Is this also the case for commitWithin on adds, deletes, deleteByQuery?
> distributed indexing forwards non-applicable request params
> ---
>
>
.
> distributed indexing forwards non-applicable request params
> ---
>
> Key: SOLR-3883
> URL: https://issues.apache.org/jira/browse/SOLR-3883
> Project: Solr
>
rams for commits: softCommit,
waitSearcher?
I've got to brush up on the NOW param...
I think the commit params are all fine - we build a command from the orig
request and then pass that on.
We do need to do UpdateParams.UPDATE_CHAIN, but I am not 100% on what else yet.
>
are necessary/desirable for updates.
Brainstorming off the top of my head:
- Params for updates: commitWithin, NOW?
- Params for commits: softCommit, waitSearcher?
> distributed indexing forwards no
the past, but it was tied up in
an issue with a lot of other stuff.
Was trying to avoid having to guess everything that should go, but looks like
the reverse may be worse.
> distributed indexing forwards non-applicable request
Yonik Seeley created SOLR-3883:
--
Summary: distributed indexing forwards non-applicable request
params
Key: SOLR-3883
URL: https://issues.apache.org/jira/browse/SOLR-3883
Project: Solr
Issue
view=revision&revision=1365014
to trunk and 4x branch that fixes DBQ still doing leader-type stuff even when
FROMLEADER was set.
> handle deleteByQuery reorders in distributed indexing
> -
>
>
> handle deleteByQuery reorders in distributed indexing
> -
>
> Key: SOLR-3559
> URL: https://issues.apache.org/jira/browse/SOLR-3559
> Project: Solr
> Is
[
https://issues.apache.org/jira/browse/SOLR-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley resolved SOLR-3559.
Resolution: Fixed
Fix Version/s: 4.0
> handle deleteByQuery reorders in distribu
this. DBQ needs to clear *all*
maps in the update log (i.e. prevMap, prevMap2) otherwise lookups can get a
false hit during a concurrent commit.
> handle deleteByQuery reorders in distributed indexing
> -
>
>
and
associated deleteByQueries more as a single unit in the update handler rather
than in the distrib update processor.
> handle deleteByQuery reorders in distributed indexing
> -
>
> Ke
only fail with greater than 1 write
thread in conjunction with realtime-get.
> handle deleteByQuery reorders in distributed indexing
> -
>
> Key: SOLR-3559
> URL: https://issues.apa
m not sure at
this point if it's a logic bug, a concurrency bug, or a test bug...
> handle deleteByQuery reorders in distributed indexing
> -
>
> Key: SOLR-3559
> URL: https:/
versions. clearIndex() with it's DBQ of *:* was messing things up since it
could also delete documents in the future (docs that were using fake versions
that were lower than the DBQ).
Time to add in the stress tests.
> handle deleteByQuery reorders in distributed
.
Unfortunately some other tests now fail.
> handle deleteByQuery reorders in distributed indexing
> -
>
> Key: SOLR-3559
> URL: https://issues.apache.org/jira/b
; handle deleteByQuery reorders in distributed indexing
> -
>
> Key: SOLR-3559
> URL: https://issues.apache.org/jira/browse/SOLR-3559
> Project: Solr
> Issue Type: Improvement
>
[
https://issues.apache.org/jira/browse/SOLR-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yonik Seeley reassigned SOLR-3559:
--
Assignee: Yonik Seeley
> handle deleteByQuery reorders in distributed index
Yonik Seeley created SOLR-3559:
--
Summary: handle deleteByQuery reorders in distributed indexing
Key: SOLR-3559
URL: https://issues.apache.org/jira/browse/SOLR-3559
Project: Solr
Issue Type
2011/5/19 Yury Kats :
> I'm curious to know whether Distributed Indexing is on the roadmap for 4.0.
> Looking at JIRA and past dev list discussions, there seem to have been
> efforts around DI in the past, but no recent activity.
>
> Is this being worked on, postponed or
Hi,
I'm curious to know whether Distributed Indexing is on the roadmap for 4.0.
Looking at JIRA and past dev list discussions, there seem to have been
efforts around DI in the past, but no recent activity.
Is this being worked on, postponed or deemed not important enough all together?
T
[
https://issues.apache.org/jira/browse/SOLR-2293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Høydahl closed SOLR-2293.
-
Resolution: Duplicate
Use SOLR-2358
> SolrCloud distributed index
e first, and then the add.
And the commit at the end could sneak in front of some adds and
deletes still in progress on other threads.
For true distributed indexing, I think we'll want a version number
somehow (perhaps based on timestamp by default) so updates can be
ordered, and all nodes can
I've uploaded a patch of what we've done so far:
https://issues.apache.org/jira/browse/SOLR-2358
It's still very much work in progress and there are some obvious issues
which are being resolved at the moment (such as the inefficient method of
waiting for all the docs to be processed before distri
I haven't had time to follow all of this discussion, but this issue might help:
https://issues.apache.org/jira/browse/SOLR-2355
It's an implementation of the basic
http://localhost:8983/solr/update/csv?shards=shard1,shard2...
-Yonik
http://lucidimagination.com
On Mon, Feb 7, 2011 at 8:55 AM, Upa
Surely you want to be implementing an UpdateRequestProcessor,
rather than a RequestHandler.
The ContentStreamHandlerBase, in the handleRequestBody method
gets an UpdateRequestProcessor and uses it to process the
request. What we need is that handleRequestBody method to, as you
have suggested, chec
ctionsPerHost and maxTotalConnections
params? Won't the values used for distributed search be a
little high for distributed indexing?
You are right, these will likely be lower for distributed
indexing, however I'd suggest not worrying about it for now, as
it is easy to twea
Hey,
We're making good progress, but our DistributedUpdateRequestHandler is
having a bit of an identity crisis, so we thought we'd ask what other
people's opinions are. The current situation is as follows:
We've added a method to ContentStreamHandlerBase to check if an update
request is distribut
rocessor uses a MultiThreadedHttpConnectionManager to send
> parallel updates to shards, can anyone give some appropriate values to be
> used for the defaultMaxConnectionsPerHost and maxTotalConnections params?
> Won't the values used for distributed search be a little high for
> di
both
needs effectively?
3. Our update processor uses a
MultiThreadedHttpConnectionManager to send parallel updates to
shards, can anyone give some appropriate values to be used for
the defaultMaxConnectionsPerHost and maxTotalConnections
params? Won't the values used for distributed se
maxTotalConnections params?
Won't the values used for distributed search be a little high for
distributed indexing?
Thanks,
Alex
On Tue, 01 Feb 2011 19:52 -0800, "Lance Norskog"
wrote:
> Another use case is that N indexers operate independently, all pulling
> data from the same database. Each has a separate query to get the
> documents in its policy.
But surely in this case, you are externalising the policy, and Solr
do
Another use case is that N indexers operate independently, all pulling
data from the same database. Each has a separate query to get the
documents in its policy.
On Tue, Feb 1, 2011 at 12:38 PM, Upayavira wrote:
>
> On Tue, 01 Feb 2011 19:04 +, "Alex Cowell" wrote:
>
> I noticed there is a
On Tue, 01 Feb 2011 19:04 +, "Alex Cowell"
wrote:
I noticed there is a comment in the
org.apache.solr.servlet.DirectSolrConnection class which
reads, "//Find a way to turn List into
File/SolrDocument". Did anyone find a way to do this?
Turns out that comment was left over from som
>
> I noticed there is a comment in the
> org.apache.solr.servlet.DirectSolrConnection class which reads, "//Find a
> way to turn List into File/SolrDocument". Did anyone find a
> way to do this?
>
Turns out that comment was left over from some experimenting one of our team
was doing. But I suppos
>
> Your code looks fine to me, except it should take in a SolrDocument
> object or list of, rather than strings. Then, for your Hash version, you
> can take a hash of the "id" field.
>
As far as I can see I have access to a List that
> represents all of the files being POSTed. Do I want to open t
Hello
Thanks for your prompt reply.
In regards to using a SolrDocument instead of Strings (and I agree
that List doesn't seem to be the best way of going) how do I
get reference to a SolrDoc?
As far as I can see I have access to a List that
represents all of the files being POSTed. Do I want to
On Tue, 01 Feb 2011 00:26 +, "William Mayor"
wrote:
> Hi Guys
>
> I've had a go at creating the ShardDistributionPolicy interface and a
> few implementations. I've created a patch
> (https://issues.apache.org/jira/browse/SOLR-2341) let me know what
> needs doing.
> Currently I assume that t
(I'm sending this on behalf of William, a guy on our team working on
ShardDistributedPolicy):
Hi Guys
I've had a go at creating the ShardDistributionPolicy interface and a
few implementations. I've created a patch
(https://issues.apache.org/jira/browse/SOLR-2341) let me know what
needs doing.
Cu
Hi Guys
I've had a go at creating the ShardDistributionPolicy interface and a
few implementations. I've created a patch
(https://issues.apache.org/jira/browse/SOLR-2341) let me know what
needs doing.
Currently I assume that the documents passed to the policy will be
represented by some kind of id
Lance,
Firstly, we're proposing a ShardDistributionPolicy interface for which
there is a default (mod of the doc ID) but other implementations are
possible. Another easy implementation would be a randomised or round
robin one.
As to threading, the first task would be to put all of the source
docu
I would suggest that a DistributedRequestUpdateHandler run
single-threaded, doing only one document at a time. If I want more
than one, I run it twice or N times with my own program.
Also, this should have a policy object which decides exactly how
documents are distributed. There are different tec
Hello Yonik,
On Thu, 2011-01-27 at 08:01 -0500, Yonik Seeley wrote:
> Making it easy for clients I think is key... one should be able to
> update any node in the solr cluster and have solr take care of the
> hard part about updating all relevant shards. This will most likely
> involve an update p
Hi Yonik and Upayavira,
Thank you both for your insightful responses. We now have a much better
understanding of how to implement distributed indexing, although no doubt
more issues will emerge along the way.
Just to clarify (and for critique), our approach goes something like this:
We will use
updating all relevant shards. This will most likely
> involve an update processor. This approach allows all existing update
> methods (including things like CSV file upload) to still work
> correctly.
>
> Does that then imply that distributed indexing would become the default
> m
to mimic in a distributed indexing scenario. (without
effectively implementing distributed transactions).
I guess the simplest would be to find a way to report back to the user
that documents for these shards succeeded, and documents for these
shards failed, and here's the error. The issue he
an update processor. This approach allows all
existing update
methods (including things like CSV file upload) to still work
correctly.
Does that then imply that distributed indexing would become the
default method of indexing? What if a user, for some reason,
wanted to only target one specific
Hi Soheb,
On Wed, 26 Jan 2011 16:29 +, "Soheb Mahmood"
wrote:
> We are going to implement distributed indexing for Solr - without the
> use of SolrCloud (so it can be easily up-scaled). We have a deadline by
> February to get this done, so we need to get cracking ;)
:-
pdate
> methods (including things like CSV file upload) to still work
> correctly.
>
Does that then imply that distributed indexing would become the default
method of indexing? What if a user, for some reason, wanted to only target
one specific node in a cluster? Wouldn't a message need t
On Wed, Jan 26, 2011 at 11:29 AM, Soheb Mahmood wrote:
> We were wondering if there was a simple way of
> applying these changes we wrote in Java across all the other languages.
Making it easy for clients I think is key... one should be able to
update any node in the solr cluster and have solr ta
above. You see, we are UCL University students that have
been given this task to contribute something to the Apache SolrCloud
open source project.
We are doing this higher level goal of basically adding native
distributed indexing into Solr so it indirectly benefits SolrCloud in
the future. We a
al
> indexing?
>
> Regards,
>
> Alex
>
>
>
> On Wed, Jan 26, 2011 at 4:29 PM, Soheb Mahmood wrote:
>
>> Hello,
>>
>> We are going to implement distributed indexing for Solr - without the
>> use of SolrCloud (so it can be easily up-scaled). We h
are no more documents left to index.
Also, how will we deal with failures? Should we simply return a list of all
documents which weren't indexed or have a retry period after the initial
indexing?
Regards,
Alex
On Wed, Jan 26, 2011 at 4:29 PM, Soheb Mahmood wrote:
> Hello,
>
> We
Hello,
We are going to implement distributed indexing for Solr - without the
use of SolrCloud (so it can be easily up-scaled). We have a deadline by
February to get this done, so we need to get cracking ;)
So far, we've had a look at the solr classes and thought about
distributed indexi
SolrCloud distributed indexing
--
Key: SOLR-2293
URL: https://issues.apache.org/jira/browse/SOLR-2293
Project: Solr
Issue Type: New Feature
Components: SolrCloud
Reporter: Jan Høydahl
Add
63 matches
Mail list logo
