It was a very good explanation, Jack!
I believe I have heard most of it before, so it is really not new for
me. I DO understand that the name "replica" and "replication-factor" CAN
be justified, but it requires a long and thorough explanation. And thats
the point. A good name for a concept means that:
* The name is among the first that pops up in your mind when you think
about the concept, or at least you can make a very short explanation why
you choose this name for that concept
* When a (fairly educated) newcomer hears the name for the first time,
his first thoughts about the concept it covers is as close as possible
to the actual concept
Good metrics for whether or not we have good names must therefore be
1) The frequency of questions about the concepts behind the names
2) The frequency of wrong usage of names (cases where people actually
didnt understand the concept behind the name, didnt ask (1. above) and
just used the name for what he thought it meant)
3) The length of the explanation of why you chose this name for that concept
Ad 1)
I counted several questions just this week. Especially I noted "Replica
(Replica of _what_?)" in the original post of this thread. Whether we
want it or not, newcomers will keep "not getting" the concept of replica
or getting it wrong. Why? Because it is a bad name.
Ad 2)
I also counted several cases where names where used completely wrong
this week.
Ad 3)
Take a look at the length of Jacks great post below, and take a look at
the length of this mail-thread.
I believe we will do better on the metrics if we use
node/collection/shard/shard-instance/index instead of
node/collection/shard/replica/(core/)index, and use instances-per-shard
instead of replication-factor. And say that "core" is the same as a
"shard-instance", but typically used in a non/pre-Cloud context. That
index is a physical lucene thing - and nothing but that. That
collections and shards are logical concepts. That a shard-instance is a
physical instance of a shard implemented using a lucene index persisting
its data on physical disk.
My only interest here is to try to pull the project in a good direction.
You just get my opinion. Keep it simple and no bullshit.
This entire discussion is great I think, but it probably belong on
dev-list (or maybe on a JIRA).
I belive Alexandre Rafalovitch got his answer already :-) To the level a
clean answer exists at the moment.
Regards, Per Steffensen
On 1/4/13 2:54 PM, Jack Krupansky wrote:
Replication makes perfect sense even if our explanations so far do not.
A shard is an abstraction of a subset of the data for a collection.
A replica is an instance of the data of the shard and instances of
Solr servers that have indicated a readiness to service queries and
updates for the data. Alternatively, a replica is a node which has
indicated a readiness to receive and serve the data of a shard, but
may not have any data at the moment.
Lets describe it operationally for SolrCloud: If data comes in to any
replica of a shard it will automatically and quickly be "replicated"
to all other replicas of the shard. If a new replica of a shard comes
up it will be streamed all of the data from the another replica of the
shard. If an existing replica of a shard restarts or reconnects to the
cluster, it will be streamed updates of any new data since it was last
updated from another replica of the shard.
Replication is simply the process of assuring that all replicas are
kept up to date. That's the same abstract meaning as for Master/Slave
even though the operational details are somewhat different. The goal
remains the same.
Replication factor is the number of instances of the data of the shard
and instances of Solr servers that can service queries and updates for
the data. Alternatively, the replication factor is the number of nodes
of the SolrCloud cluster which have indicated a readiness to receive
and serve the data of a shard, but may not have any data at the moment.
A node is an instance of Solr running in a Java JVM that has indicated
to the Zookeeper ensemble of a SolrCloud cluster that it is ready to
be a replica for a shard of a collection. [The latter part of that is
a bit too fuzzy - I'm not sure what the node tells Zookeeper and who
does shard assignment. I mean, does a node explicitly say what shard
it wants to be, or is that assigned by Zookeeper, or is that a node's
choice/option? But none of that changes the fact that a node
"registers" with Zookeeper and then somehow becomes a replica for a
shard.]
A node (instance of a Solr server) can be a replica of shards from
multiple collections (potentially multiple shards per collection). A
node is not a replica per se, but a container that can serve multiple
collections. A node can serve as multiple replicas, each of a
different collection.
My only interest here on this user list is to understand and explain
the terms we have today and that SEEM to be working for the most part,
even though we may not have defined them carefully enough and used
them consistently enough.
If somebody want to propose an alternative terminology - fine, discuss
that on the dev list and/or file a Jira.
I won't claim that my definitions are perfect (yet), but perfecting
the definitions (for users) should be separated from changing the
terms themselves.
-- Jack Krupansky
-----Original Message----- From: Per Steffensen
Sent: Friday, January 04, 2013 2:49 AM
To: solr-user@lucene.apache.org
Subject: Re: Terminology question: Core vs. Collection vs...
On 1/3/13 5:58 PM, Walter Underwood wrote:
A "factor" is multiplied, so multiplying the leader by a
replicationFactor of 1 means you have exactly one copy of that shard.
I think that recycling the term "replication" within Solr was
confusing, but it is a bit late to change that.
wunder
Yes, the term "factor" is not misleading, but the term "replication" is.
If we keep calling shard-instances for "Replica" I guess "replicaFactor"
will be ok - at least much better than "replicationFactor". But it would
still be better with e.g. "ShardInstance" and "InstancesPerShard"
