Jan, I feel terrible for leaving you hanging on this - I missed this email
entirely. Seems some of these should be made JIRA issues if they are not
already?
bq. j) Question: Is ReplicationHandler ZK-aware yet?
As I think you now know, not yet ;)
- Mark
On Feb 14, 2011, at 4:40 PM, Jan Høydahl wrote:
> Some more comments:
>
> f) For consistency, the JAVA OPTIONS should all be prefixed with solr.* even
> if they are related to embedded ZK
> -Dsolr.hostPort=8900 -Dsolr.zkRun -Dsolr.zkHost=localhost:9900
> -Dsolr.zkBootstrap_confdir=./solr/conf
>
> g) I often share parts of my config between cores, e.g. a common schema.xml
> or synonyms.xml
> In the file-based mode I can thus use ../../common_conf/synonyms.xml or
> similar.
> I have not tried to bootstrap such a config into ZK but I assume it will
> not work
> ZK mode should support such a use case either by supporting notations like
> ".."
> or by allowing an explicit zk name space:
> zk://configs/common-cfg/synonyms.xml
>
> h) Support for dev / test / prod environments
> In real life you want to develop in one environment, test in another and
> run production in a third
> Thus, the ZK data structure should have a clear separation between logical
> feature configuration and
> physical deployment config.
>
> Perhaps a new level above /COLLECTIONS could be used to model this, e.g.
> /ENV/PROD/COLLECTIONS/WEB/SHARDS/shardA/prod01.server.com:8080
> /ENV/PROD/COLLECTIONS/WEB/SHARDS/shardB/prod02.server.com:8080
> /ENV/PROD/COLLECTIONS/FILES/SHARDS/shardA/prod03.server.com:8080
> /ENV/TEST/COLLECTIONS/WEB/SHARDS/shardA/test01.server.com:8080
> /ENV/TEST/COLLECTIONS/WEB/SHARDS/shardB/test01.server.com:9090
>
> /ENV/TEST/COLLECTIONS/FILES[@configName=TESTFILES]/SHARDS/shardA/test01.server.com:7070
>
> When starting solr we may specify environment: -Dsolr.env=TEST (or
> configure a default)
> The main benefit is that we can maintain and store one single ZK config in
> our SCM,
> distribute the same configs to all servers, and if you like, point all envs
> to the same ZK ensemble.
>
> In the future, we can use this for automatic install of a new node as well:
> By simply adding a ZK entry on the right place, the node can discover "who
> it is" from ZK.
>
> i) Ideally, no config inside conf should contain host names.
> My DIH config will most likely include server names, which will be
> different between TEST and PROD
> This could be solved as above, by letting the collection in TEST use
> another configName than PROD,
> but for some use cases, it might be more elegant to swap out a hardcoded
> string with a ZK node
> in a generic way, such as jdbcString="my-hardcoded-string" to
> jdbcString="${zk://ENV/PROD/jdbcstrA}"
>
> j) Question: Is ReplicationHandler ZK-aware yet?
>
> --
> Jan Høydahl, search solution architect
> Cominvent AS - www.cominvent.com
>
> On 10. feb. 2011, at 16.10, Jan Høydahl wrote:
>
>> Hi,
>>
>> I have so far just tested the examples and got a N by M cluster running. My
>> feedback:
>>
>> a) First of all, a major update of the SolrCloud Wiki is needed, to clearly
>> state what is in which version, what are current improvement plans and get
>> rid of outdated stuff. That said I think there are many good ideas there.
>>
>> b) The "collection" terminology is too much confused with "core", and should
>> probably be made more distinct. I just tried to configure two cores on the
>> same Solr instance into the same collection, and that worked fine, both as
>> distinct shards and as same shard (replica). The wiki examples give the
>> impression that "collection1" in
>> localhost:8983/solr/collection1/select?distrib=true is some magic collection
>> identifier, but what it really does is doing the query on the *core* named
>> "collection1", looking up what collection that core is part of and
>> distributing the query to all shards in that collection.
>>
>> c) ZK is not designed to store large files. While the files in conf are
>> normally well below the 1M limit ZK imposes, we should perhaps consider
>> using a lightweight distributed object or k/v store for holding the /CONFIGS
>> and let ZK store a reference only
>>
>> d) How are admins supposed to update configs in ZK? Install their favourite
>> ZK editor?
>>
>> e) We should perhaps not be so afraid to make ZK a requirement for Solr in
>> v4. Ideally you should interact with a 1-node Solr in the same manner as you
>> do with a 100-node Solr. An example is the Admin GUI where the "schema" and
>> "solrconfig" links assume local file. This requires decent tool support to
>> make ZK interaction intuitive, such as "import" and "export" commands.
>>
>> --
>> Jan Høydahl, search solution architect
>> Cominvent AS - www.cominvent.com
>>
>> On 19. jan. 2011, at 21.07, Mark Miller wrote:
>>
>>> Hello Users,
>>>
>>> About a little over a year ago, a few of us started working on what we
>>> called SolrCloud.
>>>
>>> This initial bit of work was really a combination of laying some base work
>>> - figuring out how to integrate ZooKeeper with Solr in a limited way,
>>> dealing with some infrastructure - and picking off some low hanging search
>>> side fruit.
>>>
>>> The next step is the indexing side. And we plan on starting to tackle that
>>> sometime soon.
>>>
>>> But first - could you help with some feedback?ISome people are using our
>>> SolrCloud start - I have seen evidence of it ;) Some, even in production.
>>>
>>> I would love to have your help in targeting what we now try and improve.
>>> Any suggestions or feedback? If you have sent this before, I/others likely
>>> missed it - send it again!
>>>
>>> I know anyone that has used SolrCloud has some feedback. I know it because
>>> I've used it too ;) It's too complicated to setup still. There are still
>>> plenty of pain points. We accepted some compromise trying to fit into what
>>> Solr was, and not wanting to dig in too far before feeling things out and
>>> letting users try things out a bit. Thinking that we might be able to
>>> adjust Solr to be more in favor of SolrCloud as we go, what is the ideal
>>> state of the work we have currently done?
>>>
>>> If anyone using SolrCloud helps with the feedback, I'll help with the
>>> coding effort.
>>>
>>> - Mark Miller
>>> -- lucidimagination.com
>>
>
- Mark Miller
lucidimagination.com
