[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 I updated the instructions and the `solr.zookeeper` business is legacy naming, but I think it does fit since it's specific to zookeeper rather than connecting to ES master. I dunno, that's just what I told myself ;) ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 +1 ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/929 Yes I pushed the change to zookeeper with Swagger. Probably just a minor detail he missed in his instructions. You can use either "localhost:9983" or "node1:9983". That address is the embedded zookeeper that Solr starts up in this case but in general it is whatever zookeeper you configure Solr to use. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 @merrimanr did you push the configs to zookeeper? ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user merrimanr commented on the issue: https://github.com/apache/metron/pull/929 +1 from me pending approval from others. Tested this in full dev and worked fine. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 Everything worked, just wondering about the questions above before I check off. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 don't we have to push the new config to zookeeper? ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 Why is it "solr.zookeeper" : "localhost:9983" and not solr.ip? ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 Ok, I wrote up a detailed testing plan to get this running in full-dev. I want to add a couple of unit tests around the new properties that I added, but at that point, this is feature complete and ready to review. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 Cool, I'll run the test plan ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 Ok, I updated the configs to expose all the settings that I could find for committing and documented them. By default we operate safely, though. Though now you can use soft commits rather than durable commits if you so desire and wish to live dangerously. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 LOL, nah, I was just being crotchety. Let me see what I can do and you tell me what you think :) ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 See, we explored it. Thanks!! ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 Actually, I can make an option to let you do a soft commit instead of a hard commit if that'll be an extra layer of configuration. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 You can set it to not manually commit (set `solr.commitPerBatch` to `false` and no committing happens), but you're risking losing data if a worker dies. Honestly, want a durable commit with a fsync before you ack the tuples in a batch, otherwise you're courting data loss. This is the same strategy we do for ES and HDFS (though commit there is a fsync). That being said, I'm sensitive to performance issues around that that people may have, so I let people turn it off with a strong warning in the docs (also this was legacy behavior in the SolrWriter). ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 Sorry, maybe I misunderstood what I have read here: https://lucidworks.com/2013/08/23/understanding-transaction-logs-softcommit-and-commit-in-sorlcloud/ I thought that you could set to not have to manually commit. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user cestella commented on the issue: https://github.com/apache/metron/pull/929 What do you mean "auto-commit"? By default it will commit at the batch level as the other writers do. You can turn that behavior off and it'll commit when the Solr client decides at the cost of possibly losing data. Regarding solr vs solrcloud, to my understanding solr is the product and solrcloud is a way of running Solr. The other way is solr standalone. Originally we named things after the product, which I think is correct, and have made explicit that we support Solr running in Solr Cloud mode in the docs. ---
[GitHub] metron issue #929: METRON-1448: Update SolrWriter to conform to new collecti...
Github user ottobackwards commented on the issue: https://github.com/apache/metron/pull/929 Should we explore auto-commit? Also, maybe we should rename all this stuff to SolrCloud, since Solr and SolrCloud *are* different in some use of terms and concepts? ---
