Re: [PR] SOLR-14673: Add bin/solr stream CLI [solr]
epugh commented on PR #2479: URL: https://github.com/apache/solr/pull/2479#issuecomment-2149443128 Thanks for sharing the feedback @gerlowskija ! I think the value of the tool is only there if your second comment about being able to run a streaming expression locally is valid, and then having it do what yoru first comment highlights falls out easy, otherwise it really is a thin wrapper/duplication of the `bin/solr api` call. Especially without any special value add in formatting tuples or error handling etc. I do believe the second part is the really cool thing, that I can run a streaming expression locally and use it to process some data. We clearly need some way of specifying where the processing is happening, in the cluster or locally. I was trying to think if we have any other places in Solr where we define "Where am I doing work" that might provide a name for a parameter. `bin/solr stream --environment cluster BLAH` ? The `search()` expression has a `qt` parameter.. `bin/solr stream -qt=/stream BLAH` ? Reading through docs more, we have the `parralel()` and it refers to workers.Maybe the command should be something like `bin/solr stream --workers=local BLAH` which would run on your laptop, and if you don't specify --workers then it runs on the cluster via `/stream`? I have found that lots of streaming expressions don't require a Solr connection, especially during development. I'm just iterating on the logic, and I'm starting and ending iwth tuples.. it's only later when I get the mappings etc working that I then move to adding in my `search()` or `update()` clauses. Also, as far as docs go, we have a LONG way to go in Streaming expressions. It's both the best docuemnted code, with all the howtos and guides, but also, I find a million expressions that exist but don't show up in our reference docs ;-). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
Re: [PR] SOLR-14673: Add bin/solr stream CLI [solr]
gerlowskija commented on PR #2479: URL: https://github.com/apache/solr/pull/2479#issuecomment-2145825393 A few high-level questions/concerns: 1. `bin/solr` already has an "api" tool, which can be used to invoke streaming expressions e.g. `bin/solr api -get "$SOLR_URL/techproducts/stream?expr=search(techproducts)"`. I'm all for syntactic-sugar, but I wonder whether this is worth the maintenance cost if the main thing that it "buys" us is saving people from having to provide the full API path as the "api" tool requires? 2. If I'm reading the PR correctly, it looks like one other capability of the proposed `bin/solr stream` tool is that it can evaluate streams "locally" in some cases i.e. without a full running Solr. Which is pretty cool - you could imagine a real super-user doing some pretty involved ETL that builds off of an expression like: `update(techproducts, unique(cat(...)))`. But I'd worry about some of the documentation challenges surrounding this. For instance, how would a user know which expressions can be run locally, and which require a Solr to execute on? For expressions that have a mix of both locally and remotely-executed clauses, is there any way for a user to know which clauses are executed where? To clarify - I think the upside here is pretty cool, I'm just worried about the documentation end and what we might need to make it usable by folks in practice. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
[PR] SOLR-14673: Add bin/solr stream CLI [solr]
epugh opened a new pull request, #2479: URL: https://github.com/apache/solr/pull/2479 https://issues.apache.org/jira/browse/SOLR-14673 # Description Bring in code that @joel-bernstein wrote, but using the SolrCLI infrastructure. # Solution Another CLI client ;-) # Tests Copied over the basic tests from the patch. I still need to write an integration style test and ideally one that exercies the basic auth. # Checklist Please review the following and check all that apply: - [ ] I have reviewed the guidelines for [How to Contribute](https://github.com/apache/solr/blob/main/CONTRIBUTING.md) and my code conforms to the standards described there to the best of my ability. - [ ] I have created a Jira issue and added the issue ID to my pull request title. - [ ] I have given Solr maintainers [access](https://help.github.com/en/articles/allowing-changes-to-a-pull-request-branch-created-from-a-fork) to contribute to my PR branch. (optional but recommended) - [ ] I have developed this patch against the `main` branch. - [ ] I have run `./gradlew check`. - [ ] I have added tests for my changes. - [ ] I have added documentation for the [Reference Guide](https://github.com/apache/solr/tree/main/solr/solr-ref-guide) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org - To unsubscribe, e-mail: issues-unsubscr...@solr.apache.org For additional commands, e-mail: issues-h...@solr.apache.org
