Re: Streaming large updates with SolrJ
Thanks Hoss, that's a good explanation. I don't have much experience with the non-sugar parts of the API, so this was a good summary. I suppose I can at least help out the client heap size this way. On Wed, Jul 2, 2014 at 10:14 PM, Chris Hostetter hossman_luc...@fucit.org wrote: : Now that I think about it, though, is there a way to use the Update Xml : messages with something akin to the cloud solr server? I only see examples : posting to actual Solr instances, but we really need to be able to take : advantage of the zookeepers to send our updates to the appropriate servers. Part of your confusion may be that there are 2 different way of leveraging the SolrServer APIs (either CloudSolrServer, or any other SolrServer implementation)... * syntactic sugar apis like SolrServer.add(...) which require SolrInputDocuments * the lower level methods like SolrRequest.process(solrServer) ...with the later, you can subclass AbstractUpdateRequest and implement getContentStreams() to send whatever (lazy constructed) stream of bytes you want to Solr. Altenatively: you could conider subclassing SolrInputField with something thta knows how to lazy fetch the data you want to stream across the wire, and then (unless i'm missing something?) you can still use the sugar APIs with SolrInputDocuments but only individual field values will need to exist in RAM at any one time (as the BinaryWriter or XmlWriter calls SolrInputField.getValues() on your custom class to stream over the wire) However: if you are using SolrCloud, none of this will help you work arround the previuosly mentioned SOLR-6199, which affects how much RAM Solr needs to use on the server side when forwarding docs arround to replicas. -Hoss http://www.lucidworks.com/ -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13*
Re: Streaming large updates with SolrJ
: Now that I think about it, though, is there a way to use the Update Xml : messages with something akin to the cloud solr server? I only see examples : posting to actual Solr instances, but we really need to be able to take : advantage of the zookeepers to send our updates to the appropriate servers. Part of your confusion may be that there are 2 different way of leveraging the SolrServer APIs (either CloudSolrServer, or any other SolrServer implementation)... * syntactic sugar apis like SolrServer.add(...) which require SolrInputDocuments * the lower level methods like SolrRequest.process(solrServer) ...with the later, you can subclass AbstractUpdateRequest and implement getContentStreams() to send whatever (lazy constructed) stream of bytes you want to Solr. Altenatively: you could conider subclassing SolrInputField with something thta knows how to lazy fetch the data you want to stream across the wire, and then (unless i'm missing something?) you can still use the sugar APIs with SolrInputDocuments but only individual field values will need to exist in RAM at any one time (as the BinaryWriter or XmlWriter calls SolrInputField.getValues() on your custom class to stream over the wire) However: if you are using SolrCloud, none of this will help you work arround the previuosly mentioned SOLR-6199, which affects how much RAM Solr needs to use on the server side when forwarding docs arround to replicas. -Hoss http://www.lucidworks.com/
Re: Streaming large updates with SolrJ
Now that I think about it, though, is there a way to use the Update Xml messages with something akin to the cloud solr server? I only see examples posting to actual Solr instances, but we really need to be able to take advantage of the zookeepers to send our updates to the appropriate servers. Thanks, Joe On Sat, Jun 28, 2014 at 5:17 PM, Joe Gresock jgres...@gmail.com wrote: Yeah, I think that's what I'll have to do, Mikhail. I was just testing the waters to see if there was a way to do it with SolrJ. On Sat, Jun 28, 2014 at 4:11 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: Joe, if the heap is so tight, couldn't you post http://wiki.apache.org/solr/UpdateXmlMessages by own optimized code? On Sat, Jun 28, 2014 at 3:13 AM, Joe Gresock jgres...@gmail.com wrote: Is there a standard way to stream updates to Solr using SolrJ? Specifically, we have some atomic updates for large field values (hundreds of MB) we'd like to send. We're currently sending partial updates using SolrInputDocument objects, but we'd love to be able to keep less on the heap in our client code. Thanks, Joe -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13*
Re: Streaming large updates with SolrJ
Joe, Looking at CloudSolrServer we can see how challenging it could be. I can suggest just a clue - you can extend BinaryResponseParser and make it lazy, that's what you need. Then, you can set to for LBHttpSolrServer, and pass it into CloudSolrServer. Wish you a good journey! On Sun, Jun 29, 2014 at 7:09 PM, Joe Gresock jgres...@gmail.com wrote: Now that I think about it, though, is there a way to use the Update Xml messages with something akin to the cloud solr server? I only see examples posting to actual Solr instances, but we really need to be able to take advantage of the zookeepers to send our updates to the appropriate servers. Thanks, Joe On Sat, Jun 28, 2014 at 5:17 PM, Joe Gresock jgres...@gmail.com wrote: Yeah, I think that's what I'll have to do, Mikhail. I was just testing the waters to see if there was a way to do it with SolrJ. On Sat, Jun 28, 2014 at 4:11 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: Joe, if the heap is so tight, couldn't you post http://wiki.apache.org/solr/UpdateXmlMessages by own optimized code? On Sat, Jun 28, 2014 at 3:13 AM, Joe Gresock jgres...@gmail.com wrote: Is there a standard way to stream updates to Solr using SolrJ? Specifically, we have some atomic updates for large field values (hundreds of MB) we'd like to send. We're currently sending partial updates using SolrInputDocument objects, but we'd love to be able to keep less on the heap in our client code. Thanks, Joe -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com
Re: Streaming large updates with SolrJ
Hi Joe, I think it is not possible with SolrJ/SolrInputDocument. Please vote/watch the issue : https://issues.apache.org/jira/browse/SOLR-6199 Ahmet On Saturday, June 28, 2014 2:13 AM, Joe Gresock jgres...@gmail.com wrote: Is there a standard way to stream updates to Solr using SolrJ? Specifically, we have some atomic updates for large field values (hundreds of MB) we'd like to send. We're currently sending partial updates using SolrInputDocument objects, but we'd love to be able to keep less on the heap in our client code. Thanks, Joe -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength. *-Philippians 4:12-13*
Re: Streaming large updates with SolrJ
Joe, if the heap is so tight, couldn't you post http://wiki.apache.org/solr/UpdateXmlMessages by own optimized code? On Sat, Jun 28, 2014 at 3:13 AM, Joe Gresock jgres...@gmail.com wrote: Is there a standard way to stream updates to Solr using SolrJ? Specifically, we have some atomic updates for large field values (hundreds of MB) we'd like to send. We're currently sending partial updates using SolrInputDocument objects, but we'd love to be able to keep less on the heap in our client code. Thanks, Joe -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com
Re: Streaming large updates with SolrJ
Yeah, I think that's what I'll have to do, Mikhail. I was just testing the waters to see if there was a way to do it with SolrJ. On Sat, Jun 28, 2014 at 4:11 PM, Mikhail Khludnev mkhlud...@griddynamics.com wrote: Joe, if the heap is so tight, couldn't you post http://wiki.apache.org/solr/UpdateXmlMessages by own optimized code? On Sat, Jun 28, 2014 at 3:13 AM, Joe Gresock jgres...@gmail.com wrote: Is there a standard way to stream updates to Solr using SolrJ? Specifically, we have some atomic updates for large field values (hundreds of MB) we'd like to send. We're currently sending partial updates using SolrInputDocument objects, but we'd love to be able to keep less on the heap in our client code. Thanks, Joe -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13* -- Sincerely yours Mikhail Khludnev Principal Engineer, Grid Dynamics http://www.griddynamics.com mkhlud...@griddynamics.com -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13*
Streaming large updates with SolrJ
Is there a standard way to stream updates to Solr using SolrJ? Specifically, we have some atomic updates for large field values (hundreds of MB) we'd like to send. We're currently sending partial updates using SolrInputDocument objects, but we'd love to be able to keep less on the heap in our client code. Thanks, Joe -- I know what it is to be in need, and I know what it is to have plenty. I have learned the secret of being content in any and every situation, whether well fed or hungry, whether living in plenty or in want. I can do all this through him who gives me strength.*-Philippians 4:12-13*
