[
https://issues.apache.org/jira/browse/SOLR-7535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15080578#comment-15080578
]
Joel Bernstein commented on SOLR-7535:
--------------------------------------
Locally I won't be able to simulate a large cluster exporting at 10+ million
per-second. I've mainly just been thinking about the mechanics of Blocking IO.
It's possible that Blocking IO will work just fine in this scenario:
1) Each shard exports Tuples until its network memory buffer is full, then it
will block.
2) Each worker will read Tuples and perform the indexing.
3) As the workers read more Tuples it will clear space in the shards network
memory buffer to export more tuples.
In this scenario we won't have long periods of blocking on the shards, it will
be lot's of stops and starts. Which in theory should be fine.
Maybe the thing to do is to go with the current implementation and we can
always add a buffering implementation if we need to.
> Add UpdateStream to Streaming API and Streaming Expression
> ----------------------------------------------------------
>
> Key: SOLR-7535
> URL: https://issues.apache.org/jira/browse/SOLR-7535
> Project: Solr
> Issue Type: New Feature
> Components: clients - java, SolrJ
> Reporter: Joel Bernstein
> Assignee: Joel Bernstein
> Priority: Minor
> Attachments: SOLR-7535.patch, SOLR-7535.patch, SOLR-7535.patch,
> SOLR-7535.patch, SOLR-7535.patch, SOLR-7535.patch
>
>
> The ticket adds an UpdateStream implementation to the Streaming API and
> streaming expressions. The UpdateStream will wrap a TupleStream and send the
> Tuples it reads to a SolrCloud collection to be indexed.
> This will allow users to pull data from different Solr Cloud collections,
> merge and transform the streams and send the transformed data to another Solr
> Cloud collection.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
