On Mon, Feb 2, 2009 at 9:20 PM, Jon Baer <jonb...@gmail.com> wrote: > Hi, > > Sorry I know this exists ... > > "If an API supports chunking (when the dataset is too large) multiple calls > need to be made to complete the process. XPathEntityprocessor supports this > with a transformer. If transformer returns a row which contains a field * > $hasMore* with a the value "true" the Processor makes another request with > the same url template (The actual value is recomputed before invoking ). A > transformer can pass a totally new url too for the next call by returning a > row which contains a field *$nextUrl* whose value must be the complete url > for the next call." > > But is there a true example of it's use somewhere? Im trying to figure out > if I know before import that I have 56 "pages" to index how to set this up > properly. (And how to set it up if pages need to be determined by something > in the feed, etc). Let us assume that we are working w/ Solr xml interface as the datasource the url may contain start=x&rows=y. assume that we have 100's of rows to be fetched and we wish to chunk it.
you can change the variable 'start' on each xml fetched (it does not hurt even if it is set for each row) and you can compute find $hasMore from the xml itself. Setting a variable can be done by putting it into the returned row from a transformer. > > Thanks. > > - Jon > -- --Noble Paul
