Thanks for your quick answer, and the extensive explanation.
It appears I did not understand at all how the cron indexing was working.
I like your suggestion of reindexing only changes pages with [(index
list=game)] and automatically indexing from the last indexing time only.
Then it would probably be enough to delete site.index.game to force a full
indexing (not only from last indexing time).
I would enjoy such features very much in the next release, or even one
later. I have something working well for now, so this is not urgent.
Cheers,
Tiffany
Le lundi 11 janvier 2016 16:03:42 UTC+1, Dan a écrit :
>
> Ok, I understand what you are saying now. And I note I didn't answer your
> question, either...
>
> So here's the general way indexing is supposed to work as I remember it:
>
> 1) If you set indexing to cron in site.config, changed pages get logged on
> page site.index.cron--and the pages are NOT indexed. You then HAVE TO use a
> cron job to call some public page you create with the index function on it.
> Otherwise nothing will happen. The idea of this is to completely avoid page
> lags when you link indexing to page views.
>
> 2) Depending on the parameters you put on the cron index function several
> things could happen:
>
> [(index)] -- nothing happens. no pages specified
> [(index cron=10)] -- index 10 pages on site.index.cron. They will be
> checked for all matching rules
> [(index page.1,page.2,page.3)] -- index the 3 specified pages. They will
> be checked for all matching rules
> [(index rule=game)] -- indexes 25 (indexBatch) pages on the
> site.index.game list using the game rule defined on site.index
>
> Now the auto indexing feature, if set to cron will never populate pages
> like site.index.game. It only populates site.index.cron. Nor will it catch
> pages changes by ftp. So yes, you do need to use the action.index script to
> populate them. Using BoltWire's script capabilities to do that via a cron
> job was brilliant. Never would have thought of doing that.
>
> Having said that, I see the merit in allowing a batch parameter in the
> index function to optionally override the default setting. So [(index
> rule=game batch=10)] should be able to overrule the default site.config
> setting of 25. This is easily done by making these changes:
>
> Change the first line of BOLTFindex to
> if ($args['rule'] != '') $out = BOLTindex($args['rule'], $args['batch']);
>
> Change the first two lines of BOLTindex to
> function BOLTindex($rule='', $batch='') {
> $batch = BOLTinit(BOLTconfig('indexBatch', 25), $batch);
>
> Another problem that occurs to me reflecting on all this is [(index)]
> should default to the equivalent of cron=indexBatch. This requires a slight
> rewrite of the index function. It also has the added advantage that you can
> do [(index 10)] and it is the equivalent of [(index cron=10)]. I plan to
> include the changes above and below to the next release, unless you see
> some reason not to.
>
> function BOLTFindex($args) {
> if ($args['rule'] != '') $out = BOLTindex($args['rule'], $args['batch']);
> // works through a rule if no list
> else {
> $count = BOLTinit(BOLTconfig('indexBatch', 25), $args[1], $args['cron']);
> if (is_number($count)) {
> $cron = BOLTinit('site.index.cron', BOLTconfig('cronIndex'));
> $content = trim(BOLTloadpage($cron));
> if ($content == '') return BOLTmessage('index_current', $args);
> $pages = explode("\n", $content);
> $pages = array_unique($pages);
> if (count($pages) > $count) {
> $list = array_slice($pages, 0, $count);
> $content = implode("\n", array_slice($pages, $count));
> }
> else {
> $list = $pages;
> $content = ' ';
> }
> BOLTsavepage($cron, $content);
> }
> else {
> if ($count == '') return BOLTmessage('index_empty', $args);
> $list = explode(',', $count);
> }
> $out = BOLTindexAuto($list); // indexes pages
> }
> BOLTmessage('index_complete', $args);
> return $out;
> }
>
> Ok, now for the big question--the problem you actually raised. How do I
> automatically index changed pages which I have uploaded by ftp or via some
> custom rule? Your answer is the right one of course. But it might be
> possible to simplify it further by creating a plugin, or extending the
> index function. Here's what I envision:
>
> 1) Either call it [(indexer rule=game)] or perhaps [(index list=game)].
> The first is a plugin, the latter an extension of the core. Either way,
> this tells BoltWire to create an updated list for the specified rule.
> 2) The script simply looks up the rule, generates a pagelist based on the
> rule parameters, and appends it to page site.index.game.
> 3) To be really smart, we would somehow record the last time that rule was
> "listed" and only include pages changes since then.
>
> If we could get #3 to work well, we could make this happen automatically
> anytime you call [(index rule=game)] before starting the indexing process.
> That way it just works. As for accomplishing #3, I propose saving the
> last listed date as an info var hidden in a data var on site.index. It
> might be a bit tricky as I suspect there could be situations where you miss
> pages--but can't think of any. We could also always use the action index
> anytime we wanted to reindex an entire group of pages.
>
> I'm sure I could pull this together in 10 or 15 minutes. Will wait for
> your response to see if you have any thoughts or feedback?
>
> Cheers,
> Dan
>
> On Sun, Jan 10, 2016 at 2:34 AM Tiffany Grenier <[email protected]
> <javascript:>> wrote:
>
>> Hi,
>>
>> About the cron, I checked the BOLTFindex function, and it goes as follows:
>> if ($args['rule'] != '') $out = BOLTindex($args['rule']); // works
>> through a rule if no list
>> else {
>> if (is_number($args['cron'])) {
>> So I don't see how rule and cron could be used at the same time...
>>
>> In the end, I created several cron commands to page
>> action.index&rule=rule1, action.index&rule=rule2 etc. The current rate is
>> one every two minutes, each every twenty minutes (yes, I have quite a
>> number of indexes). It works fine, using the default indexBatch value and
>> updating the list of pages to be indexed everytime a change is made on the
>> website by members.
>> I also added a "buildIndexes" script that I included with all the rules
>> on a page I visit whenever I do massive changes from the FTP interface.
>> Then I just have to way that the whole indexing has been processed by my
>> index cron rules. If I know I will change main pages on a regular basis
>> (use page not relevant at the moment, but you never know), I can also
>> create a cron rule for this page. It spares me the necessity of login in
>> and clicking my way through all indexes in action.index without starting
>> the indexing process.
>> It is a simple copy from tha action.index indexing building form, and
>> goes as follows (although I don't think it could be of use for anyone else):
>> code.script.inde
>> authkey: indexing
>> source: 'site.index::{?build}'
>> reset: search='{=source} fmt=list count=false dir=pages'
>> search:
>> [if set {?build}]edit: content='{=search}' page='site.index.{?build}'[if]
>> That's it,
>> Cheers,
>> Tiffany/Maelite
>>
>>
>>
>>
>>
>> Le jeudi 7 janvier 2016 19:42:13 UTC+1, Dan a écrit :
>>
>>> Thank you!. I fixed the typo, and moved the messages you indicated to
>>> site.messages.
>>>
>>> As for #3, I believe you can use the cron parameter to do the
>>> equivalent. Batch is probably a better word, but I'm not sure the change is
>>> worth the disruption...
>>>
>>> [(index rule=site cron={?quantity})]
>>>
>>> Cheers,
>>> Dan
>>>
>>>
>>>
>>> On Tue, Jan 5, 2016 at 2:57 AM Tiffany Grenier <[email protected]>
>>> wrote:
>>>
>> Hi,
>>>>
>>>> 3 quick notes regarding indexing:
>>>>
>>>> - I just noticed a typo in BOLTindex:
>>>> if ($myrule == '') return "Index rule //$myrule// not found";
>>>> should be
>>>> if ($myrule == '') return "Index rule //$rule// not found";
>>>> Otherwise this warning gives no info at all...
>>>> - Also, I think the "Indexing is current" and "No pages to index"
>>>> warnings should be localized and moved to site.messages.
>>>> - And lastly, a small question: if I define 3 indexing rules and
>>>> set indexing to cron, nothing will happen, right? If so, could you
>>>> please
>>>> add a "batch" parameter to the indexing function such that I can put in
>>>> cron.index (which is visited through a cron job):
>>>> [(index rule=site batch={?quantity})]
>>>> [(index rule=game batch={?quantity})]
>>>> [(index rule=memberstats batch={?quantity})]
>>>> I know it will kind of do that automaticaly everytime a page is
>>>> modified in the website, but i just modified a whole bunch of pages
>>>> through
>>>> FTP and it's a real pain to re-index it all manually... I am heavily
>>>> using
>>>> content of index.game and index.memberstats and need them up-to-date
>>>> the
>>>> fastest way possible.
>>>>
>>>> Cheers,
>>>> Tiffany
>>>>
>>>> --
>>>> You received this message because you are subscribed to the Google
>>>> Groups "BoltWire" group.
>>>>
>>> To unsubscribe from this group and stop receiving emails from it, send
>>>> an email to [email protected].
>>>> To post to this group, send email to [email protected].
>>>
>>>
>>>> Visit this group at https://groups.google.com/group/boltwire.
>>>> For more options, visit https://groups.google.com/d/optout.
>>>>
>>> --
>> You received this message because you are subscribed to the Google Groups
>> "BoltWire" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to [email protected] <javascript:>.
>> To post to this group, send email to [email protected]
>> <javascript:>.
>> Visit this group at https://groups.google.com/group/boltwire.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
--
You received this message because you are subscribed to the Google Groups
"BoltWire" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to [email protected].
To post to this group, send email to [email protected].
Visit this group at https://groups.google.com/group/boltwire.
For more options, visit https://groups.google.com/d/optout.
