Hi Erwan
That's very helpful, thank you.
It looks as though most of the hits are actually Tobias's experiments with
reworking is[missing].
> The system/shadow tiddlers were excluded because originally I wasn't
renaming the tiddlers, thus causing unwanted interactions on the search
wiki. I guess that with the renaming (simple prefix with $:/<wiki id>/)
there is little risk of interaction, but I'm not sure to know about all the
possible kinds of "interactions" in TW. Anyway I can do some tests with
them, but if you or any other TW expert could tell me which aspects to be
careful about when merging all the tiddlers, that could be helpful. I know
at least that I can't include plugin tiddlers, and currently I also exclude
theme and javascript tiddlers to be safe (?).
I've been dealing with the same problem for work I've been doing on the
plugin library. The code I've got looks like this:
$tw.wiki.addTiddler(new $tw.Tiddler($tw.wiki.getCreationFields(),tiddler,{
title: event.data.cookies.infoTitlePrefix + event.data.cookies.url + "/" +
tiddler.title,
"original-title": tiddler.title,
text: "",
type: "text/vnd.tiddlywiki",
"original-type": tiddler.type,
"plugin-type": undefined,
"original-plugin-type": tiddler["plugin-type"],
"module-type": undefined,
"original-module-type": tiddler["module-type"],
tags: ["$:/tags/RemoteAssetInfo"],
"original-tags": $tw.utils.stringifyList(tiddler.tags || []),
"server-url": event.data.cookies.url
},$tw.wiki.getModificationFields()));
As you can see, it's mangling the incoming title by applying a prefix
(which I believe you already do), and also neutralising the plugin-type and
module-type fields, and moving the tags to another field. The last is
probably overkill - the key thing is to remove $:/tags/* tags.
Best wishes
Jeremy
On Sun, Feb 8, 2015 at 1:18 AM, Erwan <erwan.m...@gmail.com> wrote:
>
> Hi Jeremy,
>
> The system/shadow tiddlers were excluded because originally I wasn't
> renaming the tiddlers, thus causing unwanted interactions on the search
> wiki. I guess that with the renaming (simple prefix with $:/<wiki id>/)
> there is little risk of interaction, but I'm not sure to know about all the
> possible kinds of "interactions" in TW. Anyway I can do some tests with
> them, but if you or any other TW expert could tell me which aspects to be
> careful about when merging all the tiddlers, that could be helpful. I know
> at least that I can't include plugin tiddlers, and currently I also exclude
> theme and javascript tiddlers to be safe (?).
>
> Indexing all public wikis seems more problematic to me, as some authors
> might not want that: additionally to my emails on the TW list I have sent
> an individual email (and later a reminder) to every known author in my
> list, so at this point I have to assume that those who did not answer are
> not interested. Since my system requires storing their tiddlers in the
> search wiki, it wouldn't be appropriate to do it without their consent. It
> would be different if the content was not made public (even hidden as
> system tiddlers), but I don't see how to do differently since it is the TW
> search system which does all the work.
>
> About the particular case of is[missing], I did a manual grep search on my
> local machine among the 29 wikis in my list (tagged CommunityWikis,
> including the non-indexed wikis; 5715 tiddlers in total) and found 25
> occurrences (more precisely lines containing at least one occurrence); see
> the detailed results attached. By comparison I found:
> - 135 is[current]
> - 19 is[image]
> - 5 is[orphan]
> - 67 is[shadow]
> - 389 is[system]
> - 44 is[tiddler]
> - 7 is[tag]
>
> I was even imagining that this could be automatized, actually: some kind
> of asynchronous search triggered by some particular tag/tiddler in an
> indexed wiki, for which you would see the results the day after in a
> special tiddler of the search wiki... But that seems to me like a terribly
> complicated system for a very specific use case! (and that's not even
> addressing the potentially excessive workload on my machine)
>
> Regards
> Erwan
>
>
>
> On 07/02/15 13:35, Jeremy Ruston wrote:
>
> Hi Erwan
>
> Well done for your work on this, it's good to see this sort of
> experimentation.
>
> Something that's been cropping up over in GitHub discussions is that it
> might be useful to be able to search across all community wikis to see if
> certain constructions are being used heavily in the wild. For example,
> we've realised that the `[is[missing]]` filter has the wrong semantics,
> which we'd like to fix but it would be useful to have some idea if it is
> widely used it in its present form.
>
> So I guess I'm asking for the ability to search system and shadow
> tiddlers, and perhaps to broaden the search to all publicly accessible TW5
> wikis. If only there was a way to get Google/DuckDuckGo to return a list
> all TW5 documents that it has crawled...
>
> Best wishes
>
> Jeremy.
>
>
>
>
> On Sat, Feb 7, 2015 at 1:04 PM, Tobias Beer <beertob...@gmail.com> wrote:
>
>> Hi Erwan,
>>
>> Will be curious how your aggregator will handle all the redirects and
>> renaming I'm pushing though right now. ^^
>>
>> Best wishes, Tobias.
>> --
>> You received this message because you are subscribed to the Google Groups
>> "TiddlyWiki" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to tiddlywiki+unsubscr...@googlegroups.com.
>> To post to this group, send email to tiddlywiki@googlegroups.com.
>> Visit this group at http://groups.google.com/group/tiddlywiki.
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>
>
> --
> Jeremy Ruston
> mailto:jeremy.rus...@gmail.com
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To post to this group, send email to tiddlywiki@googlegroups.com.
> Visit this group at http://groups.google.com/group/tiddlywiki.
> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To post to this group, send email to tiddlywiki@googlegroups.com.
> Visit this group at http://groups.google.com/group/tiddlywiki.
> For more options, visit https://groups.google.com/d/optout.
>
--
Jeremy Ruston
mailto:jeremy.rus...@gmail.com
--
You received this message because you are subscribed to the Google Groups
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.
