[tw] Re: [TW5] Community Search

[Tobias Beer]

Thanks for all the work, Erwan.

I certainly appreciate that you've come to automate that indexing mechanism 
via the fields

   - *follow=yes*
   - *url=http://foo.bar.com*

at reference tiddlers such as those listed here...

   - Tobias Beer @ tb5 
   - or here respectively: $:/follow @ tb5 
   

Best wishes, Tobias.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: [TW5] Community Search

[Felix Küppers]

Great job. Really useful.

On Sunday, February 1, 2015 at 8:58:26 PM UTC+1, Erwan wrote:
>
>
> Hi everyone, 
>
> The Community Search wiki keeps growing, slowly but surely, with now 
> 3383 indexed tiddlers from 36 different wikis (half of them thanks to 
> Tobias!): 
> https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html 
>
> It would be nice to make it cover a larger set of wikis, as this is what 
> a search engine is for, obviously! So if you are a wiki author and are 
> fine with your content being indexed, please let me know! I can't do it 
> without your agreement. 
>
> Recently I added some documentation, and made some improvements intended 
> to give wiki authors more control over what is indexed from their wiki. 
> So if you have concerns about that, please have a look at 
>
> https://rawgit.com/erwanm/tw-aggregator/master/tw-community-search.html#AuthorDocumentation
>  
> And of course don't hesitate to tell me about any bug you find or 
> feature you would like. 
>
> Regards, 
> Erwan 
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: [TW5] Community Search

[Danielo Rodríguez]

I have to agree with the congratulations.

The new features makes much more powerful this awesome tool.
Thank you.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: [TW5] Community Search

[Jed Carty]

Adding a new wiki worked, but it only used the first word of the title with 
spaces.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

[tw] Re: [TW5] Community Search

[Tobias Beer]

Hi Erwan,

Will be curious how your aggregator will handle all the redirects and 
renaming I'm pushing though right now. ^^

Best wishes, Tobias. 

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Erwan]

Happy that you like it, thank you!

btw these new features (especially the whitelist one) haven't really 
been thoroughly tested, so if you use them please check that it works as 
expected for your wiki. and of course let me know in case it doesn't!


Erwan

On 04/02/15 16:33, Danielo Rodríguez wrote:

I have to agree with the congratulations.

The new features makes much more powerful this awesome tool.
Thank you.
--
You received this message because you are subscribed to the Google 
Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to tiddlywiki+unsubscr...@googlegroups.com 
.
To post to this group, send email to tiddlywiki@googlegroups.com 
.

Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Felix Küppers]

Hi Erwan,

would it be possible to introduce a tag like e.g. "#news" or "#release 
notes" that can be used by wiki-owners to force some tiddlers to be listed 
chronologically as a separate category in your search wiki? Mat already 
suggested a similar thing but then it would be possible for plugin 
developers to tag release note tiddlers with #news and people can see what 
is happening - this is actually pretty close to a RSS-feed.

-Felix

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Erwan]

Hi Felix,

Yes, I'm thinking about something like that as well. Originally I was 
going to simply use the date of the tiddler to automatically generate a 
"news event".


I think the difficulty lies mostly in how this features fits with 
different cases:


 * automatic "news" when a plugin is updated based on the date
 * some authors might not want to use it although their plugins are
   useful, so it's important that users don't assume that this kind of
   news will show up for all plugins
 * people could forget to remove their "news" tag when it's not news
   anymore, so the system problably has to do some cleanup based on the
   date
 * possibly storing the news to keep track of the history?

Also it's probably better to propose a special tiddler where the author 
can write their own filter instead of a predefined tag, since it's more 
generic and gives more freedom to the author.


I'm trying to clarify for myself how to make all this work nicely 
together, anybody is welcome to give their suggestions!



Erwan


On 06/02/15 08:17, Felix Küppers wrote:

Hi Erwan,

would it be possible to introduce a tag like e.g. "#news" or "#release 
notes" that can be used by wiki-owners to force some tiddlers to be 
listed chronologically as a separate category in your search wiki? Mat 
already suggested a similar thing but then it would be possible for 
plugin developers to tag release note tiddlers with #news and people 
can see what is happening - this is actually pretty close to a RSS-feed.


-Felix
--
You received this message because you are subscribed to the Google 
Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to tiddlywiki+unsubscr...@googlegroups.com 
.
To post to this group, send email to tiddlywiki@googlegroups.com 
.

Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Tobias Beer]

>
> Also it's probably better to propose a special tiddler where the author 
> can write their own filter instead of a predefined tag, since it's more 
> generic and gives more freedom to the author.
>
> I'm trying to clarify for myself how to make all this work nicely 
> together, anybody is welcome to give their suggestions!
>
Not sure if that works, but if I can specify a filter that works for me and 
works for you (+ the site constraint), then that would be it, I believe... 
e.g. "news is everything tagged foo not older than x".

Best wishes, Tobias. 

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Erwan]

Hi Jed
This is fixed now.

Erwan

On 06/02/15 05:41, Jed Carty wrote:
Adding a new wiki worked, but it only used the first word of the title 
with spaces.

--
You received this message because you are subscribed to the Google 
Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to tiddlywiki+unsubscr...@googlegroups.com 
.
To post to this group, send email to tiddlywiki@googlegroups.com 
.

Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Jeremy Ruston]

Hi Erwan

Well done for your work on this, it's good to see this sort of
experimentation.

Something that's been cropping up over in GitHub discussions is that it
might be useful to be able to search across all community wikis to see if
certain constructions are being used heavily in the wild. For example,
we've realised that the `[is[missing]]` filter has the wrong semantics,
which we'd like to fix but it would be useful to have some idea if it is
widely used it in its present form.

So I guess I'm asking for the ability to search system and shadow tiddlers,
and perhaps to broaden the search to all publicly accessible TW5 wikis. If
only there was a way to get Google/DuckDuckGo to return a list all TW5
documents that it has crawled...

Best wishes

Jeremy.




On Sat, Feb 7, 2015 at 1:04 PM, Tobias Beer  wrote:

> Hi Erwan,
>
> Will be curious how your aggregator will handle all the redirects and
> renaming I'm pushing though right now. ^^
>
> Best wishes, Tobias.
>
> --
> You received this message because you are subscribed to the Google Groups
> "TiddlyWiki" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to tiddlywiki+unsubscr...@googlegroups.com.
> To post to this group, send email to tiddlywiki@googlegroups.com.
> Visit this group at http://groups.google.com/group/tiddlywiki.
> For more options, visit https://groups.google.com/d/optout.
>



-- 
Jeremy Ruston
mailto:jeremy.rus...@gmail.com

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Erwan]

Hi Jeremy,

The system/shadow tiddlers were excluded because originally I wasn't 
renaming the tiddlers, thus causing unwanted interactions on the search 
wiki. I guess that with the renaming (simple prefix with $://) 
there is little risk of interaction, but I'm not sure to know about all 
the possible kinds of "interactions" in TW. Anyway I can do some tests 
with them, but if you or any other TW expert could tell me which aspects 
to be careful about when merging all the tiddlers, that could be 
helpful. I know at least that I can't include plugin tiddlers, and 
currently I also exclude theme and javascript tiddlers to be safe (?).


Indexing all public wikis seems more problematic to me, as some authors 
might not want that: additionally to my emails on the TW list I have 
sent an individual email (and later a reminder) to every known author in 
my list, so at this point I have to assume that those who did not answer 
are not interested. Since my system requires storing their tiddlers in 
the search wiki, it wouldn't be appropriate to do it without their 
consent. It would be different if the content was not made public (even 
hidden as system tiddlers), but I don't see how to do differently since 
it is the TW search system which does all the work.


About the particular case of is[missing], I did a manual grep search on 
my local machine among the 29 wikis in my list (tagged CommunityWikis, 
including the non-indexed wikis; 5715 tiddlers in total) and found 25 
occurrences (more precisely lines containing at least one occurrence); 
see the detailed results attached. By comparison I found:

- 135 is[current]
- 19 is[image]
- 5 is[orphan]
- 67 is[shadow]
- 389 is[system]
- 44 is[tiddler]
- 7 is[tag]

I was even imagining that this could be automatized, actually: some kind 
of asynchronous search triggered by some particular tag/tiddler in an 
indexed wiki, for which you would see the results the day after in a 
special tiddler of the search wiki... But that seems to me like a 
terribly complicated system for a very specific use case! (and that's 
not even addressing the potentially excessive workload on my machine)


Regards
Erwan


On 07/02/15 13:35, Jeremy Ruston wrote:

Hi Erwan

Well done for your work on this, it's good to see this sort of 
experimentation.


Something that's been cropping up over in GitHub discussions is that 
it might be useful to be able to search across all community wikis to 
see if certain constructions are being used heavily in the wild. For 
example, we've realised that the `[is[missing]]` filter has the wrong 
semantics, which we'd like to fix but it would be useful to have some 
idea if it is widely used it in its present form.


So I guess I'm asking for the ability to search system and shadow 
tiddlers, and perhaps to broaden the search to all publicly accessible 
TW5 wikis. If only there was a way to get Google/DuckDuckGo to return 
a list all TW5 documents that it has crawled...


Best wishes

Jeremy.




On Sat, Feb 7, 2015 at 1:04 PM, Tobias Beer > wrote:


Hi Erwan,

Will be curious how your aggregator will handle all the redirects
and renaming I'm pushing though right now. ^^

Best wishes, Tobias.
-- 
You received this message because you are subscribed to the Google

Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it,
send an email to tiddlywiki+unsubscr...@googlegroups.com
.
To post to this group, send email to tiddlywiki@googlegroups.com
.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.




--
Jeremy Ruston
mailto:jeremy.rus...@gmail.com 
--
You received this message because you are subscribed to the Google 
Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to tiddlywiki+unsubscr...@googlegroups.com 
.
To post to this group, send email to tiddlywiki@googlegroups.com 
.

Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.
ae-railroad/tiddlers/$__core.tid:"text": "/*\\\r\ntitle: 
$:/core/modules/filters/is/missing.js\r\ntype: 
application/javascript\r\nmodule-type: isfilteroperator\r\n\r\nFilter function 
for [is[missing]]

Re: [tw] Re: [TW5] Community Search

[Erwan]

Hi Tobias,

To be honest I wouldn't be surprised at all if you find bugs ;)
Let me know what happens.

Regards
Erwan

On 07/02/15 13:04, Tobias Beer wrote:

Hi Erwan,

Will be curious how your aggregator will handle all the redirects and 
renaming I'm pushing though right now. ^^


Best wishes, Tobias.
--
You received this message because you are subscribed to the Google 
Groups "TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send 
an email to tiddlywiki+unsubscr...@googlegroups.com 
.
To post to this group, send email to tiddlywiki@googlegroups.com 
.

Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.


--
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Tobias Beer]

>
> To be honest I wouldn't be surprised at all if you find bugs ;) 
> Let me know what happens.
>

It works pretty well, actually showing me my own bugs and dead links ;-)

Best wishes, Tobias.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.

Re: [tw] Re: [TW5] Community Search

[Jeremy Ruston]

Hi Erwan

That's very helpful, thank you.

It looks as though most of the hits are actually Tobias's experiments with
reworking is[missing].

> The system/shadow tiddlers were excluded because originally I wasn't
renaming the tiddlers, thus causing unwanted interactions on the search
wiki. I guess that with the renaming (simple prefix with $://)
there is little risk of interaction, but I'm not sure to know about all the
possible kinds of "interactions" in TW. Anyway I can do some tests with
them, but if you or any other TW expert could tell me which aspects to be
careful about when merging all the tiddlers, that could be helpful. I know
at least that I can't include plugin tiddlers, and currently I also exclude
theme and javascript tiddlers to be safe (?).

I've been dealing with the same problem for work I've been doing on the
plugin library. The code I've got looks like this:


$tw.wiki.addTiddler(new $tw.Tiddler($tw.wiki.getCreationFields(),tiddler,{
title: event.data.cookies.infoTitlePrefix + event.data.cookies.url + "/" +
tiddler.title,
"original-title": tiddler.title,
text: "",
type: "text/vnd.tiddlywiki",
"original-type": tiddler.type,
"plugin-type": undefined,
"original-plugin-type": tiddler["plugin-type"],
"module-type": undefined,
"original-module-type": tiddler["module-type"],
tags: ["$:/tags/RemoteAssetInfo"],
"original-tags": $tw.utils.stringifyList(tiddler.tags || []),
"server-url": event.data.cookies.url
},$tw.wiki.getModificationFields()));

As you can see, it's mangling the incoming title by applying a prefix
(which I believe you already do), and also neutralising the plugin-type and
module-type fields, and moving the tags to another field. The last is
probably overkill - the key thing is to remove $:/tags/* tags.

Best wishes

Jeremy


On Sun, Feb 8, 2015 at 1:18 AM, Erwan  wrote:

>
> Hi Jeremy,
>
> The system/shadow tiddlers were excluded because originally I wasn't
> renaming the tiddlers, thus causing unwanted interactions on the search
> wiki. I guess that with the renaming (simple prefix with $://)
> there is little risk of interaction, but I'm not sure to know about all the
> possible kinds of "interactions" in TW. Anyway I can do some tests with
> them, but if you or any other TW expert could tell me which aspects to be
> careful about when merging all the tiddlers, that could be helpful. I know
> at least that I can't include plugin tiddlers, and currently I also exclude
> theme and javascript tiddlers to be safe (?).
>
> Indexing all public wikis seems more problematic to me, as some authors
> might not want that: additionally to my emails on the TW list I have sent
> an individual email (and later a reminder) to every known author in my
> list, so at this point I have to assume that those who did not answer are
> not interested. Since my system requires storing their tiddlers in the
> search wiki, it wouldn't be appropriate to do it without their consent. It
> would be different if the content was not made public (even hidden as
> system tiddlers), but I don't see how to do differently since it is the TW
> search system which does all the work.
>
> About the particular case of is[missing], I did a manual grep search on my
> local machine among the 29 wikis in my list (tagged CommunityWikis,
> including the non-indexed wikis; 5715 tiddlers in total) and found 25
> occurrences (more precisely lines containing at least one occurrence); see
> the detailed results attached. By comparison I found:
> - 135 is[current]
> - 19 is[image]
> - 5 is[orphan]
> - 67 is[shadow]
> - 389 is[system]
> - 44 is[tiddler]
> - 7 is[tag]
>
> I was even imagining that this could be automatized, actually: some kind
> of asynchronous search triggered by some particular tag/tiddler in an
> indexed wiki, for which you would see the results the day after in a
> special tiddler of the search wiki... But that seems to me like a terribly
> complicated system for a very specific use case! (and that's not even
> addressing the potentially excessive workload on my machine)
>
> Regards
> Erwan
>
>
>
> On 07/02/15 13:35, Jeremy Ruston wrote:
>
> Hi Erwan
>
>  Well done for your work on this, it's good to see this sort of
> experimentation.
>
>  Something that's been cropping up over in GitHub discussions is that it
> might be useful to be able to search across all community wikis to see if
> certain constructions are being used heavily in the wild. For example,
> we've realised that the `[is[missing]]` filter has the wrong semantics,
> which we'd like to fix but it would be useful to have some idea if it is
> widely used it in its present form.
>
>  So I guess I'm asking for the ability to search system and shadow
> tiddlers, and perhaps to broaden the search to all publicly accessible TW5
> wikis. If only there was a way to get Google/DuckDuckGo to return a list
> all TW5 documents that it has crawled...
>
>  Best wishes
>
>  Jeremy.
>
>
>
>
> On Sat, Feb 7, 2015 at 1:04 PM, Tobias Beer  wrote:
>
>> Hi Erwan,
>>
>>

[tw] Re: [TW5] Community search to aggregate specific #tiddlers, @Erwan ?

[Felix Küppers]

wow mat, nice idea.

-- 
You received this message because you are subscribed to the Google Groups 
"TiddlyWiki" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to tiddlywiki+unsubscr...@googlegroups.com.
To post to this group, send email to tiddlywiki@googlegroups.com.
Visit this group at http://groups.google.com/group/tiddlywiki.
For more options, visit https://groups.google.com/d/optout.