[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Mvolz]

Mvolz added a comment.


  In T206613#5311274 , 
@Smalyshev wrote:
  
  > I've reindexed testwikidata last week, the patch should already be there.
  
  Oh great, looks like it works!
  
  
https://test.wikidata.org/w/api.php?action=query=search=haswbstatement:P168=10.1371/journal.pcbi.1002947
  
https://test.wikidata.org/w/api.php?action=query=search=haswbstatement:P168=10.1371/JOURNAL.pcbi.1002947

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mvolz
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, rosalieper, FloNight, 
Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Smalyshev]

Smalyshev added a comment.


  I've reindexed testwikidata last week, the patch should already be there.

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, rosalieper, FloNight, 
Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Mvolz]

Mvolz added a comment.


  We're going to try to deploy this first on test.wikidata, can this be 
reindexed as well? Or do they both get re-indexed at the same time?

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mvolz
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, rosalieper, FloNight, 
Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Mvolz]

Mvolz added a comment.


  No worries; I wasn't aware the train was broken and wondered if the
  reindexing had to be triggered manually or something, thanks for the update!

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mvolz
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, rosalieper, FloNight, 
Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Mvolz]

Mvolz added a comment.


  In T206613#5235476 , 
@Smalyshev wrote:
  
  > Now after it's deployed reindex will be needed.
  
  
  Do you know when this will be?

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mvolz
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, 
QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, rosalieper, FloNight, 
Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Mvolz]

Mvolz added a comment.


  Also I should point out that using an indexed search is *much* better 
performance wise!
  
  In T206613#5235476 , 
@Smalyshev wrote:
  
  > Now after it's deployed reindex will be needed.
  
  
  Yay! Thank you!

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Mvolz
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, Premeditated, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, 
rosalieper, FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[gerritbot]

gerritbot added a comment.


  Change 514198 **merged** by jenkins-bot:
  [mediawiki/extensions/WikibaseCirrusSearch@master] Add case-insenstitive 
subfield for statement field
  
  https://gerrit.wikimedia.org/r/514198

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, ET4Eva, CucyNoiD, 
Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, 
LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, 
FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[dcausse]

dcausse added a comment.


  @Smalyshev I totally agree, I was suggesting a UX where a first attempt 
search would try to match using the haswbstatement keyword (switched to case 
insensitive) and then a second try could be made using the fulltext mode if the 
first attempt is unsuccessful.

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, ET4Eva, CucyNoiD, 
Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, 
LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, 
FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Smalyshev]

Smalyshev added a comment.


  @dcausse agree but I think it still makes sense to make it case insensitive, 
since most data there are either case-insensitive or at least case there is 
rarely used to distinguish between things (i.e. having `pcbi.100123` in one 
item and `PCBI.100123` in another is not likely). So insensitive makes more 
sense to me, I guess.

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: Smalyshev
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, ET4Eva, CucyNoiD, 
Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, 
LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, 
FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[dcausse]

dcausse added a comment.


  we should also note we index this data in the main filter field which means 
that for searches that are unlikely to be ambiguous (IDs and such) one could 
simply search for 10.1371/journal.pcbi.1002947 
.
 Benefit is that it's tolerant to small variation in punctuation but also 
accept partial searches like:
  journal.pcbi.1002947 

 or even with small variations: journal pcbi 1002947 
.
  
  So instead of giving up with no results this kind of searches could be tried 
if a human is behind to select/accept/validate a result.

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, ET4Eva, CucyNoiD, 
Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, 
LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, 
FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[dcausse]

dcausse added a comment.


  @Smalyshev switching the main field for statements to `lowercase_keyword` 
won't break anything, it's like a new field it'll be taken into account just 
after the next reindex. I would advise against a new field here, the 
cardinality would nearly double.

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: dcausse
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
E.S.A-Sheild, darthmon_wmde, Premeditated, joker88john, ET4Eva, CucyNoiD, 
Nandana, NebulousIris, Gaboe420, Versusxo, Majesticalreaper22, Giuliamocci, 
Adrian1985, Cpaulf30, Lahi, Gq86, Baloch007, Darkminds3113, Bsandipan, Lordiis, 
GoranSMilovanovic, Adik2382, Th3d3v1ls, Ramalepe, Liugev6, QZanden, EBjune, 
LawExplorer, WSH1906, Avner, Lewizho99, Maathavan, Gehel, _jensen, rosalieper, 
FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[gerritbot]

gerritbot added a comment.


  Change 514198 had a related patch set uploaded (by Smalyshev; owner: 
Smalyshev):
  [mediawiki/extensions/WikibaseCirrusSearch@master] Add case-insenstitive 
subfield for statement field
  
  https://gerrit.wikimedia.org/r/514198

TASK DETAIL
  https://phabricator.wikimedia.org/T206613

EMAIL PREFERENCES
  https://phabricator.wikimedia.org/settings/panel/emailpreferences/

To: gerritbot
Cc: EBernhardson, WMDE-leszek, Multichill, Aklapper, Lydia_Pintscher, aude, 
debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, 
darthmon_wmde, Premeditated, ET4Eva, Nandana, Lahi, Gq86, Darkminds3113, 
GoranSMilovanovic, QZanden, EBjune, LawExplorer, Avner, Gehel, _jensen, 
rosalieper, FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb
___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Mvolz]

Mvolz added a comment.

In T206613#4664719, @Smalyshev wrote:
I don't think removing case sensitivity would be a lot of manual work, but it will require a reindex to change the index. I'm not sure why we decided on it being case-sensitive, I'll try to figure it out and if there's no reason we can change it. Note that this will apply for all fields, so if there are properties where case does matter it may get things wrong.


As long as it's possible to get the original case from the API then you can remove false positives in the case sensitive case by doing another call for each result and then comparing equality. Whereas if there are no results than it's very hard to get a result as you have to try every case permutation - all lower, all upper, camel case, sentence case, title case or completely random :). That said, string values are now available from the general search which does work, so maybe there isn't a need? i.e. https://www.wikidata.org/w/api.php?action=""> works.TASK DETAILhttps://phabricator.wikimedia.org/T206613EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MvolzCc: Multichill, Aklapper, Lydia_Pintscher, aude, debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Avner, Gehel, FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs

[Wikidata-bugs] [Maniphest] [Commented On] T206613: Search of wikidata string property values using haswbstatement is case sensitive

[Multichill]

Multichill added a comment.
@Smalyshev  what do you think? I haven't run into this myself. My feeling is that case insensitive is probably better, but would that require a lot of work?TASK DETAILhttps://phabricator.wikimedia.org/T206613EMAIL PREFERENCEShttps://phabricator.wikimedia.org/settings/panel/emailpreferences/To: MultichillCc: Multichill, Aklapper, Lydia_Pintscher, aude, debt, Smalyshev, Lea_Lacroix_WMDE, ArthurPSmith, Esc3300, dcausse, Mvolz, Nandana, Lahi, Gq86, Darkminds3113, GoranSMilovanovic, QZanden, EBjune, LawExplorer, Avner, Gehel, FloNight, Wikidata-bugs, jayvdb, Mbch331, jeremyb___
Wikidata-bugs mailing list
Wikidata-bugs@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikidata-bugs