Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:
 |  needs_information
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by pili):

 This was discussed during the ux team meeting a couple of weeks ago
 (http://meetbot.debian.net/tor-meeting/2019/tor-
 meeting.2019-10-01-13.59.log.html):

 14:32:23  we have a bot that pulls new translations from
 transifex without any verification. so at some point we should do a
 verification. maybe as part of the pulling from transifex.
 14:32:33  but anyway we can discuss on the ticket if you have
 any ideas
 14:32:44  it should be the bot that does the checks
 14:32:49  not git with a hook
 14:33:45  something called from https://gitweb.torproject.org
 /translation-tools.git/tree/update_translations
 14:34:12  sounds like this is a discussion that we can maybe take to
 #tor-l10n after? :)
 14:34:29  I just wanted to make the right people aware of it
 14:34:33  sounds like this is not a gitadmin ticket
 14:35:04  let's move it to the right owner then and discuss it with
 them, does anyone know who that should be?
 14:35:55  whoever maintains that bot
 14:36:09  it is a bit big for me to code a l10n verification
 script in shellscript tbh
 14:36:26  maybe bolkm can help us here?
 14:36:33  i maintain the process of running the bot and make
 little changes, but a verification thing.. uff...
 14:37:27  which bot is this and where is the code?
 14:38:31  the code is at https://gitweb.torproject.org
 /translation-tools.git/tree/update_translations and is the bot that pushes
 the transifex updates to https://gitweb.torproject.org/translation.git
 14:39:16  lives at
 https://db.torproject.org/machines.cgi?host=majus
 14:40:30  and we think that the bot should check that special
 characters are escaped?
 14:40:45  I think python escapes the characters automatically
 nowadays
 14:40:56  there should be something called here:
 https://gitweb.torproject.org/translation-
 tools.git/tree/update_translations#n39
 14:41:10  before the commits to check if the diff is legit
 14:41:52  so basically we need a confident bash coder :)
 14:41:59  the escaping of certain characters on the android
 files is not according to standard, and when adding ' in the translations,
 the build will break. transifex didnt wanted to create a new parsing
 because it is against the escaping rules of dtd files

 We need someone to help emmapeel with this verification script.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:
 |  needs_information
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by emmapeel):

 The people of transifex says they cannot make a different escaping than
 the one on the dtd specs, so we should create a githook for the files at
 https://gitweb.torproject.org/translation.git/tree/en-
 US/torbrowser_strings.dtd?h=tba-torbrowserstringsdtd to be tested with the
 compare-locales mentioned by geko.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:
 |  needs_information
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gk):

 * cc: pili (added)
 * status:  new => needs_information


Comment:

 emmapeel: what do we need to get this bug resolved/unblocked? Luckily, I
 ran a test build before starting the build for 8.5.4 in order to figure
 whether we could use the latest translations for mobile. But it turns out
 we can't as 21677307d87d3216eede51a6ad36bf26e7937667 is the last good
 commit before `'`'s get added unescaped to `ca`. That alone is unfortunate
 but it's even more so that this blocks other unaffected locales from
 getting the latest translations.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by eighthave):

 I think it is important to have a CI test that ensures that the
 translations aren't breaking anything.  For F-Droid, I wrote an emulator
 test that loads all strings in all locales.  It has found many crasher
 bugs.  Feel free to take it under the license you need, it will need some
 tweaking to work in a different app (it is GPLv3+ in F-Droid):
 
https://gitlab.com/fdroid/fdroidclient/blob/master/app/src/androidTest/java/org/fdroid/fdroid/LocalizationTest.java

 IMHO, the translation tool should properly escape the translator's input
 whenever possible.  I believe Weblate does a pretty good job of this.
 Weblate checks are also quite useful for this kind of thing

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by emmapeel):

 regarding how other projectss do it, they do checks by themselves, see for
 example in Tails:

 https://trac.torproject.org/projects/tor/ticket/30054

 transifex does not provide any kind of checks (Weblate does check some
 stuff tho, tsck tsck).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by emmapeel):

 well there are other types of documents that escape characters, like PO
 files I believe. It would be a matter of creating a different type of
 document. I need to check with erinm on that.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-

Comment (by gk):

 Okay, I talked to Pike who helped me a lot (thanks for that). Mozilla does
 not have an automatic way to escape strings properly. However, they have a
 tool called `compare-locales` that is used to make sure to check for
 issues before a build breaks. See: https://developer.mozilla.org/en-
 US/docs/Mozilla/Projects/compare-locales for usage info. Moreover, I got
 told that we can run their tool by creating an `l10n.toml` to run against
 the l10n repo, with the `android-dtd` tests as defined in
 https://searchfox.org/mozilla-
 
central/rev/d143f8ce30d1bcfee7a1227c27bf876a85f8cede/mobile/android/locales/l10n.toml#131-136.

 Thus, I guess instead of using their l10n repo we could point to a
 location of our choice to check for Android issues.

 emmapeel: Is that enough to get you started?

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
-+-
 Reporter:  gk   |  Owner:
 |  emmapeel
 Type:  defect   | Status:  new
 Priority:  Medium   |  Milestone:
Component:  Community/Translations   |Version:
 Severity:  Normal   | Resolution:
 Keywords:  TorBrowserTeam201904,|  Actual Points:
  GeorgKoppen201904  |
Parent ID:   | Points:
 Reviewer:   |Sponsor:
-+-
Changes (by gk):

 * keywords:   => TorBrowserTeam201904, GeorgKoppen201904


--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by gk):

 Replying to [comment:6 emmapeel]:
 > Yes, I'd consider this ticket done when transifex can escape the chars
 themselves and present translators with a normal string, doing the
 escaping on the background.
 >
 > This will prevent future problems, because right now we have to review
 this 'by hand'.

 Or maybe there is some other automated step we can do, keeping Transifex
 out of the loop. I'll ask around what Mozilla is doing.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by emmapeel):

 Yes, I'd consider this ticket done when transifex can escape the chars
 themselves and present translators with a normal string, doing the
 escaping on the background.

 This will prevent future problems, because right now we have to review
 this 'by hand'.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  | Points:
 Reviewer:  |Sponsor:
+--
Changes (by gk):

 * parent:  #30016 =>


Comment:

 Replying to [comment:4 emmapeel]:
 > I think i cleaned up a bit, let me know if it is still broken.
 >
 > i also added a note for the translators, but it does not force the
 escaping.

 Thanks, this works for me. I'll unparent the ticket. Not sure if you want
 to keep it open for having a less ad hoc way of dealing with this problem
 implemented on the server-/repository-side. (Assuming that's the place
 where we want to make sure the escaping is properly done).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  #30016  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by emmapeel):

 I think i cleaned up a bit, let me know if it is still broken.

 i also added a note for the translators, but it does not force the
 escaping.

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  #30016  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by emmapeel):

 formatting rules: ​https://developer.android.com/guide/topics/resources
 /string-resource#FormattingAndStyling

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  #30016  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by emmapeel):

 I will talk with erinm and find out if that can be a feature request for
 Transifex

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

Re: [tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  | Resolution:
 Keywords:  |  Actual Points:
Parent ID:  #30016  | Points:
 Reviewer:  |Sponsor:
+--

Comment (by gk):

 I agree we should not require translators to know which entities to escape
 and which not. However, importing the strings as they currently are breaks
 the build. Could we have a script running over the translated strings
 adding necessary escaping if that is missed before the final strings are
 available in the translations repo?

 I wonder how other projects are dealing with that problem as I guess we
 are hardly the only ones getting hit by it...

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs

[tor-bugs] #30054 [Community/Translations]: Special characters are not escaped in translations and break the build

[Tor Bug Tracker & Wiki]

#30054: Special characters are not escaped in translations and break the build
+--
 Reporter:  gk  |  Owner:  emmapeel
 Type:  defect  | Status:  new
 Priority:  Medium  |  Milestone:
Component:  Community/Translations  |Version:
 Severity:  Normal  |   Keywords:
Actual Points:  |  Parent ID:  #30016
   Points:  |   Reviewer:
  Sponsor:  |
+--
 As said in comment:14:ticket:27125 there are special characters (like the
 `'`) that need to get escaped in strings, otherwise the build breaks.
 While we did that in `en-US` that does not happen in translations (yet).

--
Ticket URL: 
Tor Bug Tracker & Wiki 
The Tor Project: anonymity online
___
tor-bugs mailing list
tor-bugs@lists.torproject.org
https://lists.torproject.org/cgi-bin/mailman/listinfo/tor-bugs