Firefox Nightly Fluent core moved to Rust

[Zibi Braniecki (Gandalf)]

Hi all,

This is a PSA about the change that landed last night in bug 1560038 [0].

We migrated the core of Fluent Localization System in Gecko from JavaScript
[1] to Rust [2].

The expected benefits are:
 - Lays groundwork for shared memory for Fluent (Fission) (next steps in
bug 1613705 [3])
 - Removal of JS/XPCOM engine bootstrap from startup path for Fluent/DOM
 - Slight performance improvement with hopes for greater wins in bug
1613705 [3].
 - Full AST parser allows for better diagnostics and error reporting

This is a change that should result in no observable difference in behavior
but since it is a significant change in the core of the system I'd like to
ask all users of Nightly to watch for any unexpected behavior related to
localization and report it on Bugzilla (Core::Internationalization) or on
Matrix (#l10n-dev:mozilla.org).

Thank you!
zb.

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1560038
[1] https://github.com/projectfluent/fluent.js/tree/master/fluent-bundle
[2] https://crates.io/crates/fluent
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1613705
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Semantic Locale and Language management in Gecko

[Zibi Braniecki (Gandalf)]

Hi all!

If you ever need to work with language/locale identifiers, we now have a
clean API to use for all Rust, C++ and JavaScript.

The most common scenario in which you may encounter the need is when you
want to verify that, say, Firefox UI locale is English, or that the
regional variant is "US".

Historically you'd write this code as:

```js
const locale = Services.locale.appLocaleAsBCP47;
if (locale.substr(0, 2) == "en") {
  // it's English!
}
```

Unicode Language Identifiers are a bit more complicated, and such DIY code
is likely to break in all sorts of creative ways.
You can read the complete spec [0] and implement some of that parsing
yourself, but there's now an easier way - Locale API!

=== How to use ==

The API supports parsing, canonicalizing, validating, serializing,
modifying etc, but here I'm just going to show how testing for a subtag
works:

1) JavaScript

Andre Bargul implemented `Intl.Locale` API, part of the upcoming edition of
EcmaScript 2020! This API allows you to parse the language identifier
string and operate on it semantically:

```js
const locale = new Locale(Services.locale.appLocaleAsBCP47);
if (locale.language == "en") {
  // it's English!
}
```

You can find full docs here:
https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Locale

2) C++

```cpp
nsAutoCString locale
LocaleService::GetInstance()->GetAppLocaleAsBCP47(locale);
Locale loc = Locale(locale);
if (loc.GetLanguage().Equals("en")) {
  // it's English!
}
```

You can find full docs here:
https://searchfox.org/mozilla-central/source/intl/locale/MozLocale.h

3) Rust

```rust
// No access to LocaleService yet!

let locale = LanguageIdentifier::try_from("en-US").unwrap_or_default();
if (locale.language() == "en") {
  // it's English!
}
```

You can find full docs here: https://docs.rs/unic-langid/0.8.0/unic_langid/

== Why it's important ==

In order to improve our handling of Firefox localization, multilingualism
and internationalization, we need to be able to operate on the whole range
of possible language and locale identifiers.

With an API for operating on them centralized in a single place, we can
extend our features providing improved support for our users and their
regional, linguistic and cultural preferences encoded as part of the
identifier (for example: "en-DE-u-hc-h12-fw-sun" is English in Germany with
12h clock with first day of the week set to Sunday)

Before we can enable such customizations, we need to clean up our codebase
to make sure that all code that works with those identifiers will not break.

It may sound theoretical, but there are real cases where major software
(Android) had to implement terrible dirty hacks in their code to support
pseudolocalization and language fallbacking because the codebase was
sprinkled with DYI language identifier parsing and just could handle the
standard.

Fortunately, Gecko is already much cleaner than that, and we only have a
small number of places which we have to switch to Locale API, but that code
is hard to find or lint for, and we'd like to make sure that all new code
uses the correct logic.

== Language Negotiation ==

Tangible to that is the topic of language negotiation. If you ever need to
match one language identifier to another, please use `Locale::Matches`.
If you need to negotiate, use `LocaleService::NegotiateLanguages`.

== Next steps ==

Over the next months we'll be looking to migrate the remaining cases around
our codebase [1]. Then we'll want to turn on Unicode Extensions in
MozLocale class.

We'll also want to switch LocaleService API to return instances of `Locale`
in C++ and JS.

If you have any questions or feedback,
please let me know!

Thank you,
zb.

[0] https://unicode.org/reports/tr35/#Identifiers
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1433329
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Deprecating DTD for localization in Firefox UI

[Zibi Braniecki (Gandalf)]

Hi all,

tl;dr: Please use Fluent instead of adding or changing existing DTD strings.

== DTD and Fluent ==

DTD has been used as the core localization format and API inside of Firefox
since the dawn of Mozilla, allowing us to localize our XUL and other XML
files.

Unfortunately, DTD comes with a set of quite painful limitations, including
the so called Yellow Screen of Death which causes a crash at startup and
requires a complicated manual process by each user to recover. We’ve
recently been affected by this on beta [1].

For a couple years now, we’ve been working on a replacement for DTD - a new
localization system called Fluent [2] which (among other things) addresses
the shortcomings of DTD.

== Deprecating DTD for localization in browser chrome ==

Now that Fluent meets the performance requirements to replace DTD, we would
like to deprecate the DTD file format as a localization mechanism in
Firefox.

Please could you:

   1.

   Avoid introducing any new DTD strings in mozilla-central.
   2.

   If existing DTD strings would need to be changed, migrate them to Fluent
   [3] instead of updating the message identifier in DTD.


There are a few small areas where integrating Fluent is not entirely
trivial (most notably UA widgets) and those will be exempted from the
deprecation.

In case you believe there are technical reasons to continue using DTD in
your project, please consult fluent-reviewers group [4], which has been
recently extended with additions of Jared Wein, Gijs Kruitbosch and Edward
Lee. They can be reached in #fluent on Slack or fluent-reviewers on
phabricator.

The change does not affect release or beta uplifts, since we don’t change
strings in those. ESR uplifts will be exempted from the deprecation.

== Dashboard ==

In order to visualize our progress, we recently updated our dashboard:
https://arewefluentyet.com/

The new version introduces three milestones:

M1 - Removal of DTD from browser.xhtml - addresses Yellow Screen of Death
[5]

M2 - Migration of the startup path to Fluent [6]

Mx - Migration of the whole mozilla-central to Fluent [7]

Additional milestones may be added as we identify them.

Each one of these milestones will enable a set of new features for Firefox,
and deprecating DTD for localization is aiming to help us achieve them
sooner.


If you have any questions, let me know!

Cheers,

zb.

[1]
https://www.reddit.com/r/firefox/comments/d5wq0z/firefox_crashes_on_browser_startup_and_flashes/


[2] https://firefox-source-docs.mozilla.org/intl/l10n/l10n/index.html

[3]
https://firefox-source-docs.mozilla.org/intl/l10n/l10n/fluent_migrations.html

[4] fluent-reviewers group  (#fluent on Slack):

   -

   Francesco Lodolo
   -

   Jared Wein
   -

   Gijs Kruitbosch
   -

   Edward Lee
   -

   Axel Hecht
   -

   Staś Małolepszy
   -

   Zibi Braniecki

[5] https://bugzilla.mozilla.org/show_bug.cgi?id=1579477

[6] https://bugzilla.mozilla.org/show_bug.cgi?id=1501881

[7] https://bugzilla.mozilla.org/show_bug.cgi?id=1581212
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Updates to mozilla/intl/mozILocaleService and mozilla/intl/mozIOSPreferences

[Zibi Braniecki (Gandalf)]

Hi all,

A quick update on the LocaleService and OSPreferences APIs.

We just landed an update [0] that utilizes recently introduced new XPIDL
feature of handling Arrays[1]!

That means that we were able to turn all `getX` and `setX` methods into
properties, which means that instead of
`Services.locale.getRequestedLocales` and
`Services.locale.setRequestedLocales` you'll now have
`Services.locale.requestedLocales` and so on.

The change for OSPreferences will land soon [2].

The new APIs:

 - LocaleService.requestedLocales (rw)
 - LocaleService.availableLocales (rw)
 - LocaleService.packagedLocales (ro)
 - LocaleService.regionalPrefsLocales (ro)
 - OSPreferences.systemLocales (ro)
 - OSPreferences.regionalPrefsLocales (ro)

Documentation is available in the IDL files:
 -
https://searchfox.org/mozilla-central/source/intl/locale/mozILocaleService.idl
 -
https://searchfox.org/mozilla-central/source/intl/locale/mozIOSPreferences.idl

and in the docs: https://firefox-source-docs.mozilla.org/intl/locale.html

Also, PSA - the new Array<> API is awesome and you should switch to it!

Cheers,
zb.

[0] https://bugzilla.mozilla.org/show_bug.cgi?id=1491394
[1] https://bugzilla.mozilla.org/show_bug.cgi?id=1474369
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1493220
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

[L10n][BiDi] Fluent now controls content directionality

[Zibi Braniecki (Gandalf)]

Hi all,

A small update from the Fluent world - with landing of bug 1480798 Fluent
now controls the directionality of its roots.

That means that you don't need to do any special magic for RTL locales, as
we will set `localedir` and `dir` (XUL and HTML) attributes on the document
element and flip it correctly in case of a locale change.

For most of the layout it is advised to write as much of it as possible in
a directionality-independent way (so first/last rather than left/right),
but if special rules for direction are necessary, `html[dir=rtl]` and
`html[dir=ltr]` should be enough!

Once exception is pseudolocales, where strategy `bidi` does not currently
automatically trigger directionality switch. I filled bug 1481325 to fix
that.

Enjoy,
zb.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Pseudolocalization in Firefox Nightly!

[Zibi Braniecki (Gandalf)]

Hi all!

We've just landed a new hot developer-oriented feature in Firefox Nightly -
Pseudolocalization.

Pseudolocalization allows developers to quickly test their UI against fake
localizations helping us discover internationalization issues earlier in
the cycle.

You can find an introduction video here:
https://www.youtube.com/watch?v=pmT9PINv6nE
An introduction blog post here:
https://diary.braniecki.net/2018/06/07/pseudolocalization-in-firefox/

And documentation here:
https://firefox-source-docs.mozilla.org/intl/l10n/docs/l10n/fluent_tutorial.html#pseudolocalization

The functionality is directly related to the new localization framework -
Fluent - and in result works in areas covered by Fluent - mainly
Preferences at the moment.
As Fluent becomes available in other parts of our codebase,
pseudolocalization will come with it.

We're looking forward to hear your feedback and requests for more features!

zb.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Everything you always wanted to know about I18n/L10n/L12y, but....

[Zibi Braniecki (Gandalf)]

Hi!

I'm very proud to introduce the latest addition to our family of "explain
ourselves" efforts - shiny, new developer docs about locale management,
internationalization and localization practices at Mozilla.

https://firefox-source-docs.mozilla.org/intl/index.html

We put a lot of effort to make it a "cup of coffee per topic" size wise,
and hope you'll get to easily identify the topics relevant to your work.

The first topic - Locale Management - is for that time when you need to
make your code work with languages, select them, manage, negotiate etc.

The other two - UI Internationalization and Localization - are important if
you ever work with any front-end code at Mozilla.

We focused on Firefox and Gecko apps, but almost everything applies
universally across all of our portfolio.

I'd love to hear your feedback on whether we are making progress in turning
complex and hermetic field of knowledge more approachable and easier to
work with

If you any questions, please let me know! We'll do our best to keep those
documents, and our extensive in-source-code docs in sync and up to date.

Enjoy your coffee :)
zb.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: PSA: Chrome-only WebIDL interfaces no longer require DOM peer review

[Zibi Braniecki (Gandalf)]

On Thu, Mar 8, 2018 at 7:09 PM, Bobby Holley  wrote:

> I just looked at the first 10 methods/attributes on that interface. None
> of them are remotely performance-sensitive, and several are test-only. If
> we see certain methods on it show up in profiles, we should move those
> methods to WebIDL, rather than converting things wholesale per-interface.
>
>

Are you suggesting that we build two IDLs per interface - one for
perfomance-bound methods and the other for "non-performance-critical" ones?

I'm just a used of XPIDL/WebIDL but I noticed that almost every interface I
build or use has a mixture of those two types.

>From that perspective, ability to write a single IDL file and mark which
functions should be perf-critical (or the opposite) would be significantly
better UX and more fine-tunable over time.

But if your recommendation is to go for two, I would really appreciate a
tutorial explaining how to design a C++ and JS API with those two IDLs
separating perf-critical methods from others.

zb.
p.s. I understand your concern about sinking time into developing a WebIDL
for XPIDL replacement. At the same time, my understanding is that no matter
how much time something took in the past, we (in general) design our
architecture for a much longer future. So if we can introduce something
soon that will allow all new APIs to benefit from it, the argument of how
many APIs we already have that use the old model is diminishing.
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

BCP47 normalization of Language Tags in Gecko/Firefox

[Zibi Braniecki (Gandalf)]

Hi all!

A couple changes just landed in Gecko that will tighten our handling of
language tags and locales to BCP47 standard.

tl;dr:
 - Use mozilla::locale::Locale for any operations on Locale codes in C++
 - Expect LocaleService::NegotiateLanguages to return canonicalized BCP47
langauge tags

== Locale ==

We gained mozilla::intl::Locale class [0] which allows any C++ code to
validate, normalize and parse language tags.

Please, start using this code whenever you find yourself having to work
with language tags and analyze them.

For example instead of trying to write `langTag.Substring(0,
2).Equals("en")` which is insanely brittle, use:

```
Locale loc = Locale(langTag);
loc.GetLanguage().Equals("en");
```

You can also compare directly `loc == loc2` or using `.Match` which can
take ranges (to make `en` be treated as `en-*-*-*` and match `en-US`)

For JS code, we just advanced the `Intl.Locale` proposal to Stage 3 [0] at
this weeks TC39 meeting and will implement it in SpiderMonkey soon [1].

The locale class implements a direct subset of BCP47 and will be expanded
over time to handle more (all?) of it.

As we will want to start using Unicode Extension Keys around the codebase,
it becomes important that we can rely on all language tags being correctly
parsed and that means that we don't have DIY code around our codebase that
attempts to parse them on its own.

== Language Negotiation ==

There is also a change to our language negotiation handled by
mozilla::locale::LocaleService::NegotiateLanguages and
`Services.locale.negotiateLanguages`.

We now canonicalize the output (because we use the Locale class mentioned
above). That means that if you pass `ES-es` language tag, we'll return
`es-ES`.

It is important that we conform our codebase to the BCP47 standard and if
you expect to face non-canonicalized code, you can always use Locale class
or `Intl.getCanonicalLocales` ECMA402 API to get the canonicalized version
of any locale list.

If you have any questions, please, let me know!
zb.
p.s. We still support *one* extension to BCP47 which is 3-letter variants,
and we do this exclusively for the `ja-JP-mac` case. There's a conversation
about how to get rid of it [3] as it's the sole reason we have to maintain
both `LocaleService::GetAppLocalesAsLangTags` and
`LocaleService::GetAppLocalesAsBCP47`.


[0]
https://hg.mozilla.org/integration/autoland/file/tip/intl/locale/MozLocale.h#l16
[1] https://github.com/tc39/proposal-intl-locale/
[2] https://bugzilla.mozilla.org/show_bug.cgi?id=1433303
[3] https://bugzilla.mozilla.org/show_bug.cgi?id=1424953
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

Re: `general.useragent.locale` is no more. All hail `intl.locale.requested`

[Zibi Braniecki (Gandalf)]

Yes!
Thank you for bringing it up. I forgot to mention it in my original message.

We also removed `intl.locale.matchOS` and turned an empty
`intl.locale.requested` to mean "use OS locales" [0].

zb.

[0]
https://hg.mozilla.org/integration/autoland/file/tip/intl/locale/LocaleService.cpp#l75

On Tue, Dec 5, 2017 at 3:47 PM, Mike Hommey <m...@glandium.org> wrote:

> On Tue, Dec 05, 2017 at 03:15:06PM -0800, Zibi Braniecki (Gandalf) wrote:
> > Hi all,
> >
> > We just landed a major patch which replaces `general.useragent.locale`
> pref
> > with a new pref `intl.locale.requested`.
> >
> > Historically, `general.useragent.locale` has been widely used to set a
> > locale for Firefox UI.
> >
> > This year, we introduced a full new API called
> mozilla::intl::LocaleService
> > which allows for setting and reading via setRequestedLocales and
> > getRequestedLocales, respectively.
> >
> > Behind some linting and checking, the API still used
> > `general.useragent.locale` which limited us due to the nature of the pref
> > and how it stored data.
> >
> > With the change, we introduce a new pref - `intl.locale.requested`, which
> > can be set in the same manner if needed, but can also handle a list of
> > locales separated via `,` character and is validated to accept only
> > well-formed BCP47 language tags [1] making our locale handling much more
> > flexible and resilient.
> >
> > This is one of the last major changes in the grand rewrite of how Gecko
> > handles locales and language negotiation.
> >
> > If you need to read/write the requested locales it is *highly* preferred
> > that you use the (mozI)LocaleService API over reading/writing to the pref
> > itself, but if you must, the code will be able to handle your change with
> > grace.
> >
> > Over last months we removed all direct writes/reads of the pref, so I
> hope
> > there is nothing remaining, and we also introduced a migration code in
> > nsBrowserGlue for 59 users, but if you happen to encounter a regression,
> > please report it and CC me.
> >
> > If you have any questions about it, let me know!
>
> Any change wrt intl.locale.matchOS?
>
> Mike
>
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform

`general.useragent.locale` is no more. All hail `intl.locale.requested`

[Zibi Braniecki (Gandalf)]

Hi all,

We just landed a major patch which replaces `general.useragent.locale` pref
with a new pref `intl.locale.requested`.

Historically, `general.useragent.locale` has been widely used to set a
locale for Firefox UI.

This year, we introduced a full new API called mozilla::intl::LocaleService
which allows for setting and reading via setRequestedLocales and
getRequestedLocales, respectively.

Behind some linting and checking, the API still used
`general.useragent.locale` which limited us due to the nature of the pref
and how it stored data.

With the change, we introduce a new pref - `intl.locale.requested`, which
can be set in the same manner if needed, but can also handle a list of
locales separated via `,` character and is validated to accept only
well-formed BCP47 language tags [1] making our locale handling much more
flexible and resilient.

This is one of the last major changes in the grand rewrite of how Gecko
handles locales and language negotiation.

If you need to read/write the requested locales it is *highly* preferred
that you use the (mozI)LocaleService API over reading/writing to the pref
itself, but if you must, the code will be able to handle your change with
grace.

Over last months we removed all direct writes/reads of the pref, so I hope
there is nothing remaining, and we also introduced a migration code in
nsBrowserGlue for 59 users, but if you happen to encounter a regression,
please report it and CC me.

If you have any questions about it, let me know!
zb.

[1] https://tools.ietf.org/html/bcp47
___
dev-platform mailing list
dev-platform@lists.mozilla.org
https://lists.mozilla.org/listinfo/dev-platform