Hi, Did you see PR #12332? https://github.com/django/django/pull/12332 אורי u...@speedy.net
On Sat, Jan 11, 2020 at 3:38 AM Matemática A3K <matematica....@gmail.com> wrote: > Trying to recap all the discussion done in the mailing list, Trac and > Github: > > The problem that was originally reported in #30439 was about mixed plural > forms in catalogs bundled with Django, which led to broken translations. > > Then, it added the not announced changes in the plural forms of locales, > which led to break users' translations. > > These problems occur when catalogs gets updated in Django and users > updates their version. > > Michal proposed not merging and doing catalogs look-ups per language, > Claude proposed the "dict-merge" policy (have a dict of merged catalogs > according to their plural forms), which is a variant of Michal's. > > The "dict-merge" policy have the problem of updates in the plural forms in > Django won't reach users' catalogs. > > Whether merge or not merge the catalogs (the merging policy) is about how > to encourage the user to take action, not the fix itself. > > The fix is on the warning of the situation and on the tools for addressing > it. > > The proposed fix is: > 1- Warn the user at a system check level and at run-time > 2- Use the "not-merge" policy to encourage action. > 3- Provide the makemessages --comply-plural-forms tool so both Django and > its users can have consistent plural forms in their catalogs conveniently. > 4- Provide the LOCALE_ROOT setting and the makemessages > --collect-base-catalogs tool so users can make changes in their plural > forms conveniently and persisted across updates. > > The PR is cluttered with a lot of inconsistencies fixes across all > catalogs bundled with Django and the catalogs used for tests. > > Once a warning is raised when merging (either merging or not), most of the > tests will fail because of that warning, so all bundled catalogs (and those > used for tests) have to be aligned in order to have things working. > > Here are the files of the PR filtered: > https://github.com/django/django/pull/12280/files?file-filters%5B%5D=.py&file-filters%5B%5D=.txt > > For ensuring Django's catalogs consistency, there is a test that will fail > if there is any catalog unaligned. The next time Django's catalogs will be > updated, CI will check the consistency. If not, who will do the merge has > to run "python -m django makemessages --comply-plural-forms" and "python -m > django compilemessages" on the corresponding dir (django.conf, > django.contrib.app or the test locale) to fix it. This way, the messages > from the provider will be retained and the consistency be assured. > > There are valid reasons for users to customize their plural forms, i.e. > fixing a broken one while a fix is in the way or use another implementation > of the standard. Having to modify the source distribution for customizing > is not ideal, besides not being persistent across updates and may not be > possible in some setups. This should be done locally, at the project tree. > This is what the LOCALE_ROOT setting addresses. > > Claude already objected this fix though he did not provide reasons ( > https://github.com/django/django/pull/12280#issuecomment-571483273). > > Does anyone see any rationale, design or implementation problem in the > fix? Any comment is welcomed :) > > On Mon, Jan 6, 2020 at 4:17 PM Matemática A3K <matematica....@gmail.com> > wrote: > >> >> >> On Fri, Dec 13, 2019 at 1:04 AM Matemática A3K <matematica....@gmail.com> >> wrote: >> >>> >>> >>> On Fri, Dec 6, 2019 at 2:14 AM Matemática A3K <matematica....@gmail.com> >>> wrote: >>> >>>> >>>> >>>> On Thu, Dec 5, 2019 at 4:23 PM Maciek Olko <maciej.o...@gmail.com> >>>> wrote: >>>> >>>>> Hi, >>>>> I am wondering if Django shouldn't use Unicode Plural Rules as >>>>> standard and promote it for third-party apps. Even if sometimes number of >>>>> forms are not applicable to certain cases, there may be cases when all of >>>>> forms will be needed. >>>>> >>>> >>>> I agree for Django, for third-party translations, for what I understood >>>> (CIIW) this may be a differentiator >>>> >>> >>> Under this rationale, this would be the fix: >>> >>> .. _plural-forms: >>> >>> Plural Forms >>> ~~~~~~~~~~~~ >>> >>> Django does not support multiple plural forms in catalogs. As all >>> translation >>> catalogs are merged, only the plural form in the main Django po file >>> (located in >>> ``django/conf/locale/<lang_code>/LC_MESSAGES/django.po``) is considered. >>> >>> Plural forms in all other po files are ignored by the GNU gettext >>> merging process. >>> >>> Therefore, you shouldn't use different plural forms in your project or >>> application >>> po files, as it may lead to unexpected results. >>> >>> To prevent inconsistencies and undesired behaviors in translations, >>> Django will >>> not merge any catalog that contains a different plural form than the >>> main one >>> and will issue a warning about the conflicting catalog. >>> >>> This conflict may arise mostly in two situations: >>> >>> * when the main plural form for a language is updated >>> in Django and your po files were generated with a previous one, or >>> * when including third-party translations with a different plural form. >>> >>> Django follows the standards provided by `Unicode >>> < >>> http://www.unicode.org/cldr/charts/latest/supplemental/language_plural_rules.html >>> >`_ >>> for plural equations for each language, and encourages to align with it. >>> >>> If you had generated your catalog with a previous version of Django, the >>> standard may have changed or a bug has been fixed. >>> >>> For aligning with the new version of the standard (or addressing the >>> bug), you >>> you may use >>> djadmin:`django-admin makemessages --comply-plural-form<makemessages>` to >>> reorganize your catalog so it is aligned with it. >>> >>> The script will ask you to map each plural form in the new form to your >>> catalog's forms. After this, the messages in your catalog will be >>> reorganized >>> accordingly. In the case of an increase in the number of plurals, it >>> will fill >>> those with the previously chosen forms so you avoid having translation >>> results >>> in the fallback language or the original string - this will produce the >>> same >>> results as before though you may want to update those later for a >>> better expression of the language if it corresponds. >>> >>> In the cases where you want to use a "bleeding-edge-yet-to-be-approved" >>> plural >>> form, define the :setting:`LOCALE_ROOT` and use >>> djadmin:`django-admin makemessages >>> --collect-base-catalogs<makemessages>`. >>> This will merge all the locale catalogs bundled with Django into your >>> :setting:`LOCALE_ROOT` and use them as your main Django po file. >>> >>> .. versionchanged:: 3.1 >>> >>> Handling plural forms as described above was added. >>> >>> The problem that I see to updating plural forms in the package tree is >>> that modifying the base package seems not be a good practice to me >>> (independently of not being persistent across updates), you either copy, >>> subclass, monkey patch, etc. So, I think this is a more proper solution. >>> >>> Any thoughts? >>> >> >> Well, seems that the Three Magic Kings have left us this >> <https://github.com/django/django/pull/12280> >> >> >>> >>> >>>> >>>>> >>>>> Especially if implementing having different plural rules for various >>>>> apps is not trivial. >>>>> >>>>> For the record, here is a section of Transifex documentation that >>>>> describes their statement about plural rules and Unicode standard: >>>>> https://docs.transifex.com/localization-tips-workflows/plurals-and-genders#how-pluralized-strings-are-handled-by-transifex >>>>> . >>>>> >>>>> Regards, >>>>> Maciej >>>>> >>>>> czw., 5 gru 2019 o 08:00 Matemática A3K <matematica....@gmail.com> >>>>> napisał(a): >>>>> >>>>>> While testing the "not-merging" policy, I got this: >>>>>> https://pastebin.com/ihyAiYtc >>>>>> Those warnings should get to the Translators teams if they are not >>>>>> looking here, i.e. >>>>>> >>>>>> https://github.com/django/django/blob/master/django/contrib/sessions/locale/he/LC_MESSAGES/django.po >>>>>> is still using a 2-plurals form in a 4 plural form - it doesn't >>>>>> matter in this case because there are no translations with plurals, but >>>>>> they trigger the warning because the pf hasn't been updated and they >>>>>> should >>>>>> if the Transifex front-end use the pf of a file for showing the options >>>>>> for >>>>>> translating and to be inline with the main form. >>>>>> >>>>>> >>>>>> On Thu, Dec 5, 2019 at 12:10 AM Matemática A3K < >>>>>> matematica....@gmail.com> wrote: >>>>>> >>>>>>> >>>>>>> >>>>>>> On Wed, Dec 4, 2019 at 2:25 AM Claude Paroz <cla...@2xlibre.net> >>>>>>> wrote: >>>>>>> >>>>>>>> Le mercredi 4 décembre 2019 03:41:51 UTC+1, Matemática A3K a écrit : >>>>>>>>> >>>>>>>>> (...) >>>>>>>>> >>>>>>>>> But, then I realized that there is major caveat on this approach, >>>>>>>>> and that is that updates on the plural equation won't reach users' >>>>>>>>> catalogs, because their catalogs will be kept separately one the >>>>>>>>> plural >>>>>>>>> form differs. This would be the case that Shai pointed on Django 2.2 >>>>>>>>> with >>>>>>>>> the incorrect plural equation for HE. People who have generated their >>>>>>>>> catalogs with makemessages on 2.2 will have a wrong plural equation, >>>>>>>>> that >>>>>>>>> once it is fixed on a new release, it won't reach their catalogs >>>>>>>>> because >>>>>>>>> they will be kept apart. >>>>>>>>> >>>>>>>> >>>>>>>> Sorry, I'm not following you here. The catalog merge process is >>>>>>>> happening in realtime when Django starts, so any po file update is >>>>>>>> instantly reflected in the translation infrastructure. I don't see any >>>>>>>> caveat here. >>>>>>>> >>>>>>> >>>>>>> Yes, but only for the default language, the rest are lazily loaded >>>>>>> on demand - this is why I think it is better do the check at a system >>>>>>> level, if you have more than a language available, the others will be >>>>>>> merged for loading them once something triggers it (like an >>>>>>> "activate("LANGUAGE_CODE")) and only there you would start to see the >>>>>>> warnings. >>>>>>> >>>>>>> For the example of the caveat of the dict-merging, it would be like >>>>>>> this: >>>>>>> >>>>>>> - Someone starts its translation with Django 2.2 for Hebrew via >>>>>>> makemessages, which copies the main plural form to the new file, and >>>>>>> fills >>>>>>> the translations. >>>>>>> - The user upgrades to Django 2.2.x or up, which contains a fixed >>>>>>> plural form for Hebrew >>>>>>> - Under the "dict-merge" policy, the Django translation catalog >>>>>>> would have 2 entries, one for the main plural form (the new one) and >>>>>>> other >>>>>>> the catalog generated in the past (the user's one). >>>>>>> >>>>>>> Strings in the user catalog would use a "worse" plural equation, >>>>>>> while strings in the Django catalog will use the better one. Updates >>>>>>> won't >>>>>>> reach to users' catalogs, unless they explicitly update them, because >>>>>>> once >>>>>>> the user's po is created, makemessages won't update the header, it will >>>>>>> do >>>>>>> a msgmerge ( >>>>>>> https://github.com/django/django/blob/master/django/core/management/commands/makemessages.py#L603 >>>>>>> ). >>>>>>> >>>>>>> That's why something like "makemessages --update-plural-form" and a >>>>>>> warning about different plurals forms in the catalog would be needed, >>>>>>> even >>>>>>> under "dict-merge". >>>>>>> >>>>>>> With the current "merge" policy, because only the main form used, >>>>>>> the users would get the update but they won't be able to have parts of >>>>>>> the >>>>>>> catalog under a different pf. >>>>>>> >>>>>>> The way to have a different plural form than the main one with the >>>>>>> current code base would be by having a custom variant of the language >>>>>>> (which implements "the spirit of dict-merge": independent catalogs with >>>>>>> an >>>>>>> order of precedence). It's not ideal, you will loose the ability of >>>>>>> using >>>>>>> the browser's locale config to display the language out of the box >>>>>>> unless >>>>>>> you add extra code to handle it, something like "if locale == 'he': use >>>>>>> 'he_SP'" (no other caveat comes to my mind). But it would be they way of >>>>>>> having it RN without loosing the updates. >>>>>>> >>>>>>> >>>>>>>> >>>>>>>> Claude >>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to django-developers+unsubscr...@googlegroups.com. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/django-developers/5f2bf52a-997c-467d-b927-f2959507d32e%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/django-developers/5f2bf52a-997c-467d-b927-f2959507d32e%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to django-developers+unsubscr...@googlegroups.com. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/django-developers/CA%2BFDnhKtEB4sgUFwmyaExGLSnMU5yLkj0QZ1ubpF2gaRjBd1EQ%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/django-developers/CA%2BFDnhKtEB4sgUFwmyaExGLSnMU5yLkj0QZ1ubpF2gaRjBd1EQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Django developers (Contributions to Django itself)" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to django-developers+unsubscr...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/django-developers/CALYYG80tyJdmEw1LJaV0sa6OHQ0HMkecO2wsHJkvvyQ4%3DT0Tzw%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/django-developers/CALYYG80tyJdmEw1LJaV0sa6OHQ0HMkecO2wsHJkvvyQ4%3DT0Tzw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- > You received this message because you are subscribed to the Google Groups > "Django developers (Contributions to Django itself)" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to django-developers+unsubscr...@googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/django-developers/CA%2BFDnhJ7o%3Dhu3cG1gyN85HyjApOEeraD%2BJMG%3DbmG-jb5aj14Cw%40mail.gmail.com > <https://groups.google.com/d/msgid/django-developers/CA%2BFDnhJ7o%3Dhu3cG1gyN85HyjApOEeraD%2BJMG%3DbmG-jb5aj14Cw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group. To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CABD5YeG46Agx2pcdnwnDZPWzyp1mu1S2wkgd7WcCYQh_5XQMNA%40mail.gmail.com.
