Re: [PR] Add progress summary to translation_completeness script [airflow]
potiuk commented on PR #51537: URL: https://github.com/apache/airflow/pull/51537#issuecomment-2969784821 > Do we want this script to be runnable with breeze command? Maybe be automatically commented with output on PRs that modify translations? No need. With `uv` and `uv run` we do not need breeze to run it. For standalone scripts that do one thing, there is no neeed to have the whole breeze environment setup. > Maybe be automatically commented with output on PRs that modify translations? For the moment - as described in the process - it's more of a tool helpful for translators (to help with incremental adding of translations) and for release managers to see how much translation is left. As explained in the process - it's difficult to make it automaticly "fail" or otherwise signal translator that things are missing. Because it's the "english" translation change that will immediately make all the other translations "missing" things. Things in translation will get "missing" without involvment of the translators and it requires work of all translators to "fix" them independently. So automation in PR does not help much on PR level. What we already have have is "pre-commit" to check if there are any TODO: left. in translation after they were added with the `--add-missing`. But if someone figures a better way of automating it and workflow that can be applied here - yep, it could be used as part of it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
eladkal commented on PR #51537: URL: https://github.com/apache/airflow/pull/51537#issuecomment-2969394915 Do we want this script to be runnable with breeze command? Maybe be automatically commented with output on PRs that modify translations? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
potiuk commented on code in PR #51537:
URL: https://github.com/apache/airflow/pull/51537#discussion_r2136320376
##
airflow-core/src/airflow/ui/src/i18n/check_translations_completeness.py:
##
@@ -235,6 +245,123 @@ def get_outdated_entries_for_language(
return outdated
+def count_todo_and_total(obj):
+"""Recursively count TODO: translate entries and total entries in a dict
or list."""
+todo = 0
+total = 0
+if isinstance(obj, dict):
+for v in obj.values():
+t, n = count_todo_and_total(v)
+todo += t
+total += n
+elif isinstance(obj, list):
+for v in obj:
+t, n = count_todo_and_total(v)
+todo += t
+total += n
+else:
+total = 1
+if isinstance(obj, str) and obj.strip().startswith("TODO: translate"):
+todo = 1
+return todo, total
+
+
+def print_translation_progress(console, locale_files, missing_counts, summary):
+from rich.table import Table
+
+tables = defaultdict(lambda: Table(show_lines=True))
+all_files = set()
+for lf in locale_files:
+all_files.update(lf.files)
+for lf in locale_files:
+lang = lf.locale
+table = tables[lang]
+table.title = f"Translation Progress: {lang}"
+table.add_column("File", style="bold cyan")
+table.add_column("Missing", style="red")
+table.add_column("Extra", style="yellow")
+table.add_column("TODOs", style="magenta")
+table.add_column("Translated", style="green")
+table.add_column("Total", style="bold")
+table.add_column("Percent", style="bold")
+total_missing = 0
+total_extra = 0
+total_todos = 0
+total_translated = 0
+total_total = 0
+for filename in sorted(all_files):
+file_path = Path(LOCALES_DIR / lang / filename)
+# Always get total from English version
+en_path = Path(LOCALES_DIR / "en" / filename)
+if en_path.exists():
+with open(en_path) as f:
+en_data = json.load(f)
+file_total = sum(1 for _ in flatten_keys(en_data))
+else:
+file_total = 0
+if file_path.exists():
+with open(file_path) as f:
+data = json.load(f)
+file_missing = missing_counts.get(filename, {}).get(lang, 0)
+file_extra = len(summary.get(filename, LocaleSummary({},
{})).extra_keys.get(lang, []))
+
+# Count TODOs
+def count_todos(obj):
+if isinstance(obj, dict):
+return sum(count_todos(v) for v in obj.values())
+if isinstance(obj, list):
+return sum(count_todos(v) for v in obj)
+if isinstance(obj, str) and obj.strip().startswith("TODO:
translate"):
+return 1
+return 0
+
+file_todos = count_todos(data)
+file_translated = file_total - file_missing
+percent = 100 * file_translated / file_total if file_total
else 100
+style = (
+"bold green"
+if file_missing == 0 and file_extra == 0 and file_todos == 0
+else (
+"yellow" if file_missing < file_total or file_extra >
0 or file_todos > 0 else "red"
+)
+)
+else:
+file_missing = file_total
+file_extra = len(summary.get(filename, LocaleSummary({},
{})).extra_keys.get(lang, []))
+file_todos = 0
+file_translated = 0
+percent = 0
+style = "red"
+table.add_row(
+f"[bold reverse]{filename}[/bold reverse]",
+str(file_missing),
+str(file_extra),
+str(file_todos),
+str(file_translated),
+str(file_total),
+f"{percent:.1f}%",
+style=style,
+)
+total_missing += file_missing
+total_extra += file_extra
+total_todos += file_todos
+total_translated += file_translated
+total_total += file_total
+percent = 100 * total_translated / total_total if total_total else 100
+table.add_row(
+"All files",
+str(total_missing),
+str(total_extra),
+str(total_todos),
+str(total_translated),
+str(total_total),
+f"{percent:.1f}%",
+style="red" if percent < 100 else "bold green",
+)
+for _lang, table in tables.items():
Review Comment:
Feel free :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use th
Re: [PR] Add progress summary to translation_completeness script [airflow]
jscheffl commented on code in PR #51537:
URL: https://github.com/apache/airflow/pull/51537#discussion_r2136280830
##
airflow-core/src/airflow/ui/src/i18n/check_translations_completeness.py:
##
@@ -235,6 +245,123 @@ def get_outdated_entries_for_language(
return outdated
+def count_todo_and_total(obj):
+"""Recursively count TODO: translate entries and total entries in a dict
or list."""
+todo = 0
+total = 0
+if isinstance(obj, dict):
+for v in obj.values():
+t, n = count_todo_and_total(v)
+todo += t
+total += n
+elif isinstance(obj, list):
+for v in obj:
+t, n = count_todo_and_total(v)
+todo += t
+total += n
+else:
+total = 1
+if isinstance(obj, str) and obj.strip().startswith("TODO: translate"):
+todo = 1
+return todo, total
+
+
+def print_translation_progress(console, locale_files, missing_counts, summary):
+from rich.table import Table
+
+tables = defaultdict(lambda: Table(show_lines=True))
+all_files = set()
+for lf in locale_files:
+all_files.update(lf.files)
+for lf in locale_files:
+lang = lf.locale
+table = tables[lang]
+table.title = f"Translation Progress: {lang}"
+table.add_column("File", style="bold cyan")
+table.add_column("Missing", style="red")
+table.add_column("Extra", style="yellow")
+table.add_column("TODOs", style="magenta")
+table.add_column("Translated", style="green")
+table.add_column("Total", style="bold")
+table.add_column("Percent", style="bold")
+total_missing = 0
+total_extra = 0
+total_todos = 0
+total_translated = 0
+total_total = 0
+for filename in sorted(all_files):
+file_path = Path(LOCALES_DIR / lang / filename)
+# Always get total from English version
+en_path = Path(LOCALES_DIR / "en" / filename)
+if en_path.exists():
+with open(en_path) as f:
+en_data = json.load(f)
+file_total = sum(1 for _ in flatten_keys(en_data))
+else:
+file_total = 0
+if file_path.exists():
+with open(file_path) as f:
+data = json.load(f)
+file_missing = missing_counts.get(filename, {}).get(lang, 0)
+file_extra = len(summary.get(filename, LocaleSummary({},
{})).extra_keys.get(lang, []))
+
+# Count TODOs
+def count_todos(obj):
+if isinstance(obj, dict):
+return sum(count_todos(v) for v in obj.values())
+if isinstance(obj, list):
+return sum(count_todos(v) for v in obj)
+if isinstance(obj, str) and obj.strip().startswith("TODO:
translate"):
+return 1
+return 0
+
+file_todos = count_todos(data)
+file_translated = file_total - file_missing
+percent = 100 * file_translated / file_total if file_total
else 100
+style = (
+"bold green"
+if file_missing == 0 and file_extra == 0 and file_todos == 0
+else (
+"yellow" if file_missing < file_total or file_extra >
0 or file_todos > 0 else "red"
+)
+)
+else:
+file_missing = file_total
+file_extra = len(summary.get(filename, LocaleSummary({},
{})).extra_keys.get(lang, []))
+file_todos = 0
+file_translated = 0
+percent = 0
+style = "red"
+table.add_row(
+f"[bold reverse]{filename}[/bold reverse]",
+str(file_missing),
+str(file_extra),
+str(file_todos),
+str(file_translated),
+str(file_total),
+f"{percent:.1f}%",
+style=style,
+)
+total_missing += file_missing
+total_extra += file_extra
+total_todos += file_todos
+total_translated += file_translated
+total_total += file_total
+percent = 100 * total_translated / total_total if total_total else 100
+table.add_row(
+"All files",
+str(total_missing),
+str(total_extra),
+str(total_todos),
+str(total_translated),
+str(total_total),
+f"{percent:.1f}%",
+style="red" if percent < 100 else "bold green",
+)
+for _lang, table in tables.items():
Review Comment:
Maybe in the loop you should have skipped "en" :-D as this is always 100%
--
This is an automated message from the Apache Git Service
Re: [PR] Add progress summary to translation_completeness script [airflow]
potiuk merged PR #51537: URL: https://github.com/apache/airflow/pull/51537 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] Add progress summary to translation_completeness script [airflow]
potiuk commented on PR #51537: URL: https://github.com/apache/airflow/pull/51537#issuecomment-2956180065 https://github.com/user-attachments/assets/b440c96c-4371-44ed-aa13-750cf188ae37"; /> -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@airflow.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
