This is an automated email from the ASF dual-hosted git repository.
voonhous pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/hudi.git
The following commit(s) were added to refs/heads/asf-site by this push:
new bc7e03a1200e docs(blog): add the InfoQ article on computing time in
queue for Hudi Kafka pipelines (#19795)
bc7e03a1200e is described below
commit bc7e03a1200e77cd27940d173ab066bf09c356b8
Author: Ranga Reddy <[email protected]>
AuthorDate: Thu Sep 3 09:46:31 2026 +0530
docs(blog): add the InfoQ article on computing time in queue for Hudi Kafka
pipelines (#19795)
* docs(blog): add the InfoQ article on computing time in queue for Hudi
Kafka pipelines
"Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake
Pipelines at
Petabyte Scale", by Srikanth Mamidala, InfoQ, 26 August 2026.
Twilio ingests five trillion records a month through Hudi Streamer, and the
article is
about why Kafka offset lag is the wrong freshness signal for those
pipelines: it says
how far behind a consumer is, not how old the data is. It derives a
time-in-queue metric
by reading the Kafka checkpoint out of the latest .hoodie commit file,
seeking to that
offset in the topic, and measuring the timestamp delta against now, so it
runs off the
timeline Hudi Streamer already writes with no change to the pipeline.
Added as a redirect stub, the established shape for externally hosted
articles, with the
author declared inline rather than in authors.yml since it is a one-off
contributor.
The hero is drawn for this post at the required 1200x600 in the site's
brand palette,
rather than copied from the article. It shows one Kafka partition with the
checkpoint
recorded in the latest commit, and the same span measured twice: as an
offset count
above and as elapsed time below.
validate-blog.py rejects the twilio tag, so it is added to ALLOWED_TAGS
here, as the
script's own error message directs. It follows the existing company-tag
convention
alongside uber, walmart, robinhood, upstox, metica and southwest airlines,
and needs a
reviewer's approval.
* fix(blog-validator): raise on missing fields
The missing-field branch in validate_blog() only appended an error
when field == 'author', but 'author' is not in REQUIRED_FIELDS, so a
post with no title, category or image passed silently. Report the
missing field by name; the separate author/authors check is unchanged.
CI validates only the blog files changed in a PR, so existing posts
without an image (22 today) are unaffected; only --all reports them.
* docs(blog): shrink beyond-offset-lag hero PNG
The hero was an RGBA PNG with a fully opaque alpha channel. Quantize
it to a 256-color palette: 272 KB to 31 KB at the same 1200x600 with
no visible change.
---------
Co-authored-by: voon <[email protected]>
---
.github/scripts/validate-blog.py | 6 ++----
...omputing-time-in-queue-for-apache-hudi-pipelines.mdx | 16 ++++++++++++++++
...omputing-time-in-queue-for-apache-hudi-pipelines.png | Bin 0 -> 31032 bytes
3 files changed, 18 insertions(+), 4 deletions(-)
diff --git a/.github/scripts/validate-blog.py b/.github/scripts/validate-blog.py
index 8f15cbb08d5b..a8b8ed4089dd 100644
--- a/.github/scripts/validate-blog.py
+++ b/.github/scripts/validate-blog.py
@@ -29,7 +29,7 @@ ALLOWED_TAGS = {
'debezium', 'data skipping', 'daft', 'ai',
'table format', 'starrocks', 'halodoc', 'gdpr', 'schema', 'scd',
'observability', 'metadata', 'meetup', 'key generation', 'docker',
- 'cleaner', 'apache hive', 'apache doris', 'vector search', 'upstox',
+ 'cleaner', 'apache hive', 'apache doris', 'vector search', 'upstox',
'twilio',
'tla specification', 'streamlit', 'rag', 'presto', 'postgres',
'file sizing', 'etl', 'databricks', 'data warehouse', 'applied intuition',
'conference', 'compaction', 'bootstrap', 'apache parquet', 'announcement',
@@ -145,9 +145,7 @@ def validate_blog(filepath: str) -> list[str]:
# Check required fields
for field in REQUIRED_FIELDS:
if field not in frontmatter:
- # author or authors is acceptable
- if field == 'author' and 'authors' not in frontmatter:
- errors.append("Missing required field: 'author' or 'authors'")
+ errors.append(f"Missing required field: '{field}'")
elif not frontmatter[field]:
errors.append(f"Field '{field}' is empty")
diff --git
a/website/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.mdx
b/website/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.mdx
new file mode 100644
index 000000000000..6fe9ba9740c6
--- /dev/null
+++
b/website/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.mdx
@@ -0,0 +1,16 @@
+---
+title: "Beyond Offset Lag: Computing Time in Queue for Apache Hudi Data Lake
Pipelines at Petabyte Scale"
+authors:
+- name: Srikanth Mamidala
+category: case-study
+image:
/assets/images/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.png
+tags:
+- apache kafka
+- observability
+- hudi streamer
+- twilio
+---
+
+import Redirect from '@site/src/components/Redirect';
+
+<Redirect
url="https://www.infoq.com/articles/beyond-offset-lag-kafka-apache-hudi/";>Redirecting...
please wait!! </Redirect>
diff --git
a/website/static/assets/images/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.png
b/website/static/assets/images/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.png
new file mode 100644
index 000000000000..4297039f5c92
Binary files /dev/null and
b/website/static/assets/images/blog/2026-08-26-beyond-offset-lag-computing-time-in-queue-for-apache-hudi-pipelines.png
differ
