milenkovicm opened a new pull request, #2269:
URL: https://github.com/apache/datafusion-ballista/pull/2269
# Which issue does this PR close?
Closes #.
# Rationale for this change
Previously job ids were created randomly and as such they were not sortable,
also, there was a chance for collision in case we save and keep them for a long
tine
# What changes are included in this PR?
Replaced random job ID generation with monotonically increasing, sortable
Snowflake-style IDs.
- Added `ferroid` dependency (snowflake, atomic, base32, async-tokio
features).
- `TaskManager` now holds an
`Arc<AtomicSnowflakeGenerator<SnowflakeMastodonId, MonotonicClock>>`.
- `generate_job_id()` became async and now produces a base32-encoded
Snowflake ID instead of a random 7-char alphanumeric string (rand/Alphanumeric
usage removed).
- `TaskManager::generate_job_id()` is now `async`.
- Snowflake ids are encoded used base32 encoding, they look something like
(`[037Z8BV46C000]`)
Net effect: job IDs are now time-ordered/sortable rather than random, which
is useful for chronological ordering, log correlation, and roughly indicates
submission order.
Currently `Mastodon` style snowflake id layout has been used which does not
hold machine id. [more details about
layouts](https://lib.rs/crates/ferroid#readme-supported-layouts)
# Are there any user-facing changes?
No
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
