sunchao commented on code in PR #5365:
URL: https://github.com/apache/datafusion-comet/pull/5365#discussion_r3917732904
##########
native/core/src/parquet/schema_adapter.rs:
##########
@@ -76,6 +77,583 @@ fn schema_has_field_ids(schema: &SchemaRef) -> bool {
schema.fields().iter().any(|f| parse_field_id(f).is_some())
}
+//
---------------------------------------------------------------------------------------------
+// JVM-shipped case tables: reproduce the PLANNING JVM's
`String.toLowerCase(Locale.ROOT)`,
+// which Spark's Parquet footer field matching is built on.
+//
+// The data arrives on `NativeScanCommon` (populated by `JvmCaseTables.scala`
when
+// case_sensitive = false), generated from the very JVM that plans the query,
so the native
+// matcher is correct BY CONSTRUCTION for whatever JDK runs Spark. The one
contextual mapping
+// (`Locale.ROOT` has exactly one: Greek capital sigma U+03A3) cannot be a
per-codepoint table
+// entry, so its condition is ported as an algorithm over a shipped
per-codepoint
+// classification -- see `JvmCaseTables::lowercase`.
+//
---------------------------------------------------------------------------------------------
+
+// Sigma-scan classes: the wire contract shared with `JvmCaseTables.scala`.
The classes are
+// the UAX#29-style word-break classes (ALetter, Numeric, MidLetter, MidNum,
MidNumLet,
+// Extend, Format) as the PLANNING JVM's legacy break iterator actually
realizes them --
+// probed per codepoint from `BreakIterator.isBoundary` on the JVM side --
plus classes for
+// its pre-UAX#29 extensions (the danda and the supplementary-plane behaviors
of its UTF-16
+// DFA), with cased variants split out via the JDK's `isCased`. Any codepoint
outside every
+// shipped range -- and any class value this build does not know -- is a word
boundary: the
+// sigma context scan stops there, which is also the safe reading for class
values added by a
+// NEWER JVM-side generator.
+const CLASS_ALETTER_CASED: u8 = 1;
+const CLASS_ALETTER: u8 = 2;
+const CLASS_NUMERIC: u8 = 3;
+const CLASS_MID_LETTER: u8 = 4;
+const CLASS_MID_NUM: u8 = 5;
+const CLASS_MID_NUM_LET: u8 = 6;
+/// Cased supplementary char: attaches to the preceding word and closes it
(and forms a word
+/// of its own at raw text start).
+const CLASS_SUPP_CASED: u8 = 7;
+/// U+0964/U+0965: word-terminal, chains only into digits.
+const CLASS_DANDA: u8 = 8;
+/// U+0345, the one cased combining mark: cased only when its run is attached
to a word.
+const CLASS_EXTEND_CASED: u8 = 9;
+/// Cased digit-base (Nl Roman numerals): joins like CLASS_ALETTER_CASED when
reached
+/// directly, but bridges only mid-num punctuation, never mid-letter.
+const CLASS_NUMERIC_CASED: u8 = 10;
+/// Non-cased Mn/Me marks: riders that attach only to genuine letter/digit
bases.
+const CLASS_EXTEND: u8 = 11;
+/// Word-forming non-cased supplementary letter: a genuine letter-base that
closes the word
+/// immediately after itself.
+const CLASS_SUPP_LETTER: u8 = 12;
+/// Cf format characters: fully transparent (WB4-style) -- deleted from the
sequence before
+/// the scans run, so a pure-format rider chain bridges mid punctuation
("AΣ-<ZWJ>b" is one
+/// word exactly like "AΣ-b").
+const CLASS_FORMAT: u8 = 13;
+/// Supplementary chars that attach to the preceding word but never form one
themselves
+/// (supplementary combining marks, tag characters): a cased mark riding on
one belongs to
+/// the sigma's word only when the run hangs off a real base
(`supp_mn_anchor`).
+const CLASS_SUPP_MN: u8 = 14;
+/// Word-forming supplementary digit: like CLASS_SUPP_LETTER except a riding
cased mark
+/// carries only across mid-num (digit-context) punctuation, never mid-letter.
+const CLASS_SUPP_NUM: u8 = 15;
+/// Not on the wire: the absence of a class.
+const CLASS_BOUNDARY: u8 = 0;
+
+const CAPITAL_SIGMA: char = '\u{03A3}';
+const SMALL_SIGMA: char = '\u{03C3}';
+const SMALL_FINAL_SIGMA: char = '\u{03C2}';
+
+/// What a supplementary-mark run ultimately hangs off (see `supp_mn_anchor`).
+#[derive(PartialEq, Eq, Clone, Copy)]
+enum SuppMnAnchor {
+ None,
+ Letter,
+ Digit,
+}
+
+/// The planning JVM's case data, parsed once per scan from `NativeScanCommon`
and attached to
+/// [`SparkParquetOptions`]. Two tables:
+///
+/// - `lower`: every codepoint the JVM lowercases non-identically, with its
full (possibly
+/// multi-char, e.g. U+0130 -> "i" + U+0307) replacement; codepoints
absent here lowercase
+/// to themselves;
+/// - `class_ranges`: sorted, disjoint `(start, end, class)` codepoint
ranges holding the
+/// word-break classification the JVM probed from its own `BreakIterator`.
+///
+/// `lowercase` applies Java's algorithm over that data: per codepoint, U+03A3
takes its
+/// contextual final/non-final form via the ported `isFinalCased` condition --
word-boundary
+/// based (the JDK's legacy break-iterator word rules), NOT the
Unicode-standard Final_Sigma
+/// case-ignorable skip, so e.g. "A1Σ" lowers to "a1ς" -- and every other
codepoint takes its
+/// table replacement. `JvmCaseTables.mirrorLowercase` on the Scala side is
the line-for-line
+/// mirror of this function over the same generated data; the JVM-side parity
suite proves the
+/// pair equal to the running JDK's `String.toLowerCase(Locale.ROOT)` across
the full codepoint
+/// space (calibrated to zero mismatches on JDK 17, 21, and 25).
+#[derive(Debug)]
+pub struct JvmCaseTables {
+ /// Non-identity lowercase mappings: codepoint -> full replacement string.
+ lower: HashMap<char, String>,
+ /// Sorted, disjoint (start, end, class) inclusive codepoint ranges for
the sigma scan.
+ class_ranges: Vec<(u32, u32, u8)>,
+ /// Precomputed content hash so `SparkParquetOptions`'s derived `Hash`
stays cheap.
+ fingerprint: u64,
+}
+
+impl PartialEq for JvmCaseTables {
+ fn eq(&self, other: &Self) -> bool {
+ self.fingerprint == other.fingerprint
+ && self.class_ranges == other.class_ranges
+ && self.lower == other.lower
+ }
+}
+
+impl Eq for JvmCaseTables {}
+
+impl Hash for JvmCaseTables {
+ fn hash<H: Hasher>(&self, state: &mut H) {
+ // Equal contents always produce the equal (deterministically
computed) fingerprint,
+ // so hashing only the fingerprint is consistent with `PartialEq`.
+ state.write_u64(self.fingerprint);
+ }
+}
+
+fn is_letter_base(cls: u8) -> bool {
+ cls == CLASS_ALETTER_CASED
+ || cls == CLASS_ALETTER
+ || cls == CLASS_SUPP_CASED
+ || cls == CLASS_SUPP_LETTER
+}
+
+fn is_digit_base(cls: u8) -> bool {
+ cls == CLASS_NUMERIC || cls == CLASS_NUMERIC_CASED
+}
+
+impl JvmCaseTables {
+ /// Parse the proto representation: `lower_cp`/`lower_repl` are
index-aligned, and
+ /// `class_ranges` holds (start, end, class) triples. Malformed input
(length mismatch,
+ /// trailing partial triple, out-of-range codepoints) is dropped
entry-by-entry rather
+ /// than rejected: every dropped entry degrades one codepoint to
identity/boundary
+ /// behavior instead of failing the scan.
+ pub fn from_proto(lower_cp: &[u32], lower_repl: &[String], class_ranges:
&[u32]) -> Self {
+ let mut lower = HashMap::with_capacity(lower_cp.len());
+ for (cp, repl) in lower_cp.iter().zip(lower_repl.iter()) {
+ if let Some(c) = char::from_u32(*cp) {
+ lower.insert(c, repl.clone());
+ }
+ }
+ let ranges: Vec<(u32, u32, u8)> = class_ranges
+ .as_chunks::<3>()
+ .0
+ .iter()
+ .filter(|t| t[0] <= t[1] && t[1] <= 0x10FFFF &&
u8::try_from(t[2]).is_ok())
+ .map(|t| (t[0], t[1], t[2] as u8))
+ .collect();
+
+ let mut hasher = DefaultHasher::new();
+ for (start, end, class) in &ranges {
+ hasher.write_u32(*start);
+ hasher.write_u32(*end);
+ hasher.write_u8(*class);
+ }
+ let mut lower_sorted: Vec<(&char, &String)> = lower.iter().collect();
+ lower_sorted.sort_by_key(|(c, _)| **c);
+ for (c, repl) in lower_sorted {
+ hasher.write_u32(*c as u32);
+ hasher.write(repl.as_bytes());
+ }
+
+ Self {
+ lower,
+ class_ranges: ranges,
+ fingerprint: hasher.finish(),
+ }
+ }
+
+ /// Sigma-scan class of `c`; `CLASS_BOUNDARY` when no shipped range covers
it.
+ fn class_of(&self, c: char) -> u8 {
+ let cp = c as u32;
+ let mut lo = 0usize;
+ let mut hi = self.class_ranges.len();
+ while lo < hi {
+ let mid = (lo + hi) / 2;
+ let (start, end, class) = self.class_ranges[mid];
+ if cp < start {
+ hi = mid;
+ } else if cp > end {
+ lo = mid + 1;
+ } else {
+ return class;
+ }
+ }
+ CLASS_BOUNDARY
+ }
+
+ /// First position at or beyond `start` (stepping by `step`, i.e. -1
backward / +1
+ /// forward) whose class is not CLASS_EXTEND. Returns `None` if the scan
runs off the
+ /// array without finding one.
+ fn skip_extends(&self, cps: &[char], start: isize, step: isize) ->
Option<usize> {
+ let mut k = start;
+ while k >= 0 && (k as usize) < cps.len() {
+ if self.class_of(cps[k as usize]) != CLASS_EXTEND {
+ return Some(k as usize);
+ }
+ k += step;
+ }
+ None
+ }
+
+ /// As [`Self::skip_extends`], but also skips CLASS_EXTEND_CASED,
reporting whether one
+ /// was walked: a cased mark (U+0345) crossed while looking for a base is
itself cased
+ /// whenever the landing validates the run.
+ fn skip_extends_tracking_cased(
+ &self,
+ cps: &[char],
+ start: isize,
+ step: isize,
+ ) -> (Option<usize>, bool) {
+ let mut k = start;
+ let mut saw_cased = false;
+ while k >= 0 && (k as usize) < cps.len() {
+ let cls = self.class_of(cps[k as usize]);
+ if cls != CLASS_EXTEND && cls != CLASS_EXTEND_CASED {
+ return (Some(k as usize), saw_cased);
+ }
+ if cls == CLASS_EXTEND_CASED {
+ saw_cased = true;
+ }
+ k += step;
+ }
+ (None, saw_cased)
+ }
+
+ /// What the supplementary-mark run at `k` (CLASS_SUPP_MN) ultimately
hangs off, walking
+ /// down through further marks and supplementary chars: a letter-flavored
base, a
+ /// digit-flavored base, or nothing word-forming. A cased mark riding the
run belongs to
+ /// the sigma's word only per this anchor.
+ fn supp_mn_anchor(&self, cps: &[char], k: usize) -> SuppMnAnchor {
+ let mut m = k as isize - 1;
+ while m >= 0 {
+ let cls = self.class_of(cps[m as usize]);
+ if cls != CLASS_SUPP_MN && cls != CLASS_EXTEND && cls !=
CLASS_EXTEND_CASED {
+ break;
+ }
+ m -= 1;
+ }
+ if m < 0 {
+ return SuppMnAnchor::None;
+ }
+ match self.class_of(cps[m as usize]) {
+ CLASS_ALETTER_CASED | CLASS_ALETTER | CLASS_SUPP_CASED |
CLASS_SUPP_LETTER => {
+ SuppMnAnchor::Letter
+ }
+ CLASS_NUMERIC | CLASS_NUMERIC_CASED | CLASS_SUPP_NUM =>
SuppMnAnchor::Digit,
+ _ => SuppMnAnchor::None,
+ }
+ }
+
+ /// Backward half of the ported `isFinalCased`: is there a cased letter
before position
+ /// `i` within the sigma's word? Runs over the FORMAT-FILTERED sequence;
`leading_format`
+ /// says whether format chars were filtered off the raw text start.
+ fn scan_back_finds_cased(&self, cps: &[char], i: usize, leading_format:
bool) -> bool {
+ let mut last_letter = true; // the sigma itself is a letter
+ let mut j = i as isize - 1;
+ while j >= 0 {
+ match self.class_of(cps[j as usize]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j -= 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j -= 1;
+ }
+ CLASS_EXTEND => {
+ // Non-cased marks attach only to a real base below them;
anything else
+ // (mid punctuation, danda, boundary, text start) leaves
the run
+ // unattached.
+ let Some(k) = self.skip_extends(cps, j, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM;
+ if !is_continuer {
+ return false;
+ }
+ j = k as isize;
+ }
+ CLASS_SUPP_CASED => {
+ // Closes the preceding word, so the scan stops -- except
at RAW text
+ // start (no filtered-out leading format chars), where the
DFA keeps it
+ // joined to what follows.
+ return j == 0 && !leading_format;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach/close and never themselves cased; nothing beyond
is reachable.
+ return false;
+ }
+ CLASS_EXTEND_CASED => {
+ // Cased combining mark (U+0345): cased when its run hangs
off a base --
+ // a BMP letter/digit, a word-forming supplementary char
(which closes a
+ // word right below the mark, merging the mark into the
sigma's
+ // segment), or an ANCHORED supplementary mark.
+ let (Some(k), _) = self.skip_extends_tracking_cased(cps, j
- 1, -1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_NUM
+ {
+ return true;
+ }
+ if b == CLASS_SUPP_MN {
+ return self.supp_mn_anchor(cps, k) !=
SuppMnAnchor::None;
+ }
+ return false;
+ }
+ CLASS_DANDA => {
+ // Backward across a danda: the word part before it must
end in letters
+ // (grammar: letters, optional danda, then number+word
chains) -- or
+ // carry a riding cased mark on a word-forming base, or be
a cased
+ // supplementary char at text start -- and the danda
itself chains only
+ // into digits after it.
+ if last_letter {
+ return false;
+ }
+ let (Some(k), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ if b == CLASS_ALETTER_CASED {
+ return true;
+ }
+ if b == CLASS_SUPP_CASED {
+ return saw_cased_mark || (k == 0 && !leading_format);
+ }
+ if b == CLASS_SUPP_LETTER {
+ return saw_cased_mark;
+ }
+ if b == CLASS_SUPP_MN {
+ return saw_cased_mark
+ && self.supp_mn_anchor(cps, k) ==
SuppMnAnchor::Letter;
+ }
+ if b != CLASS_ALETTER {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ last_letter = true;
+ j = k as isize;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET)
=> {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a
genuine
+ // letter/digit base before the punctuation; scanning
backward
+ // legitimately walks marks-then-base (marks trail their
base). A cased
+ // mark walked over rides whatever the punctuation hangs
off, including
+ // a context-matching anchored supplementary mark or
supplementary
+ // digit.
+ let mw_ok = cls == CLASS_MID_LETTER || cls ==
CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls ==
CLASS_MID_NUM_LET;
+ let (Some(real_pos), saw_cased_mark) =
+ self.skip_extends_tracking_cased(cps, j - 1, -1)
+ else {
+ return false;
+ };
+ let b = self.class_of(cps[real_pos]);
+ if last_letter
+ && mw_ok
+ && saw_cased_mark
+ && b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) ==
SuppMnAnchor::Letter
+ {
+ return true;
+ }
+ if !last_letter
+ && mn_ok
+ && saw_cased_mark
+ && (b == CLASS_SUPP_NUM
+ || (b == CLASS_SUPP_MN
+ && self.supp_mn_anchor(cps, real_pos) ==
SuppMnAnchor::Digit))
+ {
+ return true;
+ }
+ let bridge_valid = (last_letter && mw_ok &&
is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b));
+ if !bridge_valid {
+ return false;
+ }
+ if saw_cased_mark {
+ return true;
+ }
+ j = real_pos as isize;
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Forward half of the ported `isFinalCased`: is there a cased letter
after position `i`
+ /// within the sigma's word? Runs over the FORMAT-FILTERED sequence.
+ fn scan_fwd_finds_cased(&self, cps: &[char], i: usize) -> bool {
+ let mut last_letter = true;
+ let mut j = i + 1;
+ while j < cps.len() {
+ match self.class_of(cps[j]) {
+ CLASS_ALETTER_CASED | CLASS_NUMERIC_CASED => return true,
+ CLASS_ALETTER => {
+ last_letter = true;
+ j += 1;
+ }
+ CLASS_NUMERIC => {
+ last_letter = false;
+ j += 1;
+ }
+ CLASS_EXTEND => {
+ // A mark run trailing the anchor is properly attached in
text order, so
+ // the run stays open past it, including onto mid
punctuation on its far
+ // side.
+ let Some(k) = self.skip_extends(cps, j as isize, 1) else {
+ return false;
+ };
+ let b = self.class_of(cps[k]);
+ let is_continuer = b == CLASS_ALETTER_CASED
+ || b == CLASS_NUMERIC_CASED
+ || b == CLASS_NUMERIC
+ || b == CLASS_EXTEND_CASED
+ || b == CLASS_ALETTER
+ || b == CLASS_SUPP_CASED
+ || b == CLASS_SUPP_LETTER
+ || b == CLASS_SUPP_MN
+ || b == CLASS_SUPP_NUM
+ || b == CLASS_DANDA
+ || b == CLASS_MID_LETTER
+ || b == CLASS_MID_NUM
+ || b == CLASS_MID_NUM_LET;
+ if !is_continuer {
+ return false;
+ }
+ j = k;
+ }
+ CLASS_SUPP_CASED | CLASS_EXTEND_CASED => {
+ // Attaches to the current word, so the scan sees it
(cased).
+ return true;
+ }
+ CLASS_SUPP_LETTER | CLASS_SUPP_MN | CLASS_SUPP_NUM => {
+ // Attach to the current word and close it; never
themselves cased, and
+ // nothing beyond is reachable.
+ return false;
+ }
+ // The danda attaches only to a word part that ends in letters
(reached
+ // after digits the word is already closed) and continues only
into a digit
+ // -- unless that digit is itself cased (a Roman numeral),
which resolves
+ // the scan immediately.
+ CLASS_DANDA if !last_letter => return false,
+ CLASS_DANDA
+ if j + 1 < cps.len() && self.class_of(cps[j + 1]) ==
CLASS_NUMERIC_CASED =>
+ {
+ return true;
+ }
+ CLASS_DANDA if j + 1 < cps.len() && self.class_of(cps[j + 1])
== CLASS_NUMERIC => {
+ last_letter = false;
+ j += 2;
+ }
+ cls @ (CLASS_MID_LETTER | CLASS_MID_NUM | CLASS_MID_NUM_LET)
=> {
+ // `<mid-letter><let>` / `<mid-num><digit>` require a
genuine
+ // letter/digit base IMMEDIATELY after the punctuation --
unlike the
+ // backward scan, marks here are never skipped past: a
mark directly
+ // after the punctuation is attached to the punctuation,
not a base, so
+ // it blocks the bridge. (Format chars are already
filtered out, which
+ // is what lets "AΣ-<ZWJ>b" bridge exactly like "AΣ-b".)
+ let mw_ok = cls == CLASS_MID_LETTER || cls ==
CLASS_MID_NUM_LET;
+ let mn_ok = cls == CLASS_MID_NUM || cls ==
CLASS_MID_NUM_LET;
+ if j + 1 >= cps.len() {
+ return false;
+ }
+ let b = self.class_of(cps[j + 1]);
+ if (last_letter && mw_ok && is_letter_base(b))
+ || (!last_letter && mn_ok && is_digit_base(b))
+ {
+ j += 1;
+ } else {
+ return false;
+ }
+ }
+ _ => return false,
+ }
+ }
+ false
+ }
+
+ /// Lowercase `s` the way the planning JVM's
`String.toLowerCase(Locale.ROOT)` does.
+ pub fn lowercase(&self, s: &str) -> String {
+ let raw: Vec<char> = s.chars().collect();
+ // Built lazily on the first sigma: the format-filtered sequence the
scans run over
+ // (WB4-style: the legacy break iterator's `<ignore>` class loops on
every DFA
+ // state), the raw->filtered index map, and whether format chars led
the raw text.
+ let mut filtered: Option<(Vec<char>, Vec<usize>, bool)> = None;
+ let mut out = String::with_capacity(s.len());
+ for (i, &c) in raw.iter().enumerate() {
+ if c == CAPITAL_SIGMA {
+ // The condition consults the ORIGINAL neighbors, exactly as
the JDK scans
+ // `src`, not the partially-lowered output. The
final/non-final target chars
+ // are Unicode-stable (pinned in `ConditionalSpecialCasing`'s
entry table).
+ let (f, idx, leading_format) = filtered.get_or_insert_with(|| {
+ let mut f = Vec::with_capacity(raw.len());
+ let mut idx = vec![0usize; raw.len()];
+ for (k, &rc) in raw.iter().enumerate() {
+ idx[k] = f.len();
+ if self.class_of(rc) != CLASS_FORMAT {
+ f.push(rc);
+ }
+ }
+ let leading_format = self.class_of(raw[0]) == CLASS_FORMAT;
+ (f, idx, leading_format)
+ });
+ let fi = idx[i];
+ let is_final = self.scan_back_finds_cased(f, fi,
*leading_format)
+ && !self.scan_fwd_finds_cased(f, fi);
+ out.push(if is_final {
+ SMALL_FINAL_SIGMA
+ } else {
+ SMALL_SIGMA
+ });
+ } else if let Some(repl) = self.lower.get(&c) {
+ out.push_str(repl);
+ } else {
+ out.push(c);
+ }
+ }
+ out
+ }
+}
+
+/// Lowercase `s` for case-insensitive field matching. With tables (populated
whenever
+/// `case_sensitive = false`, shared by the core Parquet scan and the Delta
contrib scan) this
+/// reproduces the planning JVM's `String.toLowerCase(Locale.ROOT)` exactly.
+///
+/// Without tables, fall back to Rust's `str::to_lowercase`, which agrees with
Java on all
+/// simple mappings and differs only where the two Unicode snapshots or the
sigma context
+/// diverge. This is a real, live path, not just a defensive default: the
Iceberg native scan
+/// (`SparkPhysicalExprAdapterFactory::new(_, None)`) defaults `case_sensitive
= false` and
+/// always reaches this fallback for its schema name remap, alongside
+/// `parquet_convert_struct_to_struct`'s general struct-cast matching and
Rust-only unit tests
+/// that construct `SparkParquetOptions` directly.
+pub(crate) fn java_lowercase(s: &str, tables: Option<&JvmCaseTables>) ->
String {
Review Comment:
[P2] Thanks for measuring this. I confirmed the shared Parquet path at
`020f0399` and ran an allocation-only probe of the exact matcher with JDK 21
ASCII mappings. A search across 200 names made 20,100 comparisons and 160,800
allocation or reallocation calls, versus zero with a fast path guarded by
`a.is_ascii() && b.is_ascii()`. This was not a Parquet or Spark timing run.
I agree this should be addressed before merge while retaining JVM
lowercasing whenever either name is non-ASCII. One qualification on frequency:
DataFusion skips remapping when the schemas are identical and there is no
predicate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
