andygrove commented on code in PR #372:
URL: https://github.com/apache/datafusion-spark/pull/372#discussion_r4628748579


##########
crates/spark/src/function/url/url_decode.rs:
##########
@@ -79,23 +79,21 @@ impl UrlDecode {
     ///
     /// * `Ok(Cow<str>)` - The decoded string, borrowed from `value` when there
     ///   was nothing to rewrite and owned otherwise
-    /// * `Err(DataFusionError)` - If the input is malformed or contains 
invalid UTF-8
+    /// * `Err(DataFusionError)` - If the input is malformed
     fn decode(value: &str) -> Result<Cow<'_, str>> {
         // Check if the string has valid percent encoding
         Self::validate_percent_encoding(value)?;
 
-        match Self::replace_plus(value.as_bytes()) {
+        // Bytes that are not valid UTF-8 become U+FFFD, like Java's 
`URLDecoder`
+        Ok(match Self::replace_plus(value.as_bytes()) {
             // No '+' was rewritten, so the decode can borrow from `value` 
itself.
-            Cow::Borrowed(bytes) => percent_decode(bytes)
-                .decode_utf8()
-                .map_err(|e| exec_datafusion_err!("Invalid UTF-8 sequence: 
{e}")),
+            Cow::Borrowed(bytes) => percent_decode(bytes).decode_utf8_lossy(),

Review Comment:
   Thanks for picking this up, the change is nice and contained. I compared 
this against PySpark 4.2.0 and it matches for overlong forms, truncated 
sequences, out-of-range bytes and stray continuation bytes. The one place it 
differs is UTF-8-encoded surrogates. Java's decoder replaces `ED` followed by 
`A0..BF` (and the rest of that sequence, if present) with a single U+FFFD. 
`from_utf8_lossy` gives one per byte. For example, 
`hex(url_decode('%ED%A0%80'))` is `EFBFBD` on Spark and `EFBFBDEFBFBDEFBFBD` 
here. `%ED%A0` gives `EFBFBD` on Spark and two here. A CESU-8 pair like 
`%ED%A0%80%ED%B0%80` gives two on Spark and six here. Could the decode handle 
that case? Walking `[u8]::utf8_chunks()` and folding an `ED` + `A0..BF` prefix 
(up to three bytes) into one U+FFFD should be enough. Everything else can stay 
as `decode_utf8_lossy` does it.



##########
crates/sqllogictest/test_files/url/url_decode.slt:
##########
@@ -53,7 +53,15 @@ Spark SQL!
 query error DataFusion error: Execution error: Invalid percent\-encoding: 
invalid hex sequence '%3%' at position 5
 SELECT url_decode('https%3%2F%2Fspark.apache.org'::string);
 
-# Bytes that are not valid UTF-8 are an error
-# xfail spark: Spark replaces bytes that are not valid UTF-8 with U+FFFD, 
datafusion-spark raises an error or returns NULL 
(https://github.com/apache/datafusion-spark/issues/366)
-query error DataFusion error: Execution error: Invalid UTF\-8 sequence
+# Bytes that are not valid UTF-8 become U+FFFD, like Spark
+query T
 SELECT url_decode('%FF'::string);
+----
+�
+
+query T
+SELECT url_decode(column1) FROM VALUES ('a%FFb%C3%28'), ('a%C3%A9'), (NULL);

Review Comment:
   Could the U+FFFD records use `hex(...)`? A literal `�` in the expected 
output can't show how many replacement characters came back. It would also be 
good to add the surrogate cases here, like `'%ED%A0%80'` → `EFBFBD` and 
`'%ED%A0%80a'` → `EFBFBD61`, plus one in `try_url_decode.slt`. The PySpark 
runner will then check them against Spark on every run.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to