Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-15 Thread via GitHub
yaooqinn commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2113991332 Thank you @xuzifu666. Merged to master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-15 Thread via GitHub
yaooqinn closed pull request #46509: [SPARK-48219][CORE] StreamReader Charset fix with UTF8 URL: https://github.com/apache/spark/pull/46509 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-15 Thread via GitHub
xuzifu666 commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2113986338 XSDtoSchema would not modify it, than HiveImpl had also changed can refer recent pr: https://github.com/apache/hive/pull/5243,so I Think it is nesscery to change it? @yaooqinn

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-15 Thread via GitHub
xuzifu666 commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1602578323 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala: ## @@ -48,7 +49,7 @@ object XSDToSchema extends Logging{ val in =

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-15 Thread via GitHub
xuzifu666 commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1602578323 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala: ## @@ -48,7 +49,7 @@ object XSDToSchema extends Logging{ val in =

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
HyukjinKwon commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596440366 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala: ## @@ -48,7 +49,7 @@ object XSDToSchema extends Logging{ val in =

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
xuzifu666 commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596437790 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala: ## @@ -48,7 +49,7 @@ object XSDToSchema extends Logging{ val in =

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
xuzifu666 commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596437790 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala: ## @@ -48,7 +49,7 @@ object XSDToSchema extends Logging{ val in =

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
HyukjinKwon commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596436036 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/xml/XSDToSchema.scala: ## @@ -48,7 +49,7 @@ object XSDToSchema extends Logging{ val in =

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
xuzifu666 commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596434880 ## sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java: ## @@ -171,7 +172,7 @@ protected BufferedReader loadFile(String

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
yaooqinn commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596434063 ## sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java: ## @@ -171,7 +172,7 @@ protected BufferedReader loadFile(String

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
HyukjinKwon commented on code in PR #46509: URL: https://github.com/apache/spark/pull/46509#discussion_r1596434994 ## sql/hive-thriftserver/src/main/java/org/apache/hive/service/cli/session/HiveSessionImpl.java: ## @@ -171,7 +172,7 @@ protected BufferedReader loadFile(String

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
yaooqinn commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2104161932 The change itself looks reasonable to me. I also agree with @dongjoon-hyun that we shall add a simple test, maybe in `XSDToSchemaSuite`. BTW, the PR is tagged as CORE but the

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-10 Thread via GitHub
xuzifu666 commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2103937351 @HyukjinKwon could you help to give a review? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-09 Thread via GitHub
dongjoon-hyun commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2103908417 Sorry but I'll leave this to the other reviewers, @xuzifu666 . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-09 Thread via GitHub
xuzifu666 commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2103906360 @dongjoon-hyun Could you give a final review? Thanks -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] [SPARK-48219][CORE] StreamReader Charset fix with UTF8 [spark]

2024-05-09 Thread via GitHub
xuzifu666 commented on PR #46509: URL: https://github.com/apache/spark/pull/46509#issuecomment-2103709532 > Do you think you can provide a test coverage to protect your contribution from potential future regression, @xuzifu666 ? > > > Not need @dongjoon-hyun Thanks for you

[PR] [SPARK-48219][core] StreamReader Charset fix with UTF8 [spark]

2024-05-09 Thread via GitHub
xuzifu666 opened a new pull request, #46509: URL: https://github.com/apache/spark/pull/46509 ### What changes were proposed in this pull request? Fix some StreamReader not set with UTF8 ### Why are the changes needed? May cause string decode not as expected