NJCrazyRen opened a new issue, #16424: URL: https://github.com/apache/shardingsphere/issues/16424
The question is that I have a MySQL database and there are some columns storing encrypted data, but I need plain text data in my spark application. So I tried to use ShardingSphere-proxy with correct encrypt config as middleware between my apache spark application and MySQL. It works well when I use DBeaver to execute query, I got the plain data. But the exception below occured in my spark application: requirement failed: Decimal precision 5 exceeds max precision 3 The real column data type in MySQL is decimal(5, 2), but spark got decimal(3, 2) from ShardingSphere-proxy. In fact, not only one column data type is incorrect, the data types of other decimal columns are also incorrect. I found the similarity is the precision got from ShardingSphere-proxy be 2 less than the correct precision in database. According to the spark source code, spark build query sql like 'select * from table where 1=0' to execute and get schema from the ResultSet with 'getMetaData' method. So I found this problem. I tried to change my spark application code, connecting to database directly for getting right schema, passing it to spark DataFrameReader with JDBC format for workaround. Although it can avoid the exception, but it's too much trouble. The software version in my case: Spark 2.3.2, MySQL 5.7.33, ShardingSphere-proxy 5.1.0, mysql-connector-java-8.0.26 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
