longvu-db commented on code in PR #56705: URL: https://github.com/apache/spark/pull/56705#discussion_r3527571154
########## sql/core/src/test/scala/org/apache/spark/sql/connector/ReplaceUsingTableSuiteBase.scala: ########## @@ -0,0 +1,545 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.spark.sql.connector + +import org.apache.spark.internal.config +import org.apache.spark.sql.{AnalysisException, Row} +import org.apache.spark.sql.connector.catalog.{InMemoryRowLevelOperationTable, InMemoryTable} +import org.apache.spark.sql.connector.write.{BatchWrite, ReplaceSummary, ReplaceSummaryImpl, WriterCommitMessage, WriteSummary} +import org.apache.spark.sql.execution.SparkPlan +import org.apache.spark.sql.execution.datasources.v2.{ReplaceDataExec, WriteDeltaExec} +import org.apache.spark.sql.execution.metric.SQLMetric +import org.apache.spark.sql.internal.SQLConf +import org.apache.spark.sql.types.StructType + +/** + * Shared end-to-end coverage for `INSERT INTO ... REPLACE USING` execution. + * + * Scoped replace deletes every target row whose scope-column tuple appears in the source and + * appends all source rows. Concrete subclasses fix the table layout to a group-based + * (copy-on-write) or delta-based (merge-on-read) row-level operation. Assertions are written + * against final table contents so they hold for both backends. + */ +abstract class ReplaceUsingTableSuiteBase extends RowLevelOperationSuiteBase { + + import testImplicits._ + + private val schemaString = "pk INT NOT NULL, id INT, dep STRING" + + protected def isDeltaBasedReplace: Boolean = false + + // Select the in-memory table subclass that implements SupportsRowLevelReplace via props. + override protected def extraTableProps: java.util.Map[String, String] = { + val props = new java.util.HashMap[String, String]() + props.put(InMemoryRowLevelOperationTable.SUPPORTS_ROW_LEVEL_REPLACE, "true") + props + } + + test("replace using deletes matching scopes and inserts all source rows") { Review Comment: We should add the following tests: 1. Self-INSERTing: INSERT INTO t REPLACE USING ... SELECT ... FROM t 2. Non-Deterministic formats such as Parquet, CSV,... 3. INSERT REPLACE USING analysis Exceptions: Not enough source query columns compared to table columns, too many source query columns than table columns, empty REPLACE USING clause,... 4. INSERT adjacent features: WITH SCHEMA EVOLUTION, constraint check, BY NAME (matching on a column in a different position in the table and the source query 5. Matching on multiple columns, where there exists one that carries the NULL value, if a tuple has a NULL, then it shouldn't match. 6. INSERT REPLACE USING on a partitioned table, behavior should be similar to DPO, except not matching on NULL partitions 7. Palindrome on REPLACE USING clauses, i.e. (col1, col2) and (col2, col1) 8. Different source query shape (SELECT * FROM table), SELECT * FROM TABLE with filter, VALUES (x, y, z) (in this case the source query's columns are col1, col2, ...), source table exists but empty 9. Empty target table 10. REPLACE USING column exists in table but not in source query, exists in source query but not in table, exist in neither,... 11. INSERT REPLACE USING with `optionsClause`, should still respect `optionsClause` https://github.com/apache/spark/blob/master/sql/api/src/main/antlr4/org/apache/spark/sql/catalyst/parser/SqlBaseParser.g4#L594C82-L594C96 12. REPLACE USING with struct columns in the USING clause, top-level name, or nested level column name 13. Column in the REPLACE USING clause has different data types between the source and target. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
