Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21948#discussion_r207094567
--- Diff:
sql/core/src/main/java/org/apache/spark/sql/sources/v2/writer/DataWriterFactory.java
---
@@ -50,4 +50,15 @@
* this ID will always be 0.
*/
DataWriter<T> createDataWriter(int partitionId, long taskId, long
epochId);
+
+ /**
+ * When true, Spark will reuse the same data object instance when
sending data to the data writer,
+ * for better performance. Data writers should carefully handle the data
objects if it's reused,
+ * e.g. do not buffer the data objects in a list. By default it returns
false for safety, data
--- End diff --
@rdblue did you hit this problem in iceberg?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
