[spark] branch master updated: [SPARK-38522][SS] Enrich the method contract of iterator in StateStore to not expect strong consistency on certain condition

[kabhwan]

This is an automated email from the ASF dual-hosted git repository.

kabhwan pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new c309cd1  [SPARK-38522][SS] Enrich the method contract of iterator in 
StateStore to not expect strong consistency on certain condition
c309cd1 is described below

commit c309cd136a04d831872a83ad425cacc1c557aa60
Author: Jungtaek Lim <kabhwan.opensou...@gmail.com>
AuthorDate: Wed Mar 23 10:09:06 2022 +0900

    [SPARK-38522][SS] Enrich the method contract of iterator in StateStore to 
not expect strong consistency on certain condition
    
    ### What changes were proposed in this pull request?
    
    This PR proposes to enrich the method contract of iterator in StateStore, 
that the returned iterator is not guaranteed to reflect all updates being 
performed "after" it has been initialized.
    
    ### Why are the changes needed?
    
    The lack of information on the method contract could mislead callers to 
expect some guarantees although not described. It would be nice to add to the 
contract to make clear. It could also help to implementations that they are not 
required to guarantee such thing.
    
    ### Does this PR introduce _any_ user-facing change?
    
    No
    
    ### How was this patch tested?
    
    N/A
    
    Closes #35816 from HeartSaVioR/SPARK-38522.
    
    Authored-by: Jungtaek Lim <kabhwan.opensou...@gmail.com>
    Signed-off-by: Jungtaek Lim <kabhwan.opensou...@gmail.com>
---
 .../org/apache/spark/sql/execution/streaming/state/StateStore.scala   | 4 ++++
 1 file changed, 4 insertions(+)

diff --git 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
index 5020638..64c9cd1 100644
--- 
a/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
+++ 
b/sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/state/StateStore.scala
@@ -127,6 +127,10 @@ trait StateStore extends ReadStateStore {
   /**
    * Return an iterator containing all the key-value pairs in the StateStore. 
Implementations must
    * ensure that updates (puts, removes) can be made while iterating over this 
iterator.
+   *
+   * It is not required for implementations to ensure the iterator reflects 
all updates being
+   * performed after initialization of the iterator. Callers should perform 
all updates before
+   * calling this method if all updates should be visible in the returned 
iterator.
    */
   override def iterator(): Iterator[UnsafeRowPair]
 

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org