[spark] branch master updated: [SPARK-43410][SQL] Improve vectorized loop for Packed skipValues

[sunchao]

This is an automated email from the ASF dual-hosted git repository.

sunchao pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new fd9b8a094b0 [SPARK-43410][SQL] Improve vectorized loop for Packed 
skipValues
fd9b8a094b0 is described below

commit fd9b8a094b02b958817a6e2a924ebd5df24f7fb4
Author: ClownXC <chen3494...@163.com>
AuthorDate: Mon May 8 10:06:59 2023 -0700

    [SPARK-43410][SQL] Improve vectorized loop for Packed skipValues
    
    ### What changes were proposed in this pull request?
    Improve vectorized loop for Packed skipValues
    
    ### Why are the changes needed?
    Performance improvement
    
    ### Does this PR introduce _any_ user-facing change?
    no
    
    ### How was this patch tested?
    Existing tests
    
    Closes #41092 from clownxc/clownxc-spark-43410.
    
    Authored-by: ClownXC <chen3494...@163.com>
    Signed-off-by: Chao Sun <sunc...@apple.com>
---
 .../sql/execution/datasources/parquet/VectorizedRleValuesReader.java  | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
index 12fe5697954..41ce02e3c02 100644
--- 
a/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
+++ 
b/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/VectorizedRleValuesReader.java
@@ -673,12 +673,14 @@ public final class VectorizedRleValuesReader extends 
ValuesReader
           }
           break;
         case PACKED:
+          int totalSkipNum = 0;
           for (int i = 0; i < num; ++i) {
             // Same as above, only skip non-null values from `valuesReader`
             if (currentBuffer[currentBufferIdx++] == state.maxDefinitionLevel) 
{
-              updater.skipValues(1, valuesReader);
+              ++totalSkipNum;
             }
           }
+          updater.skipValues(totalSkipNum, valuesReader);
           break;
       }
       currentCount -= num;


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org