[GitHub] [spark] xuechendi commented on pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format

2021-11-22 Thread GitBox
xuechendi commented on pull request #34396: URL: https://github.com/apache/spark/pull/34396#issuecomment-976099176 @BryanCutler and @HyukjinKwon, Please help me take a review, I updated this PR per our discussion last week Here is my last update I moved all arrow related conversions t

[GitHub] [spark] xuechendi commented on pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format

2021-11-01 Thread GitBox
xuechendi commented on pull request #34396: URL: https://github.com/apache/spark/pull/34396#issuecomment-957064031 @cloud-fan @viirya @sunchao , hi, all, I realized that what you guys said makes sense to me, instead of writing data to arrow like what other WritableColumnVector does, I can

[GitHub] [spark] xuechendi commented on pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format

2021-11-02 Thread GitBox
xuechendi commented on pull request #34396: URL: https://github.com/apache/spark/pull/34396#issuecomment-957064031 @cloud-fan @viirya @sunchao , hi, all, I realized that what you guys said makes sense to me, instead of writing data to arrow like what other WritableColumnVector does, I can

[GitHub] [spark] xuechendi commented on pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format

2021-11-02 Thread GitBox
xuechendi commented on pull request #34396: URL: https://github.com/apache/spark/pull/34396#issuecomment-957064031 @cloud-fan @viirya @sunchao , hi, all, I realized that what you guys said makes sense to me, instead of writing data to arrow like what other WritableColumnVector does, I can

[GitHub] [spark] xuechendi commented on pull request #34396: [SPARK-37124][SQL] Support RowToColumnarExec with Arrow format

2021-11-09 Thread GitBox
xuechendi commented on pull request #34396: URL: https://github.com/apache/spark/pull/34396#issuecomment-963903913 > @xuechendi thanks for the PR, I'd like to hear a little more about your use case especially how you will with batches serialized to byte arrays, and I'm also curious how muc