[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #2721: Add additional data types are supported in hash join

2022-06-20 Thread GitBox
liukun4515 commented on code in PR #2721: URL: https://github.com/apache/arrow-datafusion/pull/2721#discussion_r902228171 ## datafusion/core/src/physical_plan/hash_join.rs: ## @@ -1054,6 +1110,116 @@ fn equal_rows( DataType::LargeUtf8 => { equal_row

[GitHub] [arrow-datafusion] liukun4515 commented on a diff in pull request #2721: Add additional data types are supported in hash join

2022-06-20 Thread GitBox
liukun4515 commented on code in PR #2721: URL: https://github.com/apache/arrow-datafusion/pull/2721#discussion_r902228171 ## datafusion/core/src/physical_plan/hash_join.rs: ## @@ -1054,6 +1110,116 @@ fn equal_rows( DataType::LargeUtf8 => { equal_row

[GitHub] [arrow-rs] liukun4515 commented on issue #1777: support write page index

2022-06-20 Thread GitBox
liukun4515 commented on issue #1777: URL: https://github.com/apache/arrow-rs/issues/1777#issuecomment-1161331389 @tustvold I have read the write path in the https://github.com/apache/parquet-mr and am very familiar with the write path of parquet. Diff column chunk in the same row grou

[GitHub] [arrow-rs] viirya commented on pull request #1917: Fix max and min decimal for max precision

2022-06-20 Thread GitBox
viirya commented on PR #1917: URL: https://github.com/apache/arrow-rs/pull/1917#issuecomment-1161267491 Thank you @liukun4515 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment

[GitHub] [arrow-rs] viirya merged pull request #1917: Fix max and min decimal for max precision

2022-06-20 Thread GitBox
viirya merged PR #1917: URL: https://github.com/apache/arrow-rs/pull/1917 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apach

[GitHub] [arrow-rs] viirya closed issue #1916: Incorrect max and min decimals

2022-06-20 Thread GitBox
viirya closed issue #1916: Incorrect max and min decimals URL: https://github.com/apache/arrow-rs/issues/1916 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-

[GitHub] [arrow-rs] liukun4515 commented on a diff in pull request #1917: Fix max and min decimal for max precision

2022-06-20 Thread GitBox
liukun4515 commented on code in PR #1917: URL: https://github.com/apache/arrow-rs/pull/1917#discussion_r902141763 ## arrow/src/datatypes/datatype.rs: ## @@ -238,7 +238,7 @@ pub const MAX_DECIMAL_FOR_EACH_PRECISION: [i128; 38] = [ 999, 99

[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
AssHero commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r902105760 ## datafusion/sql/src/planner.rs: ## @@ -784,6 +784,9 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { let filter_expr = self.sql_to_rex(pre

[GitHub] [arrow] cyb70289 closed pull request #13393: ARROW-16849: [C++][compute] Replace If-Else branch with bitwise operators in aggregate_basic_internal

2022-06-20 Thread GitBox
cyb70289 closed pull request #13393: ARROW-16849: [C++][compute] Replace If-Else branch with bitwise operators in aggregate_basic_internal URL: https://github.com/apache/arrow/pull/13393 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to G

[GitHub] [arrow] cyb70289 commented on pull request #13393: ARROW-16849: [C++][compute] Replace If-Else branch with bitwise operators in aggregate_basic_internal

2022-06-20 Thread GitBox
cyb70289 commented on PR #13393: URL: https://github.com/apache/arrow/pull/13393#issuecomment-1161125480 Thanks @zhixingheyi-tian, welcome contributions -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
AssHero commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r902105760 ## datafusion/sql/src/planner.rs: ## @@ -784,6 +784,9 @@ impl<'a, S: ContextProvider> SqlToRel<'a, S> { let filter_expr = self.sql_to_rex(pre

[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
AssHero commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r902103709 ## datafusion/sql/src/planner.rs: ## @@ -2584,6 +2587,230 @@ fn extract_join_keys( } } +/// Recursively traversese expr, if expr returns false when +///

[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
AssHero commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r902103396 ## datafusion/sql/src/planner.rs: ## @@ -2584,6 +2587,230 @@ fn extract_join_keys( } } +/// Recursively traversese expr, if expr returns false when +///

[GitHub] [arrow-datafusion] AssHero commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
AssHero commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r902102694 ## datafusion/sql/src/planner.rs: ## @@ -2584,6 +2587,230 @@ fn extract_join_keys( } } +/// Recursively traversese expr, if expr returns false when +///

[GitHub] [arrow] zhixingheyi-tian commented on pull request #13393: ARROW-16849: [C++][compute] Replace If-Else branch with bitwise operators in aggregate_basic_internal

2022-06-20 Thread GitBox
zhixingheyi-tian commented on PR #13393: URL: https://github.com/apache/arrow/pull/13393#issuecomment-1161105860 Thanks folks‘ review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [arrow] zhixingheyi-tian commented on pull request #13393: ARROW-16849: [C++][compute] Replace If-Else branch with bitwise operators in aggregate_basic_internal

2022-06-20 Thread GitBox
zhixingheyi-tian commented on PR #13393: URL: https://github.com/apache/arrow/pull/13393#issuecomment-1161105284 > Double checked the CI error. This code is handling arrow decimal type, not native c++ numbers. Can we close this PR? @zhixingheyi-tian OK -- This is an automated messa

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902095243 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow] cyb70289 commented on pull request #13393: ARROW-16849: [C++][compute] Replace If-Else branch with bitwise operators in aggregate_basic_internal

2022-06-20 Thread GitBox
cyb70289 commented on PR #13393: URL: https://github.com/apache/arrow/pull/13393#issuecomment-1161057412 Double checked the CI error. This code is handling arrow decimal type, not native c++ numbers. Can we close this PR? @zhixingheyi-tian -- This is an automated message from the Apac

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902082472 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902079100 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902067672 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902067086 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902065911 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902065369 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902064862 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902064399 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902063905 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902063779 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902063056 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r902062655 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] adzcai opened a new issue, #13405: [Java] Loading complex types from a ArrowRecordBatch

2022-06-20 Thread GitBox
adzcai opened a new issue, #13405: URL: https://github.com/apache/arrow/issues/13405 Hello, I've been having some issues with loading a Parquet file with complex types into Arrow for Java, and I was hoping here would be a good place to ask. I've written the following table to a Parque

[GitHub] [arrow] vibhatha commented on pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on PR #13397: URL: https://github.com/apache/arrow/pull/13397#issuecomment-1160959973 > I probably shouldn't be turning Scalar into an Array anyway (i.e., I should probably just do what the Python UDFs do and pass a `list()` of arguments). I think we can consider a

[GitHub] [arrow] vibhatha commented on pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on PR #13397: URL: https://github.com/apache/arrow/pull/13397#issuecomment-1160958463 > All good points and thank you for taking a look! Your edit did the trick, although the segfault may have been due to something about turning a `Scalar` into an `Array`: > > ```r

[GitHub] [arrow] dragosmg commented on pull request #13196: ARROW-16407: [R] Extend `parse_date_time` to cover hour, dates, and minutes components

2022-06-20 Thread GitBox
dragosmg commented on PR #13196: URL: https://github.com/apache/arrow/pull/13196#issuecomment-1160827641 Results of benchmarking `parse_date_time()` implemented with combined formats (with and without separator) vs separate formats (either with or without separator) ```r library(dplyr

[GitHub] [arrow-rs] viirya commented on a diff in pull request #1914: Add Decimal256 API

2022-06-20 Thread GitBox
viirya commented on code in PR #1914: URL: https://github.com/apache/arrow-rs/pull/1914#discussion_r901988367 ## arrow/src/util/decimal.rs: ## @@ -17,109 +17,201 @@ //! Decimal related utils -use std::cmp::Ordering; +use crate::error::{ArrowError, Result}; +use num::bigint:

[GitHub] [arrow-rs] viirya commented on a diff in pull request #1914: Add Decimal256 API

2022-06-20 Thread GitBox
viirya commented on code in PR #1914: URL: https://github.com/apache/arrow-rs/pull/1914#discussion_r901987661 ## arrow/src/util/decimal.rs: ## @@ -17,109 +17,201 @@ //! Decimal related utils -use std::cmp::Ordering; +use crate::error::{ArrowError, Result}; +use num::bigint:

[GitHub] [arrow-rs] codecov-commenter commented on pull request #1917: Fix max and min decimal for max precision

2022-06-20 Thread GitBox
codecov-commenter commented on PR #1917: URL: https://github.com/apache/arrow-rs/pull/1917#issuecomment-1160808551 # [Codecov](https://codecov.io/gh/apache/arrow-rs/pull/1917?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+S

[GitHub] [arrow-rs] viirya commented on a diff in pull request #1917: Fix max and min decimal for max precision

2022-06-20 Thread GitBox
viirya commented on code in PR #1917: URL: https://github.com/apache/arrow-rs/pull/1917#discussion_r901965838 ## arrow/src/datatypes/datatype.rs: ## @@ -238,7 +238,7 @@ pub const MAX_DECIMAL_FOR_EACH_PRECISION: [i128; 38] = [ 999, 99

[GitHub] [arrow-rs] viirya opened a new pull request, #1917: Fix max and min decimal for max precision

2022-06-20 Thread GitBox
viirya opened a new pull request, #1917: URL: https://github.com/apache/arrow-rs/pull/1917 # Which issue does this PR close? Closes #1916. # Rationale for this change # What changes are included in this PR? # Are there any user-facing chan

[GitHub] [arrow-rs] viirya opened a new issue, #1916: Incorrect max and min decimals

2022-06-20 Thread GitBox
viirya opened a new issue, #1916: URL: https://github.com/apache/arrow-rs/issues/1916 **Describe the bug** We define `MAX_DECIMAL_FOR_EACH_PRECISION` and `MIN_DECIMAL_FOR_EACH_PRECISION` for max/min decimals per precision. But the max/min for precision 38 looks incorrect. **To

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901950361 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901938320 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow-rs] viirya commented on a diff in pull request #1914: Add Decimal256 API

2022-06-20 Thread GitBox
viirya commented on code in PR #1914: URL: https://github.com/apache/arrow-rs/pull/1914#discussion_r901905221 ## arrow/src/util/decimal.rs: ## @@ -17,109 +17,187 @@ //! Decimal related utils +use crate::error::{ArrowError, Result}; +use num::bigint::BigInt; use std::cmp::O

[GitHub] [arrow-rs] viirya commented on a diff in pull request #1914: Add Decimal256 API

2022-06-20 Thread GitBox
viirya commented on code in PR #1914: URL: https://github.com/apache/arrow-rs/pull/1914#discussion_r901903086 ## arrow/src/util/decimal.rs: ## @@ -17,109 +17,187 @@ //! Decimal related utils +use crate::error::{ArrowError, Result}; +use num::bigint::BigInt; use std::cmp::O

[GitHub] [arrow] paleolimbot commented on pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on PR #13397: URL: https://github.com/apache/arrow/pull/13397#issuecomment-1160708575 I probably shouldn't be turning Scalar into an Array anyway (i.e., I should probably just do what the Python UDFs do and pass a `list()` of arguments). -- This is an automated messa

[GitHub] [arrow] paleolimbot commented on pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on PR #13397: URL: https://github.com/apache/arrow/pull/13397#issuecomment-1160707504 All good points and thank you for taking a look! Your edit did the trick, although the segfault may have been due to something about turning a `Scalar` into an `Array`: ``` r

[GitHub] [arrow-datafusion] waynexia commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
waynexia commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r901885300 ## datafusion/sql/src/planner.rs: ## @@ -2584,6 +2587,230 @@ fn extract_join_keys( } } +/// Recursively traversese expr, if expr returns false when +//

[GitHub] [arrow-datafusion] waynexia commented on a diff in pull request #2750: try to reduce left/right/full join to inner join

2022-06-20 Thread GitBox
waynexia commented on code in PR #2750: URL: https://github.com/apache/arrow-datafusion/pull/2750#discussion_r901869070 ## datafusion/sql/src/planner.rs: ## @@ -2584,6 +2587,230 @@ fn extract_join_keys( } } +/// Recursively traversese expr, if expr returns false when +//

[GitHub] [arrow] rtpsw commented on a diff in pull request #13375: ARROW-16823: [C++] Arrow Substrait enhancements for UDF

2022-06-20 Thread GitBox
rtpsw commented on code in PR #13375: URL: https://github.com/apache/arrow/pull/13375#discussion_r901085767 ## cpp/src/arrow/engine/substrait/util.h: ## @@ -30,17 +31,45 @@ namespace substrait { /// \brief Retrieve a RecordBatchReader from a Substrait plan. ARROW_ENGINE_EXPO

[GitHub] [arrow] vibhatha commented on pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on PR #13397: URL: https://github.com/apache/arrow/pull/13397#issuecomment-1160624867 Experienced this in the interactive session. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go

[GitHub] [arrow] ArianaVillegas commented on pull request #13334: ARROW-14314: [C++] Sorting dictionary array not implemented

2022-06-20 Thread GitBox
ArianaVillegas commented on PR #13334: URL: https://github.com/apache/arrow/pull/13334#issuecomment-1160624398 @pitrou I cast index array to concrete type :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

[GitHub] [arrow] vibhatha commented on pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on PR #13397: URL: https://github.com/apache/arrow/pull/13397#issuecomment-1160611988 Another comment on the function registration is that if you try to run the test cases twice, you will get a test failure saying the function is already registered. In R, is there a way t

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901813267 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901812656 ## r/src/compute.cpp: ## @@ -574,3 +576,89 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] amol- merged pull request #13342: ARROW-16382: [Python] Disable memory mapping by default in pyarrow

2022-06-20 Thread GitBox
amol- merged PR #13342: URL: https://github.com/apache/arrow/pull/13342 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901799543 ## r/src/filesystem.cpp: ## @@ -331,3 +337,77 @@ std::string fs___S3FileSystem__region(const std::shared_ptr& f } #endif + +#if defined(ARROW_R_WITH_GCS) + +

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901798660 ## r/src/filesystem.cpp: ## @@ -33,6 +33,12 @@ const char* r6_class_name::get( return "LocalFileSystem"; } else if (type_name == "s3") { return "S3Fi

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901797516 ## r/src/arrow_types.h: ## @@ -69,6 +69,14 @@ namespace ds = ::arrow::dataset; namespace compute = ::arrow::compute; namespace fs = ::arrow::fs; +#if defined(

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901797241 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901796837 ## r/data-raw/codegen.R: ## @@ -44,7 +44,9 @@ get_exported_functions <- function(decorations, export_tag) { out <- decorations %>% filter(decoration %in%

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901795365 ## cpp/cmake_modules/ThirdpartyToolchain.cmake: ## @@ -3542,18 +3544,6 @@ macro(resolve_dependency_absl) APPEND PROPERTY I

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901794827 ## cpp/cmake_modules/ThirdpartyToolchain.cmake: ## @@ -2794,18 +2795,17 @@ macro(resolve_dependency_absl) endforeach() # Extracted the dependency info

[GitHub] [arrow] nealrichardson commented on a diff in pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson commented on code in PR #13404: URL: https://github.com/apache/arrow/pull/13404#discussion_r901794205 ## cpp/cmake_modules/ThirdpartyToolchain.cmake: ## @@ -2690,15 +2690,16 @@ macro(resolve_dependency_absl) stacktrace status statusor -

[GitHub] [arrow-ballista] andygrove commented on pull request #58: Add ballista python module

2022-06-20 Thread GitBox
andygrove commented on PR #58: URL: https://github.com/apache/arrow-ballista/pull/58#issuecomment-1160585878 Hi @Jimexist just in case you missed the previous message. Let me know what your thoughts are, -- This is an automated message from the Apache Git Service. To respond to the messag

[GitHub] [arrow] github-actions[bot] commented on pull request #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
github-actions[bot] commented on PR #13404: URL: https://github.com/apache/arrow/pull/13404#issuecomment-1160585559 https://issues.apache.org/jira/browse/ARROW-16510 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the

[GitHub] [arrow-ballista] andygrove commented on pull request #66: Update to datafusion 9.0.0

2022-06-20 Thread GitBox
andygrove commented on PR #66: URL: https://github.com/apache/arrow-ballista/pull/66#issuecomment-1160584744 Closing this since another PR went in to update to a recent version. Thanks @nl5887 -- This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [arrow] nealrichardson opened a new pull request, #13404: ARROW-16510: [R] Add bindings for GCS filesystem

2022-06-20 Thread GitBox
nealrichardson opened a new pull request, #13404: URL: https://github.com/apache/arrow/pull/13404 To do (some may be deferred to followups): - [ ] Basic R only tests (things that don't require a real or mock GCS backend) - [ ] Turn on ARROW_GCS in (some?) CI configurations - [

[GitHub] [arrow-ballista] andygrove closed pull request #66: Update to datafusion 9.0.0

2022-06-20 Thread GitBox
andygrove closed pull request #66: Update to datafusion 9.0.0 URL: https://github.com/apache/arrow-ballista/pull/66 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscri

[GitHub] [arrow] rok commented on pull request #12154: ARROW-14821: [R] Implement bindings for lubridate's floor_date, ceiling_date, and round_date

2022-06-20 Thread GitBox
rok commented on PR #12154: URL: https://github.com/apache/arrow/pull/12154#issuecomment-1160558701 @djnavarro do you mind if I resolve the merge conflicts? :) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL ab

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901756134 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901754257 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901738163 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow] vibhatha commented on a diff in pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13401: URL: https://github.com/apache/arrow/pull/13401#discussion_r901731391 ## cpp/src/arrow/engine/substrait/serde_test.cc: ## @@ -1173,5 +1175,168 @@ TEST(Substrait, JoinPlanInvalidKeys) { &ext_set)); } +TEST(Substrait, Serializ

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901727265 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901726142 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow] LinGeLin opened a new issue, #13403: How to speed up arrow's reading of S3 Parquet files?

2022-06-20 Thread GitBox
LinGeLin opened a new issue, #13403: URL: https://github.com/apache/arrow/issues/13403 I am using Arrow to do tensorflow dataset for training. Structured data is stored on S3 as parquet files. I used Arrow to construct a TFIO dataset, but after the overall test, I found that the reading spe

[GitHub] [arrow] vibhatha commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901703174 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames()

[GitHub] [arrow-datafusion] andygrove opened a new pull request, #2754: WIP: Implement extensible configuration mechanism

2022-06-20 Thread GitBox
andygrove opened a new pull request, #2754: URL: https://github.com/apache/arrow-datafusion/pull/2754 # Which issue does this PR close? Closes https://github.com/apache/arrow-datafusion/issues/138 # Rationale for this change We are getting to the point where

[GitHub] [arrow] lidavidm commented on a diff in pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
lidavidm commented on code in PR #13401: URL: https://github.com/apache/arrow/pull/13401#discussion_r901699154 ## cpp/src/arrow/engine/substrait/serde_test.cc: ## @@ -1173,5 +1175,168 @@ TEST(Substrait, JoinPlanInvalidKeys) { &ext_set)); } +TEST(Substrait, Serializ

[GitHub] [arrow] vibhatha commented on a diff in pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
vibhatha commented on code in PR #13401: URL: https://github.com/apache/arrow/pull/13401#discussion_r901698427 ## cpp/src/arrow/engine/substrait/serde_test.cc: ## @@ -1173,5 +1175,168 @@ TEST(Substrait, JoinPlanInvalidKeys) { &ext_set)); } +TEST(Substrait, Serializ

[GitHub] [arrow] vibhatha commented on pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
vibhatha commented on PR #13401: URL: https://github.com/apache/arrow/pull/13401#issuecomment-1160488517 > > Do you think it’s wise to add a CI to test Substrait related queries using this tool? > > IMO every roundtripped plan in every Substrait consumer and/or producer should also b

[GitHub] [arrow-datafusion] xudong963 merged pull request #2753: [minor] refine doc

2022-06-20 Thread GitBox
xudong963 merged PR #2753: URL: https://github.com/apache/arrow-datafusion/pull/2753 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@

[GitHub] [arrow-datafusion] xudong963 commented on pull request #2753: [minor] refine doc

2022-06-20 Thread GitBox
xudong963 commented on PR #2753: URL: https://github.com/apache/arrow-datafusion/pull/2753#issuecomment-1160460670 Thanks @Ted-Jiang -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

[GitHub] [arrow-rs] liyongjing opened a new issue, #1915: incompatible arrow schema, expected struct got List

2022-06-20 Thread GitBox
liyongjing opened a new issue, #1915: URL: https://github.com/apache/arrow-rs/issues/1915 **Describe the bug** ```rust use std::{fs::File, path::Path, sync::Arc}; use arrow::record_batch::RecordBatchReader; use parquet::{ arrow::{ArrowReader, ParquetFileArrowReader, Pro

[GitHub] [arrow] lidavidm commented on issue #13391: There is no test case using the `2.0.0-compression` test file

2022-06-20 Thread GitBox
lidavidm commented on issue #13391: URL: https://github.com/apache/arrow/issues/13391#issuecomment-1160354058 Given the Rust PR did you figure this out? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [arrow] lidavidm commented on a diff in pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
lidavidm commented on code in PR #13401: URL: https://github.com/apache/arrow/pull/13401#discussion_r901579403 ## cpp/src/arrow/engine/substrait/serde_test.cc: ## @@ -1173,5 +1175,168 @@ TEST(Substrait, JoinPlanInvalidKeys) { &ext_set)); } +TEST(Substrait, Serializ

[GitHub] [arrow] jvanstraten commented on pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
jvanstraten commented on PR #13401: URL: https://github.com/apache/arrow/pull/13401#issuecomment-1160287804 > Do you think it’s wise to add a CI to test Substrait related queries using this tool? IMO every roundtripped plan in every Substrait consumer and/or producer should also be p

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901502171 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow] paleolimbot commented on a diff in pull request #13397: ARROW-16444: [R] Implement user-defined scalar functions in R bindings

2022-06-20 Thread GitBox
paleolimbot commented on code in PR #13397: URL: https://github.com/apache/arrow/pull/13397#discussion_r901497172 ## r/src/compute.cpp: ## @@ -574,3 +576,90 @@ SEXP compute__CallFunction(std::string func_name, cpp11::list args, cpp11::list std::vector compute__GetFunctionNames

[GitHub] [arrow] vibhatha commented on pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
vibhatha commented on PR #13401: URL: https://github.com/apache/arrow/pull/13401#issuecomment-1160235282 > I don't feel qualified to comment on those design questions, but FWIW, I ran the serialized output of the test case through the validator and it's okay-ish (the validator doesn't like

[GitHub] [arrow] jvanstraten commented on pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
jvanstraten commented on PR #13401: URL: https://github.com/apache/arrow/pull/13401#issuecomment-1160229357 I don't feel qualified to comment on those design questions, but FWIW, I ran the serialized output of the test case through the validator and it's okay-ish (the validator doesn't like

[GitHub] [arrow] pitrou commented on a diff in pull request #12914: ARROW-2034: [C++] Filesystem implementation for Azure Blob Storage

2022-06-20 Thread GitBox
pitrou commented on code in PR #12914: URL: https://github.com/apache/arrow/pull/12914#discussion_r901392113 ## cpp/cmake_modules/SetupCxxFlags.cmake: ## @@ -118,12 +118,16 @@ if(NOT DEFINED CMAKE_C_STANDARD) set(CMAKE_C_STANDARD 11) endif() -# This ensures that things lik

[GitHub] [arrow] pitrou commented on a diff in pull request #12914: ARROW-2034: [C++] Filesystem implementation for Azure Blob Storage

2022-06-20 Thread GitBox
pitrou commented on code in PR #12914: URL: https://github.com/apache/arrow/pull/12914#discussion_r901391885 ## cpp/cmake_modules/SetupCxxFlags.cmake: ## @@ -118,12 +118,16 @@ if(NOT DEFINED CMAKE_C_STANDARD) set(CMAKE_C_STANDARD 11) endif() -# This ensures that things lik

[GitHub] [arrow] kou commented on a diff in pull request #12914: ARROW-2034: [C++] Filesystem implementation for Azure Blob Storage

2022-06-20 Thread GitBox
kou commented on code in PR #12914: URL: https://github.com/apache/arrow/pull/12914#discussion_r901364218 ## cpp/cmake_modules/SetupCxxFlags.cmake: ## @@ -118,12 +118,16 @@ if(NOT DEFINED CMAKE_C_STANDARD) set(CMAKE_C_STANDARD 11) endif() -# This ensures that things like c

[GitHub] [arrow] vibhatha commented on pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
vibhatha commented on PR #13401: URL: https://github.com/apache/arrow/pull/13401#issuecomment-1160100835 > I'm out Monday & Tuesday. Maybe @jvanstraten can take a look? Otherwise I can get to this on Wednesday Wednesday works for me 👍 -- This is an automated message from the Apach

[GitHub] [arrow] pitrou commented on a diff in pull request #12914: ARROW-2034: [C++] Filesystem implementation for Azure Blob Storage

2022-06-20 Thread GitBox
pitrou commented on code in PR #12914: URL: https://github.com/apache/arrow/pull/12914#discussion_r901358157 ## cpp/cmake_modules/SetupCxxFlags.cmake: ## @@ -118,12 +118,16 @@ if(NOT DEFINED CMAKE_C_STANDARD) set(CMAKE_C_STANDARD 11) endif() -# This ensures that things lik

[GitHub] [arrow] westonpace commented on pull request #13401: ARROW-16855: [C++] Adding Read Relation ToProto

2022-06-20 Thread GitBox
westonpace commented on PR #13401: URL: https://github.com/apache/arrow/pull/13401#issuecomment-1160081211 I'm out Monday & Tuesday. Maybe @jvanstraten can take a look? Otherwise I can get to this on Wednesday -- This is an automated message from the Apache Git Service. To respond to t