Re: [I] [Go][Parquet] Trouble using the C++ reader to read a Parquet file written with the Go writer [arrow]

2023-11-03 Thread via GitHub
mapleFU commented on issue #38503: URL: https://github.com/apache/arrow/issues/38503#issuecomment-1793362724 ``` # parquet-reader output.parquet > /dev/null Parquet error: Malformed levels. min: 2 max: 2 out of range. Max Level: 1 ``` This is weird 😅 This means go generate le

Re: [I] [C++] arrow_reader_writer_test.cc slow to compile [arrow]

2023-11-03 Thread via GitHub
mapleFU commented on issue #28908: URL: https://github.com/apache/arrow/issues/28908#issuecomment-1793362362 I've tried to split the page index test out. But this doesn't make it compile faster :-( https://github.com/apache/arrow/commit/f69ab13516e5b7bc3e32b3a4b20a25bdb01c0aa0 --

Re: [I] [Go][Parquet] Trouble using the C++ reader to read a Parquet file written with the Go writer [arrow]

2023-11-03 Thread via GitHub
mapleFU commented on issue #38503: URL: https://github.com/apache/arrow/issues/38503#issuecomment-1793361886 Sorry for late reply. Run case in https://github.com/tschaub/parquet-issue-38503 can generate the case? -- This is an automated message from the Apache Git Service. To respond to t

[PR] Fix incorrect results in COUNT(*) queries with LIMIT [arrow-datafusion]

2023-11-03 Thread via GitHub
msirek opened a new pull request, #8049: URL: https://github.com/apache/arrow-datafusion/pull/8049 ## Which issue does this PR close? Closes #8048. ## Rationale for this change While testing #8038, I ran into some incorrect results cases in `COUNT(*)` queries from a `

[I] Incorrect results in COUNT(*) queries involving LIMIT [arrow-datafusion]

2023-11-03 Thread via GitHub
msirek opened a new issue, #8048: URL: https://github.com/apache/arrow-datafusion/issues/8048 ### Describe the bug While testing #8038, I ran into some incorrect results cases in LIMIT queries. ### To Reproduce ```sql ❯ CREATE TABLE IF NOT EXISTS t1 (a INT) AS VALUES

Re: [I] Reading parquet file behavior change from 13.0.0 to 14.0.0 [arrow]

2023-11-03 Thread via GitHub
mapleFU commented on issue #38577: URL: https://github.com/apache/arrow/issues/38577#issuecomment-1793350830 Hmmm would you mind provide the file? A bit hard to check the scanner change without the data or logging.. -- This is an automated message from the Apache Git Service. To respond t

Re: [I] Custom data types in arrow array [arrow]

2023-11-03 Thread via GitHub
kylebarron commented on issue #38559: URL: https://github.com/apache/arrow/issues/38559#issuecomment-1793312045 @achapkowski you need to register an extension type according to the above doc. For an example of a point extension type, see https://github.com/geoarrow/geoarrow-python/pull/2/fi

Re: [I] FromSql trait not implemented for types used in sqllogictest postgres_engine [arrow-datafusion]

2023-11-03 Thread via GitHub
spaydar commented on issue #8042: URL: https://github.com/apache/arrow-datafusion/issues/8042#issuecomment-1793309244 Thanks @alamb, that fixed the issue -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above t

Re: [I] FromSql trait not implemented for types used in sqllogictest postgres_engine [arrow-datafusion]

2023-11-03 Thread via GitHub
spaydar closed issue #8042: FromSql trait not implemented for types used in sqllogictest postgres_engine URL: https://github.com/apache/arrow-datafusion/issues/8042 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] GH-38315: [Dev][CI] autotune needs additional permissions to push to PR branches [arrow]

2023-11-03 Thread via GitHub
assignUser commented on PR #38523: URL: https://github.com/apache/arrow/pull/38523#issuecomment-1793305983 @jonkeane well it is closely related, if we don't manage to get the permission working (which is the case based on you findings) there will also not be a way to do the rebase and we ca

Re: [I] [EPIC] Unify Function Interface (remove `BuiltInScalarFunction`) [arrow-datafusion]

2023-11-03 Thread via GitHub
thinkharderdev commented on issue #8045: URL: https://github.com/apache/arrow-datafusion/issues/8045#issuecomment-1793267211 > > Would like to add that supporting serialization of user-defined functions would be quite nice. > > I don't understand this question @thinkharderdev 🤔 >

Re: [PR] Add kwarg to filter columns [arrow-julia]

2023-11-03 Thread via GitHub
codecov-commenter commented on PR #412: URL: https://github.com/apache/arrow-julia/pull/412#issuecomment-1793267084 ## [Codecov](https://app.codecov.io/gh/apache/arrow-julia/pull/412?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apac

Re: [PR] MINOR: [Docs][C++] Fix a typo [arrow]

2023-11-03 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #38568: URL: https://github.com/apache/arrow/pull/38568#issuecomment-1793241230 After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit d4deb36a13711bc4069750658bb8822cc9f68f86. There were no

Re: [PR] GH-38576: [Java] Change JDBC driver to optionally preserve cookies and auth tokens when getting streams [arrow]

2023-11-03 Thread via GitHub
jduo commented on PR #38580: URL: https://github.com/apache/arrow/pull/38580#issuecomment-1793237407 @lidavidm @aiguofer -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To

[PR] GH-38576: [Java] Change JDBC driver to optionally preserve cookies and auth tokens when getting streams [arrow]

2023-11-03 Thread via GitHub
jduo opened a new pull request, #38580: URL: https://github.com/apache/arrow/pull/38580 ### Rationale for this change This change restores the original behavior of transmitting existing cookies and auth tokens when getting separate streams returned by getFlightInfo after adding support

Re: [I] go/adbc/driver/flightsql: When CookiesMiddleware is enabled, DO_GET requests have a different set of cookies [arrow-adbc]

2023-11-03 Thread via GitHub
aiguofer commented on issue #1194: URL: https://github.com/apache/arrow-adbc/issues/1194#issuecomment-1793231680 Relevant conversation in the JDBC driver: https://github.com/apache/arrow/issues/38576. Maybe this should also be configurable for ADBC drivers. -- This is an automated messag

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on code in PR #38404: URL: https://github.com/apache/arrow/pull/38404#discussion_r1382270918 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/utils/AvaticaParameterBinder.java: ## @@ -0,0 +1,233 @@ +/* + * Licensed to the Apache S

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on code in PR #38404: URL: https://github.com/apache/arrow/pull/38404#discussion_r1382269082 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/converter/impl/TimestampAvaticaParameterConverter.java: ## @@ -0,0 +1,78 @@ +/* + * Lice

Re: [PR] GH-38578: [Java][FlightSQL] Remove joda usage from flight-sql library [arrow]

2023-11-03 Thread via GitHub
github-actions[bot] commented on PR #38579: URL: https://github.com/apache/arrow/pull/38579#issuecomment-1793203219 :warning: GitHub issue #38578 **has been automatically assigned in GitHub** to PR creator. -- This is an automated message from the Apache Git Service. To respond to the mes

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on code in PR #38404: URL: https://github.com/apache/arrow/pull/38404#discussion_r1382268628 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/converter/impl/BinaryAvaticaParameterConverter.java: ## @@ -0,0 +1,49 @@ +/* + * License

[PR] GH-38578: [Java][FlightSQL] Remove joda usage from flight-sql library [arrow]

2023-11-03 Thread via GitHub
laurentgo opened a new pull request, #38579: URL: https://github.com/apache/arrow/pull/38579 ### Rationale for this change [joda](https://www.joda.org/joda-time/) is a very popular date/time manipulation library for java but the introduction of `java.time` package makes it obsolete a

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on code in PR #38404: URL: https://github.com/apache/arrow/pull/38404#discussion_r1382265569 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/converter/AvaticaParameterConverter.java: ## @@ -0,0 +1,42 @@ +/* + * Licensed to the Ap

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on code in PR #38521: URL: https://github.com/apache/arrow/pull/38521#discussion_r1382263742 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/ArrowFlightSqlClientHandler.java: ## @@ -364,6 +415,31 @@ public static final cla

Re: [I] [EPIC] Unify Function Interface (remove `BuiltInScalarFunction`) [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #8045: URL: https://github.com/apache/arrow-datafusion/issues/8045#issuecomment-1793147864 > Would like to add that supporting serialization of user-defined functions would be quite nice. I don't understand this question @thinkharderdev 🤔 The current

Re: [PR] RFC: Demonstrate what a function package might look like -- encoding expressions [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8046: URL: https://github.com/apache/arrow-datafusion/pull/8046#discussion_r1382228123 ## datafusion/expr/src/built_in_function.rs: ## @@ -710,30 +704,6 @@ impl BuiltinScalarFunction { BuiltinScalarFunction::Digest => {

Re: [I] [Python][FlightRPC] Segmentation Fault when invoking authenticate concurrently over a same FlightClient [arrow]

2023-11-03 Thread via GitHub
kou commented on issue #38565: URL: https://github.com/apache/arrow/issues/38565#issuecomment-1793122641 Could you share full backtrace? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specif

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
conbench-apache-arrow[bot] commented on PR #38521: URL: https://github.com/apache/arrow/pull/38521#issuecomment-1793114233 After merging your PR, Conbench analyzed the 5 benchmarking runs that have been run so far on merge-commit 2fb7fd94f26ba733587e2ed21f6b3625c48bf736. There were no

[PR] Improve comments [arrow-datafusion]

2023-11-03 Thread via GitHub
ozankabak opened a new pull request, #8047: URL: https://github.com/apache/arrow-datafusion/pull/8047 ## Which issue does this PR close? N/A. ## Rationale for this change Improves comments for the `PartitionSearchMode` struct (make it clear it applies to both grouping an

Re: [I] [Java][FlightRPC] Add options to JDBC to reuse credentials and cookies in getStream sub-clients [arrow]

2023-11-03 Thread via GitHub
jduo commented on issue #38576: URL: https://github.com/apache/arrow/issues/38576#issuecomment-1793111538 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
jduo commented on code in PR #38521: URL: https://github.com/apache/arrow/pull/38521#discussion_r1382207549 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/ArrowFlightSqlClientHandler.java: ## @@ -364,6 +415,31 @@ public static final class B

Re: [PR] MINOR: [Docs][C++] Fix a typo [arrow]

2023-11-03 Thread via GitHub
kou merged PR #38568: URL: https://github.com/apache/arrow/pull/38568 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apache.or

[PR] RFC: Demonstrate what a function package might look like -- encoding expressions [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb opened a new pull request, #8046: URL: https://github.com/apache/arrow-datafusion/pull/8046 ## Which issue does this PR close? Builds on https://github.com/apache/arrow-datafusion/pull/8039 ## Rationale for this change This PR demonstrates what a function package A

Re: [I] [Python] pa.scalar(pd.Timedelta) with second or millisecond unit performs invalid conversion [arrow]

2023-11-03 Thread via GitHub
BenjaminHelyer commented on issue #37291: URL: https://github.com/apache/arrow/issues/37291#issuecomment-1793063511 (Posting here for reference.) From the Pandas side, it's been decided that we won't support the C APIs, since this isn't possible to support cleanly in the general case. So th

Re: [PR] GH-38570: [R] Ensure that test-nix-libs is warning free [arrow]

2023-11-03 Thread via GitHub
jonkeane commented on PR #38571: URL: https://github.com/apache/arrow/pull/38571#issuecomment-1793060272 cc @paleolimbot in case this overlaps with anything you're working on -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and

Re: [PR] GH-38255: [Go][C++] Implement Flight SQL Bulk Ingestion [arrow]

2023-11-03 Thread via GitHub
joellubi commented on code in PR #38385: URL: https://github.com/apache/arrow/pull/38385#discussion_r1382161355 ## cpp/src/arrow/flight/sql/server.h: ## @@ -85,6 +85,32 @@ struct ARROW_FLIGHT_SQL_EXPORT PreparedStatementUpdate { std::string prepared_statement_handle; }; +/

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
jduo commented on code in PR #38521: URL: https://github.com/apache/arrow/pull/38521#discussion_r1382160777 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/ArrowFlightSqlClientHandler.java: ## @@ -364,6 +415,31 @@ public static final class B

Re: [PR] GH-38255: [Go][C++] Implement Flight SQL Bulk Ingestion [arrow]

2023-11-03 Thread via GitHub
joellubi commented on code in PR #38385: URL: https://github.com/apache/arrow/pull/38385#discussion_r1382161025 ## cpp/src/arrow/flight/sql/server.h: ## @@ -85,6 +85,32 @@ struct ARROW_FLIGHT_SQL_EXPORT PreparedStatementUpdate { std::string prepared_statement_handle; }; +/

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
jduo commented on code in PR #38521: URL: https://github.com/apache/arrow/pull/38521#discussion_r1382160777 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/ArrowFlightSqlClientHandler.java: ## @@ -364,6 +415,31 @@ public static final class B

Re: [PR] GH-38255: [Go][C++] Implement Flight SQL Bulk Ingestion [arrow]

2023-11-03 Thread via GitHub
joellubi commented on code in PR #38385: URL: https://github.com/apache/arrow/pull/38385#discussion_r1382159318 ## format/FlightSql.proto: ## @@ -149,6 +149,17 @@ enum SqlInfo { */ FLIGHT_SQL_SERVER_TRANSACTION_TIMEOUT = 101; + /* + * Retrieves a boolean value indica

Re: [PR] GH-38255: [Go][C++] Implement Flight SQL Bulk Ingestion [arrow]

2023-11-03 Thread via GitHub
joellubi commented on code in PR #38385: URL: https://github.com/apache/arrow/pull/38385#discussion_r1382158963 ## cpp/src/arrow/flight/integration_tests/test_integration.cc: ## @@ -1928,6 +1929,122 @@ class FlightSqlExtensionScenario : public FlightSqlScenario { return St

Re: [I] RowSelection::intersection Produces Invalid RowSelection [arrow-rs]

2023-11-03 Thread via GitHub
alamb commented on issue #5036: URL: https://github.com/apache/arrow-rs/issues/5036#issuecomment-1793036034 For context we found this in some internal data in IOx -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL

Re: [PR] Minor: Add more documentation about Partitioning [arrow-datafusion]

2023-11-03 Thread via GitHub
comphead commented on code in PR #8022: URL: https://github.com/apache/arrow-datafusion/pull/8022#discussion_r1382155955 ## datafusion/physical-expr/src/partitioning.rs: ## @@ -15,14 +15,94 @@ // specific language governing permissions and limitations // under the License. -

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1382153486 ## java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightServer.java: ## @@ -306,28 +329,63 @@ public Builder maxInboundMessageSize(int maxMessageSize)

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1382152594 ## java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightServer.java: ## @@ -306,28 +329,63 @@ public Builder maxInboundMessageSize(int maxMessageSize)

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1382152594 ## java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightServer.java: ## @@ -306,28 +329,63 @@ public Builder maxInboundMessageSize(int maxMessageSize)

[I] RowSelection::intersection Produces Invalid RowSelection [arrow-rs]

2023-11-03 Thread via GitHub
tustvold opened a new issue, #5036: URL: https://github.com/apache/arrow-rs/issues/5036 **Describe the bug** An invariant of `RowSelection` is that it alternates select and skip, and doesn't not contain empty `RowSelector`. This is typically enforced when a RowSelection is

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1382152233 ## java/flight/flight-sql-jdbc-core/src/test/java/org/apache/arrow/driver/jdbc/ConnectionMutualTlsTest.java: ## @@ -0,0 +1,436 @@ +/* + * Licensed to the Apache Softw

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1382151693 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/utils/ClientAuthenticationUtils.java: ## @@ -235,6 +235,57 @@ public static Inpu

Re: [PR] GH-34865: [C++][Flight RPC] Add Session management messages [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on PR #34817: URL: https://github.com/apache/arrow/pull/34817#issuecomment-1793024682 > @lidavidm are we ok on the AMD64 Windows MinGW build workflow falling over on what looks like an unrelated issue? Maybe preexisting(?) > > > CMake Error at C:/msys64/ucrt64/shar

Re: [PR] GH-34865: [C++][Flight RPC] Add Session management messages [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on PR #34817: URL: https://github.com/apache/arrow/pull/34817#issuecomment-1793024229 I would be OK removing the `experimental` tag (and possibly just documenting it in the docstring instead). The Protobuf setup we have is quite fragile and it would be some significant ef

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on code in PR #38404: URL: https://github.com/apache/arrow/pull/38404#discussion_r1382142636 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/converter/impl/TimestampAvaticaParameterConverter.java: ## @@ -0,0 +1,78 @@ +/* + * Lice

Re: [PR] GH-34865: [C++][Flight RPC] Add Session management messages [arrow]

2023-11-03 Thread via GitHub
indigophox commented on PR #34817: URL: https://github.com/apache/arrow/pull/34817#issuecomment-1793020488 @lidavidm Thanks for the feedback. Do you have any thoughts on resolving the Protobuf issue with MessageOptions::experimental being set in two separate modules (Flight/FlightSql) whic

Re: [I] [EPIC] Unify Function Interface (remove `BuiltInScalarFunction`) [arrow-datafusion]

2023-11-03 Thread via GitHub
thinkharderdev commented on issue #8045: URL: https://github.com/apache/arrow-datafusion/issues/8045#issuecomment-1793012868 Would like to add that supporting serialization of user-defined functions would be quite nice. The current approach to serialization is to basically just use a strin

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
jduo commented on PR #38404: URL: https://github.com/apache/arrow/pull/38404#issuecomment-1792998584 > @lidavidm @jduo Hey all, I've spent a lot of time on this PR. It's a little frustrating that a PR that started well after mine has merged and now I have to deal with merge conflicts while

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
jduo commented on code in PR #38404: URL: https://github.com/apache/arrow/pull/38404#discussion_r1382128804 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/converter/impl/DateAvaticaParameterConverter.java: ## @@ -35,12 +35,12 @@ public DateAvatica

Re: [PR] WIP: feat: emitting partial join results in `HashJoinStream` [arrow-datafusion]

2023-11-03 Thread via GitHub
korowa commented on code in PR #8020: URL: https://github.com/apache/arrow-datafusion/pull/8020#discussion_r1382087972 ## datafusion/sqllogictest/test_files/join_disable_repartition_joins.slt: ## @@ -72,11 +72,11 @@ SELECT t1.a, t1.b, t1.c, t2.a as a2 ON t1.d = t2.d ORDER BY a

Re: [PR] WIP: feat: emitting partial join results in `HashJoinStream` [arrow-datafusion]

2023-11-03 Thread via GitHub
korowa commented on code in PR #8020: URL: https://github.com/apache/arrow-datafusion/pull/8020#discussion_r1382087972 ## datafusion/sqllogictest/test_files/join_disable_repartition_joins.slt: ## @@ -72,11 +72,11 @@ SELECT t1.a, t1.b, t1.c, t2.a as a2 ON t1.d = t2.d ORDER BY a

Re: [PR] GH-34865: [C++][Flight RPC] Add Session management messages [arrow]

2023-11-03 Thread via GitHub
stevelorddremio commented on code in PR #34817: URL: https://github.com/apache/arrow/pull/34817#discussion_r1382089174 ## format/Flight.proto: ## @@ -503,3 +504,100 @@ message FlightData { message PutResult { bytes app_metadata = 1; } + +/* + * Request message for the "Clos

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on PR #38404: URL: https://github.com/apache/arrow/pull/38404#issuecomment-1792969508 @aiguofer sorry about that - I'll give this a review over the weekend once Github is working again. -- This is an automated message from the Apache Git Service. To respond to the messa

Re: [PR] GH-34865: [C++][Flight RPC] Add Session management messages [arrow]

2023-11-03 Thread via GitHub
indigophox commented on code in PR #34817: URL: https://github.com/apache/arrow/pull/34817#discussion_r1382112515 ## cpp/src/arrow/flight/sql/server_session_middleware.cc: ## @@ -0,0 +1,179 @@ +// Licensed to the Apache Software Foundation (ASF) under one +// or more contributor

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on code in PR #38521: URL: https://github.com/apache/arrow/pull/38521#discussion_r1382110222 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/ArrowFlightSqlClientHandler.java: ## @@ -364,6 +415,31 @@ public static final cla

Re: [PR] GH-33475: [Java] Add parameter binding for Prepared Statements in JDBC driver [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on PR #38404: URL: https://github.com/apache/arrow/pull/38404#issuecomment-1792950480 @lidavidm @jduo Hey all, I've spent a lot of time on this PR. It's a little frustrating that a PR that started well after mine has merged and now I have to deal with merge conflicts whil

Re: [PR] WIP: feat: emitting partial join results in `HashJoinStream` [arrow-datafusion]

2023-11-03 Thread via GitHub
korowa commented on PR #8020: URL: https://github.com/apache/arrow-datafusion/pull/8020#issuecomment-1792949167 Thank you @alamb , but I don't think this PR worth benchmarking it -- after running current unit tests with different batch sizes, I've found that arguments for `adjust_indices_b

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
aiguofer commented on code in PR #38521: URL: https://github.com/apache/arrow/pull/38521#discussion_r1382091835 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/ArrowFlightSqlClientHandler.java: ## @@ -364,6 +415,31 @@ public static final cla

Re: [PR] WIP: feat: emitting partial join results in `HashJoinStream` [arrow-datafusion]

2023-11-03 Thread via GitHub
korowa commented on code in PR #8020: URL: https://github.com/apache/arrow-datafusion/pull/8020#discussion_r1382087972 ## datafusion/sqllogictest/test_files/join_disable_repartition_joins.slt: ## @@ -72,11 +72,11 @@ SELECT t1.a, t1.b, t1.c, t2.a as a2 ON t1.d = t2.d ORDER BY a

Re: [I] [Java] Flight SQL JDBC driver stops receiving RecordBatches when it encounters one empty RecordBatch. [arrow]

2023-11-03 Thread via GitHub
jduo commented on issue #36912: URL: https://github.com/apache/arrow/issues/36912#issuecomment-1792936778 take -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscrib

Re: [I] [Epic] A new Scalar Function interface [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #7977: URL: https://github.com/apache/arrow-datafusion/issues/7977#issuecomment-1792921792 Here is a slightly different proposal: https://github.com/apache/arrow-datafusion/issues/8045 -- This is an automated message from the Apache Git Service. To respond to the

[I] [EPIC] Unify Function Interface (remove `BuiltInScalarFunction`) [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb opened a new issue, #8045: URL: https://github.com/apache/arrow-datafusion/issues/8045 ### Is your feature request related to a problem or challenge? This is based on the wonderful writeup from @2010YOUY01 in https://github.com/apache/arrow-datafusion/issues/7977 As prev

Re: [PR] GH-38255: [Go][C++] Implement Flight SQL Bulk Ingestion [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on code in PR #38385: URL: https://github.com/apache/arrow/pull/38385#discussion_r1382045726 ## cpp/src/arrow/flight/sql/client.cc: ## @@ -256,6 +256,88 @@ arrow::Result FlightSqlClient::ExecuteSubstraitUpdate( return update_result.record_count(); } +ar

Re: [I] Support Apache ORC File Format, and Use Sparse Index [arrow-datafusion]

2023-11-03 Thread via GitHub
waynexia commented on issue #4707: URL: https://github.com/apache/arrow-datafusion/issues/4707#issuecomment-1792911734 Drafted a [short-term roadmap](https://github.com/datafusion-contrib/datafusion-orc/issues/7) for datafusion-orc -- This is an automated message from the Apache Git Ser

Re: [PR] Encapsulate `EquivalenceClass` [arrow-datafusion]

2023-11-03 Thread via GitHub
ozankabak commented on code in PR #8034: URL: https://github.com/apache/arrow-datafusion/pull/8034#discussion_r1382050320 ## datafusion/physical-expr/src/equivalence.rs: ## @@ -20,26 +20,114 @@ use std::hash::Hash; use std::sync::Arc; use crate::expressions::Column; -use cra

Re: [PR] Push limit into aggregation for DISTINCT ... LIMIT queries [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8038: URL: https://github.com/apache/arrow-datafusion/pull/8038#discussion_r1382044493 ## datafusion/physical-plan/src/aggregates/row_hash.rs: ## @@ -266,6 +266,12 @@ pub(crate) struct GroupedHashAggregateStream { /// The spill state object

Re: [PR] Encapsulate `EquivalenceClass` [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8034: URL: https://github.com/apache/arrow-datafusion/pull/8034#discussion_r1382039934 ## datafusion/physical-expr/src/equivalence.rs: ## @@ -20,26 +20,114 @@ use std::hash::Hash; use std::sync::Arc; use crate::expressions::Column; -use crate::

Re: [PR] GH-34532: [Java][FlightSQL] Change JDBC to handle multi-endpoints [arrow]

2023-11-03 Thread via GitHub
lidavidm merged PR #38521: URL: https://github.com/apache/arrow/pull/38521 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arrow.apac

Re: [PR] GH-38570: [R] Ensure that test-nix-libs is warning free [arrow]

2023-11-03 Thread via GitHub
jonkeane commented on code in PR #38571: URL: https://github.com/apache/arrow/pull/38571#discussion_r1382037196 ## r/tools/nixlibs.R: ## @@ -828,10 +832,10 @@ quietly <- !env_is("ARROW_R_DEV", "true") not_cran <- env_is("NOT_CRAN", "true") -if (is_release) { +if (is_release

Re: [PR] GH-38570: [R] Ensure that test-nix-libs is warning free [arrow]

2023-11-03 Thread via GitHub
jonkeane commented on code in PR #38571: URL: https://github.com/apache/arrow/pull/38571#discussion_r1382036428 ## r/tools/nixlibs.R: ## @@ -268,7 +272,7 @@ get_macos_openssl_dir <- function() { openssl_root_dir <- "/usr/local" } } - return(openssl_root_dir) + o

Re: [PR] WIP: feat: emitting partial join results in `HashJoinStream` [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8020: URL: https://github.com/apache/arrow-datafusion/pull/8020#discussion_r1382024697 ## datafusion/sqllogictest/test_files/join_disable_repartition_joins.slt: ## @@ -72,11 +72,11 @@ SELECT t1.a, t1.b, t1.c, t2.a as a2 ON t1.d = t2.d ORDER BY a2

Re: [PR] Minor: Update quarterly_roadmap.md [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8044: URL: https://github.com/apache/arrow-datafusion/pull/8044#discussion_r1381986821 ## docs/source/contributor-guide/quarterly_roadmap.md: ## @@ -23,9 +23,22 @@ A quarterly roadmap will be published to give the DataFusion community visibilit

Re: [PR] Encapsulate `ProjectionMapping` as a struct [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8033: URL: https://github.com/apache/arrow-datafusion/pull/8033#discussion_r1381990394 ## datafusion/physical-expr/src/utils.rs: ## @@ -633,230 +630,4 @@ mod tests { assert_eq!(&expected, result); Ok(()) } - -#[test] -

Re: [PR] Minor: Update quarterly_roadmap.md [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8044: URL: https://github.com/apache/arrow-datafusion/pull/8044#discussion_r1381985871 ## docs/source/contributor-guide/quarterly_roadmap.md: ## @@ -23,9 +23,22 @@ A quarterly roadmap will be published to give the DataFusion community visibilit

Re: [PR] Minor: Update quarterly_roadmap.md [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8044: URL: https://github.com/apache/arrow-datafusion/pull/8044#discussion_r1381986282 ## docs/source/contributor-guide/quarterly_roadmap.md: ## @@ -23,9 +23,22 @@ A quarterly roadmap will be published to give the DataFusion community visibilit

[PR] Update: Update quarterly_roadmap.md [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb opened a new pull request, #8044: URL: https://github.com/apache/arrow-datafusion/pull/8044 ## Which issue does this PR close? N/A ## Rationale for this change In general I think it would be good for us to communicate more broadly wha we are working on, for two rea

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
lidavidm commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1381973540 ## java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightServer.java: ## @@ -306,28 +329,63 @@ public Builder maxInboundMessageSize(int maxMessageSize) {

Re: [PR] Encapsulate `ProjectionMapping` as a struct [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on PR #8033: URL: https://github.com/apache/arrow-datafusion/pull/8033#issuecomment-1792778598 FWI @ozankabak and @mustafasrepo -- I realize this may conflict with WIP PRs you may have, but I had it ready so I figured I would put it up for consideration -- This is an a

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1381957485 ## java/flight/flight-sql-jdbc-core/src/main/java/org/apache/arrow/driver/jdbc/client/utils/ClientAuthenticationUtils.java: ## @@ -235,6 +235,57 @@ public static Inpu

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1381950516 ## java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightServer.java: ## @@ -306,28 +329,63 @@ public Builder maxInboundMessageSize(int maxMessageSize)

Re: [PR] GH-38460: [Java][FlightRPC] Add mTLS support for Flight SQL JDBC driver [arrow]

2023-11-03 Thread via GitHub
prmoore77 commented on code in PR #38461: URL: https://github.com/apache/arrow/pull/38461#discussion_r1381950516 ## java/flight/flight-core/src/main/java/org/apache/arrow/flight/FlightServer.java: ## @@ -306,28 +329,63 @@ public Builder maxInboundMessageSize(int maxMessageSize)

Re: [I] array_agg with pyarrow errors with ArrowInvalid: Schema at index 0 was different [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #8032: URL: https://github.com/apache/arrow-datafusion/issues/8032#issuecomment-1792746377 Thank you for the report @Maxsparrow -- I have added this to our general epic for array implementation https://github.com/apache/arrow-datafusion/issues/6980 -- This is a

Re: [I] Inconsistent null handling in min/max [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #8031: URL: https://github.com/apache/arrow-datafusion/issues/8031#issuecomment-1792744236 FWIW I think the min/max accumulators are some of the oldest code in DataFusion. I am very much on board with the idea of standardizing on the same (total ordering) -- Thi

Re: [I] go/adbc/driver/snowflake: improved support for decimal128 types [arrow-adbc]

2023-11-03 Thread via GitHub
CurtHagenlocher commented on issue #1242: URL: https://github.com/apache/arrow-adbc/issues/1242#issuecomment-1792744006 There's a fair amount of logic in https://github.com/snowflakedb/gosnowflake/blob/master/converter.go which simply isn't reflected in the ADBC driver. Specifically, Snowfl

Re: [I] array_agg with pyarrow errors with ArrowInvalid: Schema at index 0 was different [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #8032: URL: https://github.com/apache/arrow-datafusion/issues/8032#issuecomment-1792744889 fyi @jayzhan211 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific

Re: [I] Error from example `catalog` [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #8041: URL: https://github.com/apache/arrow-datafusion/issues/8041#issuecomment-1792739403 > It seems like there weren't that many parquet files to read before? https://github.com/apache/arrow-datafusion/pull/5326#pullrequestreview-1304611968 That could be --

Re: [I] Support udf `range` [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #8028: URL: https://github.com/apache/arrow-datafusion/issues/8028#issuecomment-1792737369 Added as subtask of https://github.com/apache/arrow-datafusion/issues/6980 -- This is an automated message from the Apache Git Service. To respond to the message, please log

Re: [I] [Epic] A new Scalar Function interface [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on issue #7977: URL: https://github.com/apache/arrow-datafusion/issues/7977#issuecomment-1792733008 Here is a slightly different proposal (basically extend ScalarUDF to support all features of `BuiltInScalarFunction` and then remove `BuiltInScalarFunction`): https://github.

Re: [PR] GH-38430: [R] Add test + fix corner cases after nixlibs.R refactor [arrow]

2023-11-03 Thread via GitHub
github-actions[bot] commented on PR #38534: URL: https://github.com/apache/arrow/pull/38534#issuecomment-1792718732 Revision: 1e8d8f43a9c1ed22b0ae57714a08c6f7b05446f2 Submitted crossbow builds: [ursacomputing/crossbow @ actions-c6ce43e763](https://github.com/ursacomputing/crossbow/bra

Re: [PR] GH-38430: [R] Add test + fix corner cases after nixlibs.R refactor [arrow]

2023-11-03 Thread via GitHub
paleolimbot commented on PR #38534: URL: https://github.com/apache/arrow/pull/38534#issuecomment-1792713905 @github-actions crossbow submit r-binary-packages -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL abov

Re: [PR] Encapsulate `ProjectionMapping` as a struct [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8033: URL: https://github.com/apache/arrow-datafusion/pull/8033#discussion_r1381928130 ## datafusion/physical-plan/src/common.rs: ## @@ -375,38 +373,6 @@ pub fn batch_byte_size(batch: &RecordBatch) -> usize { batch.get_array_memory_size() }

Re: [PR] Encapsulate `ProjectionMapping` as a struct [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8033: URL: https://github.com/apache/arrow-datafusion/pull/8033#discussion_r1381927767 ## datafusion/physical-expr/src/equivalence.rs: ## @@ -42,8 +42,65 @@ use indexmap::IndexMap; pub type EquivalenceClass = Vec>; /// Stores the mapping betwee

Re: [PR] Encapsulate `ProjectionMapping` as a struct [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb commented on code in PR #8033: URL: https://github.com/apache/arrow-datafusion/pull/8033#discussion_r1381927313 ## datafusion/physical-expr/src/equivalence.rs: ## @@ -2565,4 +2625,234 @@ mod tests { Ok(()) } + +#[test] +fn test_get_indices_of_matchi

Re: [PR] Combine Equivalence and Ordering equivalence to simplify state [arrow-datafusion]

2023-11-03 Thread via GitHub
alamb merged PR #8006: URL: https://github.com/apache/arrow-datafusion/pull/8006 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@arro

  1   2   >