[GitHub] [arrow] houqp commented on pull request #7324: ARROW-9005: [Rust] [Datafusion] support sort expression
houqp commented on pull request #7324: URL: https://github.com/apache/arrow/pull/7324#issuecomment-640380594 alright, i have added nulls ordering support through sqlparser 0.2.6. only thing left is waiting for merge of #7365. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] emkornfield commented on pull request #7110: ARROW-8952: [C++] WIP Support for textual, JSON schema representation
emkornfield commented on pull request #7110: URL: https://github.com/apache/arrow/pull/7110#issuecomment-640346575 @chish42 I don't have a lot of expertise in CMake. Not sure if it helps but we've already checked in generated flatbuffers header files checked in https://github.com/apache/arrow/blob/master/cpp/src/generated/Schema_generated.h This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] andygrove commented on pull request #7324: ARROW-9005: [Rust] [Datafusion] support sort expression
andygrove commented on pull request #7324: URL: https://github.com/apache/arrow/pull/7324#issuecomment-640336526 @houqp I released sqlparser 0.2.6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] andygrove commented on a change in pull request #7371: ARROW-8736: [Rust] [DataFusion] Table API should provide a schema() method
andygrove commented on a change in pull request #7371:
URL: https://github.com/apache/arrow/pull/7371#discussion_r436435692
##
File path: rust/datafusion/src/execution/table_impl.rs
##
@@ -131,6 +132,11 @@ impl Table for TableImpl {
) -> Result> {
ctx.collect_plan(&self.plan.clone(), batch_size)
}
+
+/// Returns the schema from the logical plan
+fn schema(&self) -> &Box {
Review comment:
I'm curious, would it be possible to just return `&Schema` here? I know
we tend to return `Arc` in many places and this is a pattern I
introduced when I was less familiar with Rust, but it would be good to start
moving away from this pattern where we can.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [arrow] liyafan82 commented on pull request #7354: ARROW-8909: [Java] Out of order writes using setSafe
liyafan82 commented on pull request #7354: URL: https://github.com/apache/arrow/pull/7354#issuecomment-640323257 > +1 > > Do we ever want to support out of order writes? @rymurr Thanks a lot for your comments. I am not sure, as writing a value out of order may require moving all values after it, which can be expensive. Do you have any scenario in mind that requires this feature? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] liyafan82 commented on a change in pull request #7354: ARROW-8909: [Java] Out of order writes using setSafe
liyafan82 commented on a change in pull request #7354:
URL: https://github.com/apache/arrow/pull/7354#discussion_r436429167
##
File path: docs/source/java/vector.rst
##
@@ -118,6 +118,28 @@ Some points to note about the steps above:
no longer used, to avoid resource leak. To make sure of this, it is
recommended to place vector related operations
into a try-with-resources block.
+* For fixed width vectors (e.g. IntVector), we can set values at different
indices in arbitrary orders.
+ For variable width vectors (e.g. VarCharVector), however, we must set values
in non-decreasing order of the
+ indices. Otherwise, the values after the set positiion will become invalid.
For example, suppose we use the
+ following statements to populate a variable width vector:
+
+.. code-block:: Java
+
+VarCharVector vector = new VarCharVector("vector", allocator)) {
Review comment:
Typo fixed. Thanks a lot for your careful review.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7372: ARROW-9061: [Packaging][APT][Yum][GLib] Add Apache Arrow Datasets GLib
github-actions[bot] commented on pull request #7372: URL: https://github.com/apache/arrow/pull/7372#issuecomment-640306153 Revision: 0f9c61c6719dcc78cfe4adb91ffaa1fd554219fa Submitted crossbow builds: [ursa-labs/crossbow @ actions-296](https://github.com/ursa-labs/crossbow/branches/all?query=actions-296) |Task|Status| ||--| |centos-6-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-centos-6-amd64)| |centos-7-aarch64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |centos-7-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-centos-7-amd64)| |centos-8-aarch64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |centos-8-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-centos-8-amd64)| |debian-buster-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-debian-buster-amd64)| |debian-buster-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |debian-stretch-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-debian-stretch-amd64)| |debian-stretch-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-bionic-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-ubuntu-bionic-amd64)| |ubuntu-bionic-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-eoan-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-ubuntu-eoan-amd64)| |ubuntu-eoan-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-focal-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-ubuntu-focal-amd64)| |ubuntu-focal-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-xenial-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-296-github-ubuntu-xenial-amd64)| |ubuntu-xenial-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7372: ARROW-9061: [Packaging][APT][Yum][GLib] Add Apache Arrow Datasets GLib
kou commented on pull request #7372: URL: https://github.com/apache/arrow/pull/7372#issuecomment-640305823 @github-actions crossbow submit -g linux This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7372: ARROW-9061: [Packaging][APT][Yum][GLib] Add Apache Arrow Datasets GLib
github-actions[bot] commented on pull request #7372: URL: https://github.com/apache/arrow/pull/7372#issuecomment-640304800 https://issues.apache.org/jira/browse/ARROW-9061 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou opened a new pull request #7372: ARROW-9061: [Packaging][APT][Yum][GLib] Add Apache Arrow Datasets GLib
kou opened a new pull request #7372: URL: https://github.com/apache/arrow/pull/7372 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou closed pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
kou closed pull request #7370: URL: https://github.com/apache/arrow/pull/7370 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
kou commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640297404 +1 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7371: ARROW-8736: [Rust] [DataFusion] Table API should provide a schema() method
github-actions[bot] commented on pull request #7371: URL: https://github.com/apache/arrow/pull/7371#issuecomment-640288022 https://issues.apache.org/jira/browse/ARROW-8736 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kyle-mccarthy opened a new pull request #7371: ARROW-8736: [Rust] [DataFusion] Table API should provide a schema() method
kyle-mccarthy opened a new pull request #7371: URL: https://github.com/apache/arrow/pull/7371 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
github-actions[bot] commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640284750 Revision: 516b7be6e3e1a48d58a76899cd60a2da0b01b85b Submitted crossbow builds: [ursa-labs/crossbow @ actions-295](https://github.com/ursa-labs/crossbow/branches/all?query=actions-295) |Task|Status| ||--| |centos-6-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-centos-6-amd64)| |centos-7-aarch64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |centos-7-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-centos-7-amd64)| |centos-8-aarch64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |centos-8-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-centos-8-amd64)| |debian-buster-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-debian-buster-amd64)| |debian-buster-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |debian-stretch-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-debian-stretch-amd64)| |debian-stretch-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-bionic-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-ubuntu-bionic-amd64)| |ubuntu-bionic-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-eoan-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-ubuntu-eoan-amd64)| |ubuntu-eoan-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-focal-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-ubuntu-focal-amd64)| |ubuntu-focal-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-xenial-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-295-github-ubuntu-xenial-amd64)| |ubuntu-xenial-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
kou commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640284572 @github-actions crossbow submit -g linux This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
github-actions[bot] commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640283866 Revision: f32d9807ae6dfb9a3c706dee6103d930009ca7f4 Submitted crossbow builds: [ursa-labs/crossbow @ actions-294](https://github.com/ursa-labs/crossbow/branches/all?query=actions-294) |Task|Status| ||--| |wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux1-cp35m)| |wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux1-cp36m)| |wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux1-cp37m)| |wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux1-cp38)| |wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2010-cp35m)| |wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2010-cp36m)| |wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2010-cp37m)| |wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2010-cp38)| |wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2014-cp35m)| |wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2014-cp36m)| |wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2014-cp37m)| |wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-294-azure-wheel-manylinux2014-cp38)| |wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)| |wheel-win-cp36m|[](https://ci.a
[GitHub] [arrow] kou commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kou commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640283675 @github-actions crossbow submit -g wheel This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kou commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640283653 Thanks! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kszucs commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kszucs commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640280052 @kou I set up the environment variables using my credentials, both env variables should be available now in the travis, azure and github builds. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
github-actions[bot] commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640279770 Revision: aa9b75f5ce41359a6f8cdb0ab9b8043c4bb5a53f Submitted crossbow builds: [ursa-labs/crossbow @ actions-293](https://github.com/ursa-labs/crossbow/branches/all?query=actions-293) |Task|Status| ||--| |centos-6-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-centos-6-amd64)| |centos-7-aarch64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |centos-7-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-centos-7-amd64)| |centos-8-aarch64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |centos-8-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-centos-8-amd64)| |debian-buster-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-debian-buster-amd64)| |debian-buster-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |debian-stretch-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-debian-stretch-amd64)| |debian-stretch-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-bionic-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-ubuntu-bionic-amd64)| |ubuntu-bionic-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-eoan-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-ubuntu-eoan-amd64)| |ubuntu-eoan-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-focal-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-ubuntu-focal-amd64)| |ubuntu-focal-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |ubuntu-xenial-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-293-github-ubuntu-xenial-amd64)| |ubuntu-xenial-arm64|[](https://travis-ci.org/ursa-labs/crossbow/branches)| This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
kou commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640279549 @github-actions crossbow submit -g linux This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kou commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640277036 @kszucs Could you set `ARCHERY_DOCKER_USER` and `ARCHERY_DOCKER_PASSWORD` environment variables to https://dev.azure.com/ursa-labs/crossbow/ ? If we have them, images for wheels can be updated automatically. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
github-actions[bot] commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640275418 https://issues.apache.org/jira/browse/ARROW-9060 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
github-actions[bot] commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640274913 Revision: aa9b75f5ce41359a6f8cdb0ab9b8043c4bb5a53f Submitted crossbow builds: [ursa-labs/crossbow @ actions-292](https://github.com/ursa-labs/crossbow/branches/all?query=actions-292) |Task|Status| ||--| |debian-buster-amd64|[](https://github.com/ursa-labs/crossbow/actions?query=branch:actions-292-github-debian-buster-amd64)| This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
kou commented on pull request #7370: URL: https://github.com/apache/arrow/pull/7370#issuecomment-640274678 @github-actions crossbow submit debian-buster-amd64 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou opened a new pull request #7370: ARROW-9060: [GLib] Add support for building Apache Arrow Datasets GLib with non-installed Apache Arrow Datasets
kou opened a new pull request #7370: URL: https://github.com/apache/arrow/pull/7370 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm closed pull request #7240: ARROW-8792: [C++][Python][R][GLib] New Array compute kernels implementation and execution framework
wesm closed pull request #7240: URL: https://github.com/apache/arrow/pull/7240 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm commented on pull request #7240: ARROW-8792: [C++][Python][R][GLib] New Array compute kernels implementation and execution framework
wesm commented on pull request #7240: URL: https://github.com/apache/arrow/pull/7240#issuecomment-640250180 Closing the PR again. Please open JIRA issues or review other PRs to provide more feedback. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7369: ARROW-9059 [RUST] Fix sign in array slice_data_docstring
github-actions[bot] commented on pull request #7369: URL: https://github.com/apache/arrow/pull/7369#issuecomment-640249069 https://issues.apache.org/jira/browse/ARROW-9059 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] robert-wagner opened a new pull request #7369: ARROW-9059 [RUST] Fix sign in array slice_data_docstring
robert-wagner opened a new pull request #7369: URL: https://github.com/apache/arrow/pull/7369 Change the panic behavior in the docstring to reflect the panic behavior in the code This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] wesm closed pull request #7280: ARROW-8157: [C++][Gandiva] Support building with LLVM 9
wesm closed pull request #7280: URL: https://github.com/apache/arrow/pull/7280 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] fsaintjacques closed pull request #7353: ARROW-9037: [C++] C-ABI: do not error out when importing array with null_count == -1
fsaintjacques closed pull request #7353: URL: https://github.com/apache/arrow/pull/7353 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] nevi-me commented on a change in pull request #7365: ARROW-9007: [Rust] Support appending array data to builders
nevi-me commented on a change in pull request #7365:
URL: https://github.com/apache/arrow/pull/7365#discussion_r436354577
##
File path: rust/arrow/src/array/builder.rs
##
@@ -577,6 +632,81 @@ where
self
}
+/// Appends data from other arrays into the builder
+///
+/// This is most useful when concatenating arrays of the same type into a
builder.
+fn append_data(&mut self, data: &[ArrayDataRef]) -> Result<()> {
+if !check_array_data_type(&self.data_type(), data) {
+return Err(ArrowError::InvalidArgumentError(
+"Cannot append data to builder if data types are
different".to_string(),
+));
+}
+// determine the latest offset on the builder
+let mut cum_offset = if self.offsets_builder.len() == 0 {
+0
+} else {
+// peek into buffer to get last appended offset
+let buffer = self.offsets_builder.buffer.data();
+let len = self.offsets_builder.len();
+let (start, end) = ((len - 1) * 4, len * 4);
+let slice = &buffer[start..end];
+i32::from_le_bytes(slice.try_into().unwrap())
+};
+for array in data {
+if array.child_data().len() != 1 {
Review comment:
> But I do think we should make sure error handling behavior is
consistent, i.e. invalid input should not lead to partial append to the array.
If we are to use ArrayDataRef as input type, then I feel like we do need to
have custom validation logic for each array type as you mentioned.
Thanks, I agree. Changes made in my latest commit.
> The idea of using ArrayRef as input type is interesting. It does look like
a simpler interface for end users and could simplify the error handling logic.
The preferred way to convert `&[ArrayRef]` to `ArrayRef` will be the
`concat` kernel that you've added. concat might then forego its current
validation, and potentially be like:
```rust
pub fn concat(array_list: &[ArrayRef]) -> Result {
// get data type from first element
// create builder for data type (this'll have to cater for structs and
lists)
// pass `ArrayDataRef`to builder
// finish builder and return `ArrayRef`
}
```
> What's the downside of using ArrayRef here compared to ArrayDataRef?
`ArrayDataRef` is more flexible. If someone is creating Arrow data from raw
data, there currently isn't much flexibility for them, especially when working
with nested data structures. It might be more convenient to then create
`ArrayData` instead of going all the way to create an array only to append it
to a builder.
Constructing an `ArrayRef` to append is an extra step and at worst requires
going through `arrow::utils::make_array(data: ArrayRef)`.
The upside of `ArrayRef` is skipping the validation checks, though I wonder
what cost the checks result in. We can wait for other reviewers' opinions on
their necessity.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
github-actions[bot] commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640180883 https://issues.apache.org/jira/browse/ARROW-9058 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
github-actions[bot] commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640180722 Revision: 9287230aaf8b33352e4666494e6b72e507fc1aa3 Submitted crossbow builds: [ursa-labs/crossbow @ actions-291](https://github.com/ursa-labs/crossbow/branches/all?query=actions-291) |Task|Status| ||--| |wheel-manylinux1-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux1-cp35m)| |wheel-manylinux1-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux1-cp36m)| |wheel-manylinux1-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux1-cp37m)| |wheel-manylinux1-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux1-cp38)| |wheel-manylinux2010-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2010-cp35m)| |wheel-manylinux2010-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2010-cp36m)| |wheel-manylinux2010-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2010-cp37m)| |wheel-manylinux2010-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2010-cp38)| |wheel-manylinux2014-cp35m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2014-cp35m)| |wheel-manylinux2014-cp36m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2014-cp36m)| |wheel-manylinux2014-cp37m|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2014-cp37m)| |wheel-manylinux2014-cp38|[](https://dev.azure.com/ursa-labs/crossbow/_build/latest?definitionId=1&branchName=actions-291-azure-wheel-manylinux2014-cp38)| |wheel-osx-cp35m|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-osx-cp36m|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-osx-cp37m|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-osx-cp38|[](https://travis-ci.org/ursa-labs/crossbow/branches)| |wheel-win-cp35m|[](https://ci.appveyor.com/project/ursa-labs/crossbow/history)| |wheel-win-cp36m|[](https://ci.a
[GitHub] [arrow] kou commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kou commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640180695 Or should we mirror Boost 1.68.0 at https://dl.bintray.com/ursalabs/arrow-boost/ ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou commented on pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kou commented on pull request #7368: URL: https://github.com/apache/arrow/pull/7368#issuecomment-640180469 @github-actions crossbow submit -g wheel This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] kou opened a new pull request #7368: ARROW-9058: [Packaging][wheel] Use sourceforge.net to download Boost
kou opened a new pull request #7368: URL: https://github.com/apache/arrow/pull/7368 Because dl.bintray.com/boostorg/ sometimes reaches limitation. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] houqp commented on a change in pull request #7365: ARROW-9007: [Rust] Support appending array data to builders
houqp commented on a change in pull request #7365:
URL: https://github.com/apache/arrow/pull/7365#discussion_r436336846
##
File path: rust/arrow/src/array/builder.rs
##
@@ -577,6 +632,81 @@ where
self
}
+/// Appends data from other arrays into the builder
+///
+/// This is most useful when concatenating arrays of the same type into a
builder.
+fn append_data(&mut self, data: &[ArrayDataRef]) -> Result<()> {
+if !check_array_data_type(&self.data_type(), data) {
+return Err(ArrowError::InvalidArgumentError(
+"Cannot append data to builder if data types are
different".to_string(),
+));
+}
+// determine the latest offset on the builder
+let mut cum_offset = if self.offsets_builder.len() == 0 {
+0
+} else {
+// peek into buffer to get last appended offset
+let buffer = self.offsets_builder.buffer.data();
+let len = self.offsets_builder.len();
+let (start, end) = ((len - 1) * 4, len * 4);
+let slice = &buffer[start..end];
+i32::from_le_bytes(slice.try_into().unwrap())
+};
+for array in data {
+if array.child_data().len() != 1 {
Review comment:
I think we can leave the memory allocation performance optimization to
future PRs given that number of arrays in `data: &[ArrayDataRef]` should not be
too large.
But I do think we should make sure error handling behavior is consistent,
i.e. invalid input should not lead to partial append to the array. If we are to
use ArrayDataRef as input type, then I feel like we do need to have custom
validation logic for each array type as you mentioned.
The idea of using `ArrayRef` as input type is interesting. It does look like
a simpler interface for end users and could simplify the error handling logic.
What's the downside of using `ArrayRef` here compared to `ArrayDataRef`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [arrow] houqp commented on a change in pull request #7365: ARROW-9007: [Rust] Support appending array data to builders
houqp commented on a change in pull request #7365:
URL: https://github.com/apache/arrow/pull/7365#discussion_r436335986
##
File path: rust/arrow/src/array/builder.rs
##
@@ -841,12 +1048,91 @@ impl ArrayBuilder for StringBuilder {
}
}
+// Helper function for appending Binary and Utf8 data
+fn append_binary_data(
+builder: &mut ListBuilder,
+data_type: &DataType,
+data: &[ArrayDataRef],
+) -> Result<()> {
+if !check_array_data_type(data_type, data) {
+return Err(ArrowError::InvalidArgumentError(
+"Cannot append data to builder if data types are
different".to_string(),
+));
+}
+for array in data {
+// convert string to List to reuse list's cast
+let int_data = &array.buffers()[1];
+let int_data = Arc::new(ArrayData::new(
+DataType::UInt8,
+int_data.len(),
+None,
+None,
+0,
+vec![int_data.clone()],
+vec![],
+)) as ArrayDataRef;
+let list_data = Arc::new(ArrayData::new(
+DataType::List(Box::new(DataType::UInt8)),
+array.len(),
+None,
+array.null_buffer().map(|buf| buf.clone()),
+array.offset(),
+vec![(&array.buffers()[0]).clone()],
+vec![int_data],
+));
+builder.append_data(&[list_data])?;
+}
+Ok(())
+}
+
impl ArrayBuilder for FixedSizeBinaryBuilder {
/// Returns the builder as a non-mutable `Any` reference.
fn as_any(&self) -> &Any {
self
}
+/// Appends data from other arrays into the builder
+///
+/// This is most useful when concatenating arrays of the same type into a
builder.
+fn append_data(&mut self, data: &[ArrayDataRef]) -> Result<()> {
+if !check_array_data_type(&self.data_type(), data) {
+return Err(ArrowError::InvalidArgumentError(
+"Cannot append data to builder if data types are
different".to_string(),
+));
+}
+for array in data {
+// convert string to FixedSizeList to reuse list's append
+let int_data = &array.buffers()[0];
+let int_data = Arc::new(ArrayData::new(
+DataType::UInt8,
+int_data.len(),
+None,
+None,
+0,
+vec![int_data.clone()],
+vec![],
+)) as ArrayDataRef;
+let list_data = Arc::new(ArrayData::new(
+DataType::FixedSizeList(Box::new(DataType::UInt8),
self.builder.list_len),
Review comment:
Ha, good point, I forgot byte_width is part of the data type signature
:+1:
with regards to `value_length()` v.s. `list_len`, after thinking more about
it, I think you are right to use list_len here since these are all internal
module implementation.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [arrow] github-actions[bot] commented on pull request #7367: ARROW-9057: [Rust][Datafusion] fix projection on in memory scan
github-actions[bot] commented on pull request #7367: URL: https://github.com/apache/arrow/pull/7367#issuecomment-640170791 https://issues.apache.org/jira/browse/ARROW-9057 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [arrow] houqp opened a new pull request #7367: ARROW-9057: [Rust][Datafusion] fix projection on in memory scan
houqp opened a new pull request #7367: URL: https://github.com/apache/arrow/pull/7367 This fixes the `ArrowError::InvalidArgumentError` returned by `RecordBatch::try_new`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
