[jira] [Created] (ARROW-7455) [Ruby] Use Arrow::DataType.resolve for all GArrowDataType input
Kouhei Sutou created ARROW-7455: --- Summary: [Ruby] Use Arrow::DataType.resolve for all GArrowDataType input Key: ARROW-7455 URL: https://issues.apache.org/jira/browse/ARROW-7455 Project: Apache Arrow Issue Type: Improvement Components: Ruby Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7454) [Ruby] Add support for saving/loading TSV
Kouhei Sutou created ARROW-7454: --- Summary: [Ruby] Add support for saving/loading TSV Key: ARROW-7454 URL: https://issues.apache.org/jira/browse/ARROW-7454 Project: Apache Arrow Issue Type: Improvement Components: Ruby Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian Jira (v8.3.4#803005)
Re: Regarding pyarrow package size for aws lambda
hi Thanuj, It is straightforward to alter flags in https://github.com/apache/arrow/blob/master/python/manylinux1/build_arrow.sh and create a slimmed down wheel. See the README for instructions how to run the script (It uses docker-compose so should be easily reproducible on any machine). It's possible that we could publish a pyarrow-slim wheel on PyPI in future releases, but we would need help from volunteers to implement and maintain this. My organization is not being funded to support wheels and we have already spent too much time on wheel-related issues to take on a project like this. Note that we also have an outstanding problem of shared libraries being duplicated in the Linux wheels so that is surely exacerbating this problem. We tried to fix it but the fix caused more problems so we (at least, my colleagues and I) gave up until someone else could spend time looking at it. Thanks On Fri, Dec 20, 2019 at 4:36 PM thanuj d wrote: > > Hello, > I am Thanuj, I am trying to use pyarrow in aws lambda. The problem I > am facing is with the size more than 250mb. I tried several options but not > successful I tried removing Gandiva files from package but it gives me > corrupt file error. Can you please help me with this on how to disable > gandiva and other options. > -- > *Thanks* > > *D Thanuj,* > *dthanu...@gmail.com .*
Regarding pyarrow package size for aws lambda
Hello, I am Thanuj, I am trying to use pyarrow in aws lambda. The problem I am facing is with the size more than 250mb. I tried several options but not successful I tried removing Gandiva files from package but it gives me corrupt file error. Can you please help me with this on how to disable gandiva and other options. -- *Thanks* *D Thanuj,* *dthanu...@gmail.com .*
[jira] [Created] (ARROW-7453) [Ruby] Add support for Arrow::NullArray#[]
Kouhei Sutou created ARROW-7453: --- Summary: [Ruby] Add support for Arrow::NullArray#[] Key: ARROW-7453 URL: https://issues.apache.org/jira/browse/ARROW-7453 Project: Apache Arrow Issue Type: Improvement Components: Ruby Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7452) [GLib] Make GArrowTimeDataType abstract
Kouhei Sutou created ARROW-7452: --- Summary: [GLib] Make GArrowTimeDataType abstract Key: ARROW-7452 URL: https://issues.apache.org/jira/browse/ARROW-7452 Project: Apache Arrow Issue Type: Improvement Components: GLib Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7451) pyarrow.hdfs.connect crashes when executed asynchronously in processes
Panagiotis Nezis created ARROW-7451:
---
Summary: pyarrow.hdfs.connect crashes when executed asynchronously
in processes
Key: ARROW-7451
URL: https://issues.apache.org/jira/browse/ARROW-7451
Project: Apache Arrow
Issue Type: Bug
Components: Python
Affects Versions: 0.15.1
Reporter: Panagiotis Nezis
When trying to connect to {{hdfs}} from a {{ProcessPoolExecutor}} then the
first call raises an Exception and the function never returns (potential
deadlock?). On the other hand it works as expected with a
{{ThreadPoolExecutor}}.
Sample code that reproduces the problem follows:
{code:python}
import pyarrow as pa
from concurrent.futures import (
ThreadPoolExecutor,
ProcessPoolExecutor,
wait,
ALL_COMPLETED)
def ls():
fs = pa.hdfs.connect('hdfs://host')
print(fs.ls('/'))
# This works as expected
ls()
# Running in parallel
thread_pool = ThreadPoolExecutor(max_workers=4)
process_pool = ProcessPoolExecutor(max_workers=4)
def run(pool):
futures = [pool.submit(ls) for _ in range(5)]
wait(futures, return_when=ALL_COMPLETED)
# The thread_pool works as expected
run(thread_pool)
# The process_pool raises an exception
run(process_pool)
{code}
The following exception is raised:
{noformat}
java.lang.ClassFormatError: Incompatible magic value 1347093252 in class file
org/xml/sax/helpers/LocatorImpl
at java.lang.ClassLoader.findBootstrapClass(Native Method)
at java.lang.ClassLoader.findBootstrapClassOrNull(ClassLoader.java:1015)
at java.lang.ClassLoader.loadClass(ClassLoader.java:413)
at java.lang.ClassLoader.loadClass(ClassLoader.java:411)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at org.apache.xerces.parsers.DOMParser.parse(Unknown Source)
at org.apache.xerces.jaxp.DocumentBuilderImpl.parse(Unknown Source)
at javax.xml.parsers.DocumentBuilder.parse(DocumentBuilder.java:150)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2684)
at org.apache.hadoop.conf.Configuration.parse(Configuration.java:2672)
at
org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2746)
at
org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
at
org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
at org.apache.hadoop.fs.FileSystem.newInstance(FileSystem.java:404)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Trouble with building Arrow GLib
Hi Kou, Thanks for the quick fix. After installing the patch I am able to build Arrow-GLib now. About the issue related to Meson: I was using Python 3, the problem was solved by using conda instead of pip. Kind Regards Chengxin Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Friday, December 20, 2019 12:27 PM, Sutou Kouhei wrote: > Hi, > > > I’ve successfully installed Meson according to your suggestion, however it > > couldn’t run, probably due to some compatibility issues on the system. > > (I’ve written an email to the Helpdesk for help.) > > I think that you're using Python 2. Meson requires Python 3. > > > In the meantime of waiting for their response, I would like to know if it > > is possible to do some modification in the source code of the current > > Apache Arrow release to disable building GObject Introspection, instead of > > waiting for the next release. > > Here is a patch to make GObject Introspection optional: > > https://patch-diff.githubusercontent.com/raw/apache/arrow/pull/6072.patch > > You need to use Meson to build with this patch. > > To use configure, you need to regenerate c_glib/configure > from c_glib/configure.ac. It requires GObject Introspection. > So you can't use configure on your environment. > > Thanks, > > --- > > kou > > In > tSWuBtndqpJhkCaHTDcHDaE3za0pbqK-8fnqKT99Vv6QGVxYAjKuZUUSZ4A94U6DiZaWxc8wYc5JXiu9EoZublUwbXhlq8kar_sguY6onWI=@protonmail.ch > "Re: Trouble with building Arrow GLib" on Fri, 20 Dec 2019 09:59:27 +, > Chengxin Ma c...@protonmail.ch.INVALID wrote: > > > Hi Kou, > > I am working on the Cartesius system. The OS on it is bullx Linux. > > I’ve successfully installed Meson according to your suggestion, however it > > couldn’t run, probably due to some compatibility issues on the system. > > (I’ve written an email to the Helpdesk for help.) > > In the meantime of waiting for their response, I would like to know if it > > is possible to do some modification in the source code of the current > > Apache Arrow release to disable building GObject Introspection, instead of > > waiting for the next release. > > Kind Regards > > Chengxin > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > > On Friday, December 20, 2019 4:49 AM, Sutou Kouhei k...@clear-code.com > > wrote: > > > > > Hi, > > > Arrow GLib can provide C API but it requires GObject > > > Introspection for now. So you need to install GObject > > > Introspection to use Arrow GLib for now. > > > (I'll add a build option to build without GObject > > > Introspection. Then you can build Arrow GLib without GObject > > > Introspection in the next release.) > > > Could you show your environment? Are you using a Linux distribution? > > > You can install Meson by "pip install --user meson". > > > See also: > > > https://mesonbuild.com/Getting-meson.html#installing-meson-with-pip > > > Thanks, > > > > > > kou > > > In > > > DIVG0Hqiw9iory_bs1T6A_cf5etWsPJ0-lbAFxaJ4H2hrTm87EGUED3ztGenYN6EjVzW5_oYL1KIH4V3F_BE1dMQyu4EtbBflin-m-iGC_Q=@protonmail.ch > > > "Trouble with building Arrow GLib" on Thu, 19 Dec 2019 10:53:48 +, > > > Chengxin Ma c...@protonmail.ch.INVALID wrote: > > > > > > > Hi All, > > > > I am building Arrow GLib on a system where I'm not the admin. > > > > In the installation instructions > > > > (https://github.com/apache/arrow/tree/master/c_glib) there are two > > > > options for building GLib: "How to build by users" and "How to build by > > > > developers". I followed the "by users" section and met the following > > > > problem: > > > > "checking for gobject-introspection... configure: error: > > > > gobject-introspection-1.0 is not installed". > > > > "GObject Introspection" is mentioned explicitly in the "How to build by > > > > developers" section. Does this indicate it is optional if we build as a > > > > user? > > > > I checked "./configure --help" but didn't find a way to exclude it in > > > > the build process. > > > > I've also tried to build GObject Introspection > > > > (http://www.linuxfromscratch.org/blfs/view/svn/general/gobject-introspection.html) > > > > but Meson isn't available on the system... > > > > Could someone please offer a suggestion for me to build Arrow GLib? > > > > Kind Regards > > > > Chengxin > > > > Sent with ProtonMail Secure Email.
Re: [NIGHTLY] Arrow Build Report for Job nightly-2019-12-20-0
- conda macOS failures: ongoing issues per ARROW-7398 - test-ubuntu-18.04-cpp-cmake32: -- this failure seems to be a flaky Thrift EP download - wheel-manylinux2010-cp35m: flaky GitHub release upload - test-ubuntu-18.04-cpp-static appears legitimate, I'm opening a JIRA https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-ubuntu-18.04-cpp-static https://issues.apache.org/jira/browse/ARROW-7450 On Fri, Dec 20, 2019 at 7:48 AM Crossbow wrote: > > > Arrow Build Report for Job nightly-2019-12-20-0 > > All tasks: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0 > > Failed Tasks: > - conda-osx-clang-py27: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py27 > - conda-osx-clang-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py36 > - conda-osx-clang-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py37 > - conda-osx-clang-py38: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py38 > - gandiva-jar-osx: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-gandiva-jar-osx > - gandiva-jar-trusty: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-gandiva-jar-trusty > - test-ubuntu-18.04-cpp-cmake32: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-ubuntu-18.04-cpp-cmake32 > - test-ubuntu-18.04-cpp-static: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-ubuntu-18.04-cpp-static > - wheel-manylinux2010-cp35m: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-wheel-manylinux2010-cp35m > > Succeeded Tasks: > - centos-6: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-centos-6 > - centos-7: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-centos-7 > - centos-8: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-centos-8 > - conda-linux-gcc-py27: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py27 > - conda-linux-gcc-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py36 > - conda-linux-gcc-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py37 > - conda-linux-gcc-py38: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py38 > - conda-win-vs2015-py36: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-win-vs2015-py36 > - conda-win-vs2015-py37: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-win-vs2015-py37 > - conda-win-vs2015-py38: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-win-vs2015-py38 > - debian-buster: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-debian-buster > - debian-stretch: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-debian-stretch > - homebrew-cpp: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-homebrew-cpp > - macos-r-autobrew: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-macos-r-autobrew > - test-conda-cpp: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-cpp > - test-conda-python-2.7-pandas-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-2.7-pandas-latest > - test-conda-python-2.7: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-2.7 > - test-conda-python-3.6: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.6 > - test-conda-python-3.7-dask-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-dask-latest > - test-conda-python-3.7-hdfs-2.9.2: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-hdfs-2.9.2 > - test-conda-python-3.7-pandas-latest: > URL: > https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-pandas-latest > -
[NIGHTLY] Arrow Build Report for Job nightly-2019-12-20-0
Arrow Build Report for Job nightly-2019-12-20-0 All tasks: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0 Failed Tasks: - conda-osx-clang-py27: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py27 - conda-osx-clang-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py36 - conda-osx-clang-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py37 - conda-osx-clang-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-osx-clang-py38 - gandiva-jar-osx: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-gandiva-jar-osx - gandiva-jar-trusty: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-gandiva-jar-trusty - test-ubuntu-18.04-cpp-cmake32: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-ubuntu-18.04-cpp-cmake32 - test-ubuntu-18.04-cpp-static: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-ubuntu-18.04-cpp-static - wheel-manylinux2010-cp35m: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-wheel-manylinux2010-cp35m Succeeded Tasks: - centos-6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-centos-6 - centos-7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-centos-7 - centos-8: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-centos-8 - conda-linux-gcc-py27: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py27 - conda-linux-gcc-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py36 - conda-linux-gcc-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py37 - conda-linux-gcc-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-linux-gcc-py38 - conda-win-vs2015-py36: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-win-vs2015-py36 - conda-win-vs2015-py37: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-win-vs2015-py37 - conda-win-vs2015-py38: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-conda-win-vs2015-py38 - debian-buster: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-debian-buster - debian-stretch: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-azure-debian-stretch - homebrew-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-homebrew-cpp - macos-r-autobrew: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-travis-macos-r-autobrew - test-conda-cpp: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-cpp - test-conda-python-2.7-pandas-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-2.7-pandas-latest - test-conda-python-2.7: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-2.7 - test-conda-python-3.6: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.6 - test-conda-python-3.7-dask-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-dask-latest - test-conda-python-3.7-hdfs-2.9.2: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-hdfs-2.9.2 - test-conda-python-3.7-pandas-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-pandas-latest - test-conda-python-3.7-pandas-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-pandas-master - test-conda-python-3.7-spark-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-spark-master - test-conda-python-3.7-turbodbc-latest: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-turbodbc-latest - test-conda-python-3.7-turbodbc-master: URL: https://github.com/ursa-labs/crossbow/branches/all?query=nightly-2019-12-20-0-circle-test-conda-python-3.7-turbodbc-master -
Re: Trouble with building Arrow GLib
Hi, > I’ve successfully installed Meson according to your suggestion, however it > couldn’t run, probably due to some compatibility issues on the system. (I’ve > written an email to the Helpdesk for help.) I think that you're using Python 2. Meson requires Python 3. > In the meantime of waiting for their response, I would like to know if it is > possible to do some modification in the source code of the current Apache > Arrow release to disable building GObject Introspection, instead of waiting > for the next release. Here is a patch to make GObject Introspection optional: https://patch-diff.githubusercontent.com/raw/apache/arrow/pull/6072.patch You need to use Meson to build with this patch. To use configure, you need to regenerate c_glib/configure from c_glib/configure.ac. It requires GObject Introspection. So you can't use configure on your environment. Thanks, -- kou In "Re: Trouble with building Arrow GLib" on Fri, 20 Dec 2019 09:59:27 +, Chengxin Ma wrote: > Hi Kou, > > I am working on [the Cartesius > system](https://userinfo.surfsara.nl/systems/cartesius). The OS on it is > [bullx Linux](https://userinfo.surfsara.nl/systems/cartesius/software/rhel). > > I’ve successfully installed Meson according to your suggestion, however it > couldn’t run, probably due to some compatibility issues on the system. (I’ve > written an email to the Helpdesk for help.) > > In the meantime of waiting for their response, I would like to know if it is > possible to do some modification in the source code of the current Apache > Arrow release to disable building GObject Introspection, instead of waiting > for the next release. > > Kind Regards > Chengxin > > > Sent with ProtonMail Secure Email. > > ‐‐‐ Original Message ‐‐‐ > On Friday, December 20, 2019 4:49 AM, Sutou Kouhei > wrote: > >> Hi, >> >> Arrow GLib can provide C API but it requires GObject >> Introspection for now. So you need to install GObject >> Introspection to use Arrow GLib for now. >> (I'll add a build option to build without GObject >> Introspection. Then you can build Arrow GLib without GObject >> Introspection in the next release.) >> >> Could you show your environment? Are you using a Linux distribution? >> >> You can install Meson by "pip install --user meson". >> See also: https://mesonbuild.com/Getting-meson.html#installing-meson-with-pip >> >> Thanks, >> >> --- >> >> kou >> >> In >> DIVG0Hqiw9iory_bs1T6A_cf5etWsPJ0-lbAFxaJ4H2hrTm87EGUED3ztGenYN6EjVzW5_oYL1KIH4V3F_BE1dMQyu4EtbBflin-m-iGC_Q=@protonmail.ch >> "Trouble with building Arrow GLib" on Thu, 19 Dec 2019 10:53:48 +, >> Chengxin Ma c...@protonmail.ch.INVALID wrote: >> >> > Hi All, >> > I am building Arrow GLib on a system where I'm not the admin. >> > In the installation instructions >> > (https://github.com/apache/arrow/tree/master/c_glib) there are two options >> > for building GLib: "How to build by users" and "How to build by >> > developers". I followed the "by users" section and met the following >> > problem: >> > "checking for gobject-introspection... configure: error: >> > gobject-introspection-1.0 is not installed". >> > "GObject Introspection" is mentioned explicitly in the "How to build by >> > developers" section. Does this indicate it is optional if we build as a >> > user? >> > I checked "./configure --help" but didn't find a way to exclude it in the >> > build process. >> > I've also tried to build GObject Introspection >> > (http://www.linuxfromscratch.org/blfs/view/svn/general/gobject-introspection.html) >> > but Meson isn't available on the system... >> > Could someone please offer a suggestion for me to build Arrow GLib? >> > Kind Regards >> > Chengxin >> > Sent with ProtonMail Secure Email. > >
[jira] [Created] (ARROW-7449) [GLib] Make GObject Introspection optional
Kouhei Sutou created ARROW-7449: --- Summary: [GLib] Make GObject Introspection optional Key: ARROW-7449 URL: https://issues.apache.org/jira/browse/ARROW-7449 Project: Apache Arrow Issue Type: Improvement Components: GLib Reporter: Kouhei Sutou Assignee: Kouhei Sutou -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (ARROW-7448) [R] Parquet compression not working
claesmael...@gmail.com created ARROW-7448:
-
Summary: [R] Parquet compression not working
Key: ARROW-7448
URL: https://issues.apache.org/jira/browse/ARROW-7448
Project: Apache Arrow
Issue Type: Bug
Components: R
Affects Versions: 0.15.1
Environment: Ubuntu 19.10 + conda version of arrow
Reporter: claesmael...@gmail.com
When trying to write a parquet file with gzip compression, I get the following
error:
{code:java}
> arrow::write_parquet(mtcars, "test.parquet", "gzip")
Error in parquet___arrow___FileWriter__WriteTable(self, table, chunk_size) :
Not compatible with requested type: [type=character; target=double].
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [C++][Compute] RFC: add SIMD support to C++ kernel
Hi,
I would recommend against reinventing the wheel. It would be possible
to reuse an existing C++ SIMD library. There are several of them (Vc,
xsimd, libsimdpp...). Of course, "just use Gandiva" is another possible
answer.
Regards
Antoine.
Le 20/12/2019 à 08:32, Yibo Cai a écrit :
> Hi,
>
> I'm investigating SIMD support to C++ compute kernel(not gandiva).
>
> A typical case is the sum kernel[1]. Below tight loop can be easily optimized
> with SIMD.
>
> for (int64_t i = 0; i < length; i++) {
>local.sum += values[i];
> }
>
> Compiler already does loop vectorization. But it's done at compile time
> without knowledge of target cpu.
> Binaries compiled with avx-512 cannot run on old cpu, while binaries compiled
> with only sse4 enabled is suboptimal on new hardware.
>
> I have some proposals, would like to hear comments from community.
>
> - Based on our experience of ISA-L[2] project(optimized storage acceleration
> library for x86 and Arm), runtime dispatcher is a good approach. Basically,
> it links in codes optimized for different cpu features(sse4,avx2,neon,...)
> and selects the best one fits target cpu at first invocation. This is similar
> to gcc indirect function[3], but doesn't depend on compilers.
>
> - Use gcc FMV [4] to generate multiple binaries for one function. See sample
> source and compiled code [5].
>Though looks simple, it has many limitations: It's gcc specific feature,
> no support from clang and msvc. It only works on x86, no Arm support.
>I think this approach is no-go.
>
> - Don't do it.
>Gandiva leverages LLVM JIT for runtime code optimization. Is it duplicated
> effort to do it in C++ kernel? Will these vetorizable computations move to
> Gandiva in the future?
>
> [1]
> https://github.com/apache/arrow/blob/master/cpp/src/arrow/compute/kernels/sum_internal.h#L104-L106
> [2] https://github.com/intel/isa-l
> [3] https://willnewton.name/2013/07/02/using-gnu-indirect-functions/
> [4] https://lwn.net/Articles/691932/
> [5] https://godbolt.org/z/ajpuq_
>
Re: Trouble with building Arrow GLib
Hi Kou, I am working on [the Cartesius system](https://userinfo.surfsara.nl/systems/cartesius). The OS on it is [bullx Linux](https://userinfo.surfsara.nl/systems/cartesius/software/rhel). I’ve successfully installed Meson according to your suggestion, however it couldn’t run, probably due to some compatibility issues on the system. (I’ve written an email to the Helpdesk for help.) In the meantime of waiting for their response, I would like to know if it is possible to do some modification in the source code of the current Apache Arrow release to disable building GObject Introspection, instead of waiting for the next release. Kind Regards Chengxin Sent with ProtonMail Secure Email. ‐‐‐ Original Message ‐‐‐ On Friday, December 20, 2019 4:49 AM, Sutou Kouhei wrote: > Hi, > > Arrow GLib can provide C API but it requires GObject > Introspection for now. So you need to install GObject > Introspection to use Arrow GLib for now. > (I'll add a build option to build without GObject > Introspection. Then you can build Arrow GLib without GObject > Introspection in the next release.) > > Could you show your environment? Are you using a Linux distribution? > > You can install Meson by "pip install --user meson". > See also: https://mesonbuild.com/Getting-meson.html#installing-meson-with-pip > > Thanks, > > --- > > kou > > In > DIVG0Hqiw9iory_bs1T6A_cf5etWsPJ0-lbAFxaJ4H2hrTm87EGUED3ztGenYN6EjVzW5_oYL1KIH4V3F_BE1dMQyu4EtbBflin-m-iGC_Q=@protonmail.ch > "Trouble with building Arrow GLib" on Thu, 19 Dec 2019 10:53:48 +, > Chengxin Ma c...@protonmail.ch.INVALID wrote: > > > Hi All, > > I am building Arrow GLib on a system where I'm not the admin. > > In the installation instructions > > (https://github.com/apache/arrow/tree/master/c_glib) there are two options > > for building GLib: "How to build by users" and "How to build by > > developers". I followed the "by users" section and met the following > > problem: > > "checking for gobject-introspection... configure: error: > > gobject-introspection-1.0 is not installed". > > "GObject Introspection" is mentioned explicitly in the "How to build by > > developers" section. Does this indicate it is optional if we build as a > > user? > > I checked "./configure --help" but didn't find a way to exclude it in the > > build process. > > I've also tried to build GObject Introspection > > (http://www.linuxfromscratch.org/blfs/view/svn/general/gobject-introspection.html) > > but Meson isn't available on the system... > > Could someone please offer a suggestion for me to build Arrow GLib? > > Kind Regards > > Chengxin > > Sent with ProtonMail Secure Email.
[jira] [Created] (ARROW-7447) [Java] ComplexCopier does incorrect copy in some cases
Projjal Chanda created ARROW-7447: - Summary: [Java] ComplexCopier does incorrect copy in some cases Key: ARROW-7447 URL: https://issues.apache.org/jira/browse/ARROW-7447 Project: Apache Arrow Issue Type: Bug Components: Java Reporter: Projjal Chanda Assignee: Projjal Chanda For list union vectors complex copier doesn't do correct copy. -- This message was sent by Atlassian Jira (v8.3.4#803005)
