[
https://issues.apache.org/jira/browse/ARROW-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wes McKinney updated ARROW-2872:
--------------------------------
Description:
After pulling in ARROW-1744, I hit this error:
{code}
pyarrow/tests/test_plasma_tf_op.py::test_plasma_tf_op FAILED
[ 82%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
TensorFlow version: 1.8.0
Compiling Plasma TensorFlow Op...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+++ dirname /home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh
++ cd /home/wesm/code/arrow/python/pyarrow/tensorflow
++ pwd
+ PYARROW_TENSORFLOW_DIR=/home/wesm/code/arrow/python/pyarrow/tensorflow
++ python -c 'import tensorflow as tf; print("
".join(tf.sysconfig.get_compile_flags()))'
+
TF_CFLAGS='-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include
-D_GLIBCXX_USE_CXX11_ABI=0'
++ python -c 'import tensorflow as tf; print("
".join(tf.sysconfig.get_link_flags()))'
+
TF_LFLAGS='-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow
-ltensorflow_framework'
++ uname
+ '[' Linux == Darwin ']'
+ NDEBUG=-DNDEBUG
++ pkg-config --cflags --libs plasma arrow arrow-python
+ g++ -std=c++11 -g -shared
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc -o
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.so -DNDEBUG
-I/home/wesm/local/include
-I/home/wesm/miniconda/envs/arrow-dev/include/python3.6m
-I/home/wesm/local/include -L/home/wesm/local/lib -lplasma -larrow_python
-larrow -fPIC
-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include
-D_GLIBCXX_USE_CXX11_ABI=0
-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow
-ltensorflow_framework -O2
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc:33:10: fatal
error: arrow/adapters/tensorflow/convert.h: No such file or directory
#include "arrow/adapters/tensorflow/convert.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
use_gpu = False
@pytest.mark.plasma
def test_plasma_tf_op(use_gpu=False):
import pyarrow.plasma as plasma
> plasma.build_plasma_tensorflow_op()
pyarrow/tests/test_plasma_tf_op.py:89:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
pyarrow/plasma.py:56: in build_plasma_tensorflow_op
subprocess.check_call(["bash", script_path])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
popenargs = (['bash',
'/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh'],)
kwargs = {}, retcode = 1
cmd = ['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']
def check_call(*popenargs, **kwargs):
"""Run command with arguments. Wait for command to complete. If
the exit code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the
return code in the returncode attribute.
The arguments are the same as for the call function. Example:
check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
> raise CalledProcessError(retcode, cmd)
E subprocess.CalledProcessError: Command '['bash',
'/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']' returned non-zero
exit status 1.
../../../miniconda/envs/arrow-dev/lib/python3.6/subprocess.py:291:
CalledProcessError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/miniconda/envs/arrow-dev/lib/python3.6/subprocess.py(291)check_call()
-> raise CalledProcessError(retcode, cmd)
{code}
I misdiagnosed the problem originally (which was my stale Arrow install), but I
think it would be good to have a separate pytest mark for TensorFlow-related
unit tests since they are also time consuming. The unit test in question now
has both the plasma and tensorflow marks
was:
After pulling in ARROW-1744, I found this a little bit unfriendly:
{code}
pyarrow/tests/test_plasma_tf_op.py::test_plasma_tf_op FAILED
[ 82%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stdout
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
TensorFlow version: 1.8.0
Compiling Plasma TensorFlow Op...
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
+++ dirname /home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh
++ cd /home/wesm/code/arrow/python/pyarrow/tensorflow
++ pwd
+ PYARROW_TENSORFLOW_DIR=/home/wesm/code/arrow/python/pyarrow/tensorflow
++ python -c 'import tensorflow as tf; print("
".join(tf.sysconfig.get_compile_flags()))'
+
TF_CFLAGS='-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include
-D_GLIBCXX_USE_CXX11_ABI=0'
++ python -c 'import tensorflow as tf; print("
".join(tf.sysconfig.get_link_flags()))'
+
TF_LFLAGS='-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow
-ltensorflow_framework'
++ uname
+ '[' Linux == Darwin ']'
+ NDEBUG=-DNDEBUG
++ pkg-config --cflags --libs plasma arrow arrow-python
+ g++ -std=c++11 -g -shared
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc -o
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.so -DNDEBUG
-I/home/wesm/local/include
-I/home/wesm/miniconda/envs/arrow-dev/include/python3.6m
-I/home/wesm/local/include -L/home/wesm/local/lib -lplasma -larrow_python
-larrow -fPIC
-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include
-D_GLIBCXX_USE_CXX11_ABI=0
-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow
-ltensorflow_framework -O2
/home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc:33:10: fatal
error: arrow/adapters/tensorflow/convert.h: No such file or directory
#include "arrow/adapters/tensorflow/convert.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
use_gpu = False
@pytest.mark.plasma
def test_plasma_tf_op(use_gpu=False):
import pyarrow.plasma as plasma
> plasma.build_plasma_tensorflow_op()
pyarrow/tests/test_plasma_tf_op.py:89:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
pyarrow/plasma.py:56: in build_plasma_tensorflow_op
subprocess.check_call(["bash", script_path])
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
_ _ _
popenargs = (['bash',
'/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh'],)
kwargs = {}, retcode = 1
cmd = ['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']
def check_call(*popenargs, **kwargs):
"""Run command with arguments. Wait for command to complete. If
the exit code was zero then return, otherwise raise
CalledProcessError. The CalledProcessError object will have the
return code in the returncode attribute.
The arguments are the same as for the call function. Example:
check_call(["ls", "-l"])
"""
retcode = call(*popenargs, **kwargs)
if retcode:
cmd = kwargs.get("args")
if cmd is None:
cmd = popenargs[0]
> raise CalledProcessError(retcode, cmd)
E subprocess.CalledProcessError: Command '['bash',
'/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']' returned non-zero
exit status 1.
../../../miniconda/envs/arrow-dev/lib/python3.6/subprocess.py:291:
CalledProcessError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /home/wesm/miniconda/envs/arrow-dev/lib/python3.6/subprocess.py(291)check_call()
-> raise CalledProcessError(retcode, cmd)
{code}
If you pass {{-DARROW_PLASMA=ON}} but do not also pass
{{-DARROW_TENSORFLOW=ON}} then this occurs
> [Python] Add pytest mark to opt into TensorFlow-related unit tests
> ------------------------------------------------------------------
>
> Key: ARROW-2872
> URL: https://issues.apache.org/jira/browse/ARROW-2872
> Project: Apache Arrow
> Issue Type: Bug
> Components: Python
> Reporter: Wes McKinney
> Assignee: Wes McKinney
> Priority: Major
> Fix For: 0.10.0
>
>
> After pulling in ARROW-1744, I hit this error:
> {code}
> pyarrow/tests/test_plasma_tf_op.py::test_plasma_tf_op FAILED
> [ 82%]
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stdout
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> TensorFlow version: 1.8.0
> Compiling Plasma TensorFlow Op...
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> captured stderr
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> +++ dirname /home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh
> ++ cd /home/wesm/code/arrow/python/pyarrow/tensorflow
> ++ pwd
> + PYARROW_TENSORFLOW_DIR=/home/wesm/code/arrow/python/pyarrow/tensorflow
> ++ python -c 'import tensorflow as tf; print("
> ".join(tf.sysconfig.get_compile_flags()))'
> +
> TF_CFLAGS='-I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include
> -D_GLIBCXX_USE_CXX11_ABI=0'
> ++ python -c 'import tensorflow as tf; print("
> ".join(tf.sysconfig.get_link_flags()))'
> +
> TF_LFLAGS='-L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow
> -ltensorflow_framework'
> ++ uname
> + '[' Linux == Darwin ']'
> + NDEBUG=-DNDEBUG
> ++ pkg-config --cflags --libs plasma arrow arrow-python
> + g++ -std=c++11 -g -shared
> /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc -o
> /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.so -DNDEBUG
> -I/home/wesm/local/include
> -I/home/wesm/miniconda/envs/arrow-dev/include/python3.6m
> -I/home/wesm/local/include -L/home/wesm/local/lib -lplasma -larrow_python
> -larrow -fPIC
> -I/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow/include
> -D_GLIBCXX_USE_CXX11_ABI=0
> -L/home/wesm/miniconda/envs/arrow-dev/lib/python3.6/site-packages/tensorflow
> -ltensorflow_framework -O2
> /home/wesm/code/arrow/python/pyarrow/tensorflow/plasma_op.cc:33:10: fatal
> error: arrow/adapters/tensorflow/convert.h: No such file or directory
> #include "arrow/adapters/tensorflow/convert.h"
> ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> compilation terminated.
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> use_gpu = False
> @pytest.mark.plasma
> def test_plasma_tf_op(use_gpu=False):
> import pyarrow.plasma as plasma
>
> > plasma.build_plasma_tensorflow_op()
> pyarrow/tests/test_plasma_tf_op.py:89:
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _
> pyarrow/plasma.py:56: in build_plasma_tensorflow_op
> subprocess.check_call(["bash", script_path])
> _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _
> popenargs = (['bash',
> '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh'],)
> kwargs = {}, retcode = 1
> cmd = ['bash', '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']
> def check_call(*popenargs, **kwargs):
> """Run command with arguments. Wait for command to complete. If
> the exit code was zero then return, otherwise raise
> CalledProcessError. The CalledProcessError object will have the
> return code in the returncode attribute.
>
> The arguments are the same as for the call function. Example:
>
> check_call(["ls", "-l"])
> """
> retcode = call(*popenargs, **kwargs)
> if retcode:
> cmd = kwargs.get("args")
> if cmd is None:
> cmd = popenargs[0]
> > raise CalledProcessError(retcode, cmd)
> E subprocess.CalledProcessError: Command '['bash',
> '/home/wesm/code/arrow/python/pyarrow/tensorflow/build.sh']' returned
> non-zero exit status 1.
> ../../../miniconda/envs/arrow-dev/lib/python3.6/subprocess.py:291:
> CalledProcessError
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB
> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> > /home/wesm/miniconda/envs/arrow-dev/lib/python3.6/subprocess.py(291)check_call()
> -> raise CalledProcessError(retcode, cmd)
> {code}
> I misdiagnosed the problem originally (which was my stale Arrow install), but
> I think it would be good to have a separate pytest mark for
> TensorFlow-related unit tests since they are also time consuming. The unit
> test in question now has both the plasma and tensorflow marks
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
