[ https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742517#comment-16742517 ]
Bryan Cutler commented on SPARK-26591: -------------------------------------- I created the same virtual environment and could not reproduce, can anyone else verify? OS: 4.15.0-43-generic #46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux {noformat} Welcome to ____ __ / __/__ ___ _____/ /__ _\ \/ _ \/ _ `/ __/ '_/ /__ / .__/\_,_/_/ /_/\_\ version 2.4.0 /_/ Using Python version 3.6.7 (default, Nov 21 2018 02:32:25) SparkSession available as 'spark'. >>> import pyarrow >>> pyarrow.__version__ '0.11.1' >>> import pandas >>> pandas.__version__ '0.23.4' >>> import numpy >>> numpy.__version__ '1.15.4' >>> from pyspark.sql.functions import pandas_udf, PandasUDFType >>> from pyspark.sql.types import IntegerType, StringType >>> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) >>> slen <function <lambda> at 0x7f099a99cd90> {noformat} > Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain > environment > ------------------------------------------------------------------------------------ > > Key: SPARK-26591 > URL: https://issues.apache.org/jira/browse/SPARK-26591 > Project: Spark > Issue Type: Bug > Components: PySpark > Affects Versions: 2.4.0 > Environment: Python 3.6.7 > Pyspark 2.4.0 > OS: > {noformat} > Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 > x86_64 x86_64 GNU/Linux{noformat} > CPU: > > {code:java} > Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB > clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz > {code} > > > Reporter: Elchin > Priority: Major > Attachments: core > > > When I try to use pandas_udf from examples in > [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]: > {code:java} > from pyspark.sql.functions import pandas_udf, PandasUDFType > from pyspark.sql.types import IntegerType, StringType > slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is > crashed{code} > I get the error: > {code:java} > [1] 17969 illegal hardware instruction (core dumped) python3{code} > The environment is: > Python 3.6.7 > PySpark 2.4.0 > PyArrow: 0.11.1 > Pandas: 0.23.4 > NumPy: 1.15.4 > OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 > x86_64 x86_64 x86_64 GNU/Linux -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org