[jira] [Commented] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-17 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744823#comment-16744823
 ] 

Elchin commented on SPARK-26591:


[~bryanc] yes, I have already done it:

https://issues.apache.org/jira/browse/ARROW-4272

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-26635) illegal hardware instruction

2019-01-16 Thread Elchin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elchin resolved SPARK-26635.

Resolution: Fixed

> illegal hardware instruction
> 
>
> Key: SPARK-26635
> URL: https://issues.apache.org/jira/browse/SPARK-26635
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, Spark Core
>Affects Versions: 2.4.0
>Reporter: Elchin
>Priority: Major
>
> I can't import pyarrow:
> {code:java}
> >>> import pyarrow as pa
> [1]    31441 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Closed] (SPARK-26635) illegal hardware instruction

2019-01-16 Thread Elchin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elchin closed SPARK-26635.
--

> illegal hardware instruction
> 
>
> Key: SPARK-26635
> URL: https://issues.apache.org/jira/browse/SPARK-26635
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark, Spark Core
>Affects Versions: 2.4.0
>Reporter: Elchin
>Priority: Major
>
> I can't import pyarrow:
> {code:java}
> >>> import pyarrow as pa
> [1]    31441 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-26635) illegal hardware instruction

2019-01-16 Thread Elchin (JIRA) 
Elchin created SPARK-26635:
--

 Summary: illegal hardware instruction
 Key: SPARK-26635
 URL: https://issues.apache.org/jira/browse/SPARK-26635
 Project: Spark
  Issue Type: Bug
  Components: PySpark, Spark Core
Affects Versions: 2.4.0
Reporter: Elchin


I can't import pyarrow:
{code:java}
>>> import pyarrow as pa
[1]    31441 illegal hardware instruction (core dumped)  python3{code}
The environment is:

Python 3.6.7
 PySpark 2.4.0
 PyArrow: 0.11.1
 Pandas: 0.23.4
 NumPy: 1.15.4
 OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Closed] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-16 Thread Elchin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elchin closed SPARK-26591.
--

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Resolved] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-16 Thread Elchin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elchin resolved SPARK-26591.

Resolution: Feedback Received

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-16 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743758#comment-16743758
 ] 

Elchin commented on SPARK-26591:


[~bryanc] I even can't import pyarrow:
{code:java}
>>> import pyarrow as pa
[1]    31441 illegal hardware instruction (core dumped)  python3
{code}

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-15 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743316#comment-16743316
 ] 

Elchin commented on SPARK-26591:


[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without {{ssse3, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with }}{{support}}{{ of my cpu architecture?}}

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-15 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743316#comment-16743316
 ] 

Elchin edited comment on SPARK-26591 at 1/15/19 7:17 PM:
-

[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without _ssse3_, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with support of my cpu architecture?


was (Author: elch10):
[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without {{ssse3, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with support of my cpu architecture?

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-15 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743316#comment-16743316
 ] 

Elchin edited comment on SPARK-26591 at 1/15/19 7:16 PM:
-

[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without {{ssse3, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with support }}of my cpu architecture?


was (Author: elch10):
[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without {{ssse3, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with }}{{support}}{{ of my cpu architecture?}}

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-15 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743316#comment-16743316
 ] 

Elchin edited comment on SPARK-26591 at 1/15/19 7:16 PM:
-

[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without {{ssse3, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with support of my cpu architecture?


was (Author: elch10):
[~hyukjin.kwon] I have had similar problem with _catboost_ and there I have 
fixed problem by compiling source without {{ssse3, because my cpu architecture 
doesn't support it. Can I somehow compile _arrow_ or something that relates to 
_pandas_udf_ with support }}of my cpu architecture?

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-15 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742990#comment-16742990
 ] 

Elchin commented on SPARK-26591:


[~bryanc] there is no output, except of _illegal hardware instruction._ But 
I've already attached the [^core] file. And you can debug it with
{code:java}
gdb python3 core
{code}

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-15 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16742963#comment-16742963
 ] 

Elchin commented on SPARK-26591:


[~bryanc] may be there is the problem with my cpu architecture? Can I build 
pyspark or pyarrow with specific build options to make it work?

> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26591) Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain environment

2019-01-14 Thread Elchin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elchin updated SPARK-26591:
---
Description: 
When I try to use pandas_udf from examples in 
[documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
{code:java}
from pyspark.sql.functions import pandas_udf, PandasUDFType

from pyspark.sql.types import IntegerType, StringType

slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
crashed{code}
I get the error:
{code:java}
[1]    17969 illegal hardware instruction (core dumped)  python3{code}
The environment is:

Python 3.6.7
 PySpark 2.4.0
 PyArrow: 0.11.1
 Pandas: 0.23.4
 NumPy: 1.15.4
 OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux

  was:
When I try to use pandas_udf from examples in 
[documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
{code:java}
from pyspark.sql.functions import pandas_udf, PandasUDFType

from pyspark.sql.types import IntegerType, StringType

slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
crashed{code}
I get the error:
{code:java}
[1]    17969 illegal hardware instruction (core dumped)  python3{code}

The environment is:

Python 3.6.7
PySpark 2.4.0
PyArrow: 0.11.1
Pandas:
NumPy:
OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux



> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain 
> environment
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}
> The environment is:
> Python 3.6.7
>  PySpark 2.4.0
>  PyArrow: 0.11.1
>  Pandas: 0.23.4
>  NumPy: 1.15.4
>  OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 
> x86_64 x86_64 x86_64 GNU/Linux



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Comment Edited] (SPARK-26591) illegal hardware instruction

2019-01-12 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741387#comment-16741387
 ] 

Elchin edited comment on SPARK-26591 at 1/12/19 8:00 PM:
-

[~bryanc] I installed it through pip. And I tested it on clean virtual 
environment. And it also crushed

PyArrow version is 0.11.1. I also attached core dump, may be it can help you.


was (Author: elch10):
[~bryanc] I installed it through pip. And I tested it on clean virtual 
environment. And it also doesn't work

PyArrow version is 0.11.1. I also attached core dump, may be it can help you.

> illegal hardware instruction
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Critical
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Updated] (SPARK-26591) illegal hardware instruction

2019-01-12 Thread Elchin (JIRA) 


 [ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Elchin updated SPARK-26591:
---
Attachment: core

> illegal hardware instruction
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Critical
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Commented] (SPARK-26591) illegal hardware instruction

2019-01-12 Thread Elchin (JIRA) 


[ 
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16741387#comment-16741387
 ] 

Elchin commented on SPARK-26591:


[~bryanc] I installed it through pip. And I tested it on clean virtual 
environment. And it also doesn't work

PyArrow version is 0.11.1. I also attached core dump, may be it can help you.

> illegal hardware instruction
> 
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
>  Issue Type: Bug
>  Components: PySpark
>Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>  
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>  
>  
>Reporter: Elchin
>Priority: Critical
> Attachments: core
>
>
> When I try to use pandas_udf from examples in 
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
> crashed{code}
> I get the error:
> {code:java}
> [1]    17969 illegal hardware instruction (core dumped)  python3{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org

[jira] [Created] (SPARK-26591) illegal hardware instruction

2019-01-10 Thread Elchin (JIRA) 
Elchin created SPARK-26591:
--

 Summary: illegal hardware instruction
 Key: SPARK-26591
 URL: https://issues.apache.org/jira/browse/SPARK-26591
 Project: Spark
  Issue Type: Bug
  Components: PySpark
Affects Versions: 2.4.0
 Environment: Python 3.6.7

Pyspark 2.4.0

OS:
{noformat}
Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64 
x86_64 x86_64 GNU/Linux{noformat}
CPU:

 
{code:java}
Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
{code}
 

 
Reporter: Elchin


When I try to use pandas_udf from examples in 
[documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
{code:java}
from pyspark.sql.functions import pandas_udf, PandasUDFType

from pyspark.sql.types import IntegerType, StringType

slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is 
crashed{code}
I get the error:
{code:java}
[1]    17969 illegal hardware instruction (core dumped)  python3{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org