[jira] [Commented] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14979023#comment-14979023 ] watson xi commented on SPARK-5: --- I keep having this problem myself. It only happens itermitently and I have no idea why. Not accessing any remote hosts. > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981531#comment-14981531 ] watson xi commented on SPARK-5: --- FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost }} by prepending with {{#}} > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > Labels: starter > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981531#comment-14981531 ] watson xi edited comment on SPARK-5 at 10/29/15 11:05 PM: -- FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost}} by prepending with {{#}} was (Author: watsonix): FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost }} by prepending with {{#}} > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > Labels: starter > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981531#comment-14981531 ] watson xi edited comment on SPARK-5 at 10/29/15 11:04 PM: -- FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost }} by prepending with {{#}} was (Author: watsonix): FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost }} by prepending with {{#}} > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > Labels: starter > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14981531#comment-14981531 ] watson xi edited comment on SPARK-5 at 10/29/15 11:05 PM: -- FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost}} by prepending with {{#}} this removes the IPV6 reference to localhost but leaves the IPV4 one in place was (Author: watsonix): FYI... i figured out I have the issue when I am disconnected from network/internet. the fix for me is: comment out the following line in my /etc/hosts file: {{::1 localhost}} by prepending with {{#}} > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > Labels: starter > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14985985#comment-14985985 ] watson xi edited comment on SPARK-5 at 11/2/15 8:39 PM: started getting some intermittent socket connection errors after commenting out that line. a friends machine had the following IPV6 localhost line, which I copied and all seems to be well now: {{fe80::1%lo0 localhost}} was (Author: watsonix): started getting some intermittent socket connection errors after commenting out that line. a friends machine had the following IPV6 localhost line, which I copied and all seems to be well now: `fe80::1%lo0 localhost` > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > Labels: starter > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-11115) Host verification is not correct for IPv6
[ https://issues.apache.org/jira/browse/SPARK-5?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14985985#comment-14985985 ] watson xi commented on SPARK-5: --- started getting some intermittent socket connection errors after commenting out that line. a friends machine had the following IPV6 localhost line, which I copied and all seems to be well now: `fe80::1%lo0 localhost` > Host verification is not correct for IPv6 > - > > Key: SPARK-5 > URL: https://issues.apache.org/jira/browse/SPARK-5 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 1.5.1 > Environment: CentOS 6.7, Java 1.8.0_25, dual stack IPv4 + IPv6 >Reporter: Thomas Dudziak >Priority: Critical > Labels: starter > > When running Spark with -Djava.net.preferIPv6Addresses=true, I get this error: > 15/10/14 14:36:01 ERROR SparkContext: Error initializing SparkContext. > java.lang.AssertionError: assertion failed: Expected hostname > at scala.Predef$.assert(Predef.scala:179) > at org.apache.spark.util.Utils$.checkHost(Utils.scala:805) > at > org.apache.spark.storage.BlockManagerId.(BlockManagerId.scala:48) > at > org.apache.spark.storage.BlockManagerId$.apply(BlockManagerId.scala:107) > at > org.apache.spark.storage.BlockManager.initialize(BlockManager.scala:190) > at org.apache.spark.SparkContext.(SparkContext.scala:528) > at > org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:1017) > Looking at the code in question, it seems that the code will only work for > IPv4 as it assumes ':' can't be part of the hostname (which it clearly can > for IPv6 addresses). > Instead, the code should probably use Guava's HostAndPort class, i.e.: > def checkHost(host: String, message: String = "") { > assert(!HostAndPort.fromString(host).hasPort, message) > } > def checkHostPort(hostPort: String, message: String = "") { > assert(HostAndPort.fromString(hostPort).hasPort, message) > } -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4897) Python 3 support
[ https://issues.apache.org/jira/browse/SPARK-4897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14495152#comment-14495152 ] watson xi commented on SPARK-4897: -- Hi guys, whats the status of this project? I know a few people (including myself) who are ready to wave goodbye to Python 2 (its been 6.5 years now!)... from an outside perspective looking it, Python 3 compatibility appears close! > Python 3 support > > > Key: SPARK-4897 > URL: https://issues.apache.org/jira/browse/SPARK-4897 > Project: Spark > Issue Type: Improvement > Components: PySpark >Reporter: Josh Rosen >Assignee: Davies Liu >Priority: Minor > > It would be nice to have Python 3 support in PySpark, provided that we can do > it in a way that maintains backwards-compatibility with Python 2.6. > I started looking into porting this; my WIP work can be found at > https://github.com/JoshRosen/spark/compare/python3 > I was able to use the > [futurize|http://python-future.org/futurize.html#forwards-conversion-stage1] > tool to handle the basic conversion of things like {{print}} statements, etc. > and had to manually fix up a few imports for packages that moved / were > renamed, but the major blocker that I hit was {{cloudpickle}}: > {code} > [joshrosen python (python3)]$ PYSPARK_PYTHON=python3 ../bin/pyspark > Python 3.4.2 (default, Oct 19 2014, 17:52:17) > [GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.51)] on darwin > Type "help", "copyright", "credits" or "license" for more information. > Traceback (most recent call last): > File "/Users/joshrosen/Documents/Spark/python/pyspark/shell.py", line 28, > in > import pyspark > File "/Users/joshrosen/Documents/spark/python/pyspark/__init__.py", line > 41, in > from pyspark.context import SparkContext > File "/Users/joshrosen/Documents/spark/python/pyspark/context.py", line 26, > in > from pyspark import accumulators > File "/Users/joshrosen/Documents/spark/python/pyspark/accumulators.py", > line 97, in > from pyspark.cloudpickle import CloudPickler > File "/Users/joshrosen/Documents/spark/python/pyspark/cloudpickle.py", line > 120, in > class CloudPickler(pickle.Pickler): > File "/Users/joshrosen/Documents/spark/python/pyspark/cloudpickle.py", line > 122, in CloudPickler > dispatch = pickle.Pickler.dispatch.copy() > AttributeError: type object '_pickle.Pickler' has no attribute 'dispatch' > {code} > This code looks like it will be hard difficult to port to Python 3, so this > might be a good reason to switch to > [Dill|https://github.com/uqfoundation/dill] for Python serialization. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org