[jira] [Closed] (SPARK-17070) Zookeeper server refused to accept the client (mesos-master)
[ https://issues.apache.org/jira/browse/SPARK-17070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anh Nguyen closed SPARK-17070. -- > Zookeeper server refused to accept the client (mesos-master) > > > Key: SPARK-17070 > URL: https://issues.apache.org/jira/browse/SPARK-17070 > Project: Spark > Issue Type: Bug >Reporter: Anh Nguyen > > I started zookepper server: ./bin/zkServer.sh start-foreground > conf/cnf_zoo.cfg > and this is zookeeper server log : > ZooKeeper JMX enabled by default > Using config: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,767 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading > configuration from: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,778 [myid:] - INFO [main:DatadirCleanupManager@78] - > autopurge.snapRetainCount set to 3 > 2016-08-16 02:59:20,779 [myid:] - INFO [main:DatadirCleanupManager@79] - > autopurge.purgeInterval set to 0 > 2016-08-16 02:59:20,780 [myid:] - INFO [main:DatadirCleanupManager@101] - > Purge task is not scheduled. > 2016-08-16 02:59:20,781 [myid:] - WARN [main:QuorumPeerMain@113] - Either no > config or no quorum defined in config, running in standalone mode > 2016-08-16 02:59:20,803 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading > configuration from: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,804 [myid:] - INFO [main:ZooKeeperServerMain@95] - > Starting server > 2016-08-16 02:59:20,826 [myid:] - INFO [main:Environment@100] - Server > environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT > 2016-08-16 02:59:20,828 [myid:] - INFO [main:Environment@100] - Server > environment:host.name=zookeeper > 2016-08-16 02:59:20,829 [myid:] - INFO [main:Environment@100] - Server > environment:java.version=1.8.0_91 > 2016-08-16 02:59:20,830 [myid:] - INFO [main:Environment@100] - Server > environment:java.vendor=Oracle Corporation > 2016-08-16 02:59:20,831 [myid:] - INFO [main:Environment@100] - Server > environment:java.home=/usr/lib/jvm/java-8-oracle/jre > 2016-08-16 02:59:20,834 [myid:] - INFO [main:Environment@100] - Server > environment:java.class.path=/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/classes:/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/netty-3.7.0.Final.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/log4j-1.2.16.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../src/java/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../conf: > 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server > environment:java.io.tmpdir=/tmp > 2016-08-16 02:59:20,836 [myid:] - INFO [main:Environment@100] - Server > environment:java.compiler= > 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server > environment:os.name=Linux > 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server > environment:os.arch=amd64 > 2016-08-16 02:59:20,844 [myid:] - INFO [main:Environment@100] - Server > environment:os.version=3.13.0-92-generic > 2016-08-16 02:59:20,845 [myid:] - INFO [main:Environment@100] - Server > environment:user.name=vagrant > 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server > environment:user.home=/home/vagrant > 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server > environment:user.dir=/home/vagrant/mesos/zookeeper-3.4.8 > 2016-08-16 02:59:20,861 [myid:] - INFO [main:ZooKeeperServer@787] - tickTime > set to 2000 > 2016-08-16 02:59:20,863 [myid:] - INFO [main:ZooKeeperServer@796] - > minSessionTimeout set to -1 > 2016-08-16 02:59:20,864 [myid:] - INFO [main:ZooKeeperServer@805] - > maxSessionTimeout set to -1 > 2016-08-16 02:59:20,883 [myid:] - INFO [main:NIOServerCnxnFactory@89] - > binding to port 0.0.0.0/0.0.0.0:2181 > But, when I start mesos-master: mesos-master > --zk=zk://192.168.33.10:2128/home/vagrant/mesos/zookeeper-3.4.8 --port=5050 > --ip=192.168.33.20 --quorum=1 --work_dir=/home/vagrant/mesos/master1/work > --log_dir=/home/vagrant/mesos/master1/logs --advertise_ip=192.168.33.20 > --advertise_port=5050 --cluster=mesut-ozil --logging_level=ERROR > This is log mesos-master: zookeeper refaused to connect the mesos-master. > 2016-08-16 03:29:09,052:2107(0x7f93e2de4700):ZOO_INFO@log_env@747: Client > environment:user.name=vagrant > 2016-08-16 03:29:09,055:2107(0x7f93e2de4700):ZOO_INFO@log_env@755: Client > environment:user.home=/home/vagrant >
[jira] [Commented] (SPARK-17070) Zookeeper server refused to accept the client (mesos-master)
[ https://issues.apache.org/jira/browse/SPARK-17070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15426087#comment-15426087 ] Anh Nguyen commented on SPARK-17070: sorry [~c...@koeninger.org] and [~sowen] so to disturb you. I resolve this issue. > Zookeeper server refused to accept the client (mesos-master) > > > Key: SPARK-17070 > URL: https://issues.apache.org/jira/browse/SPARK-17070 > Project: Spark > Issue Type: Bug >Reporter: Anh Nguyen > > I started zookepper server: ./bin/zkServer.sh start-foreground > conf/cnf_zoo.cfg > and this is zookeeper server log : > ZooKeeper JMX enabled by default > Using config: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,767 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading > configuration from: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,778 [myid:] - INFO [main:DatadirCleanupManager@78] - > autopurge.snapRetainCount set to 3 > 2016-08-16 02:59:20,779 [myid:] - INFO [main:DatadirCleanupManager@79] - > autopurge.purgeInterval set to 0 > 2016-08-16 02:59:20,780 [myid:] - INFO [main:DatadirCleanupManager@101] - > Purge task is not scheduled. > 2016-08-16 02:59:20,781 [myid:] - WARN [main:QuorumPeerMain@113] - Either no > config or no quorum defined in config, running in standalone mode > 2016-08-16 02:59:20,803 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading > configuration from: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,804 [myid:] - INFO [main:ZooKeeperServerMain@95] - > Starting server > 2016-08-16 02:59:20,826 [myid:] - INFO [main:Environment@100] - Server > environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT > 2016-08-16 02:59:20,828 [myid:] - INFO [main:Environment@100] - Server > environment:host.name=zookeeper > 2016-08-16 02:59:20,829 [myid:] - INFO [main:Environment@100] - Server > environment:java.version=1.8.0_91 > 2016-08-16 02:59:20,830 [myid:] - INFO [main:Environment@100] - Server > environment:java.vendor=Oracle Corporation > 2016-08-16 02:59:20,831 [myid:] - INFO [main:Environment@100] - Server > environment:java.home=/usr/lib/jvm/java-8-oracle/jre > 2016-08-16 02:59:20,834 [myid:] - INFO [main:Environment@100] - Server > environment:java.class.path=/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/classes:/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/netty-3.7.0.Final.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/log4j-1.2.16.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../src/java/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../conf: > 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server > environment:java.io.tmpdir=/tmp > 2016-08-16 02:59:20,836 [myid:] - INFO [main:Environment@100] - Server > environment:java.compiler= > 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server > environment:os.name=Linux > 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server > environment:os.arch=amd64 > 2016-08-16 02:59:20,844 [myid:] - INFO [main:Environment@100] - Server > environment:os.version=3.13.0-92-generic > 2016-08-16 02:59:20,845 [myid:] - INFO [main:Environment@100] - Server > environment:user.name=vagrant > 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server > environment:user.home=/home/vagrant > 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server > environment:user.dir=/home/vagrant/mesos/zookeeper-3.4.8 > 2016-08-16 02:59:20,861 [myid:] - INFO [main:ZooKeeperServer@787] - tickTime > set to 2000 > 2016-08-16 02:59:20,863 [myid:] - INFO [main:ZooKeeperServer@796] - > minSessionTimeout set to -1 > 2016-08-16 02:59:20,864 [myid:] - INFO [main:ZooKeeperServer@805] - > maxSessionTimeout set to -1 > 2016-08-16 02:59:20,883 [myid:] - INFO [main:NIOServerCnxnFactory@89] - > binding to port 0.0.0.0/0.0.0.0:2181 > But, when I start mesos-master: mesos-master > --zk=zk://192.168.33.10:2128/home/vagrant/mesos/zookeeper-3.4.8 --port=5050 > --ip=192.168.33.20 --quorum=1 --work_dir=/home/vagrant/mesos/master1/work > --log_dir=/home/vagrant/mesos/master1/logs --advertise_ip=192.168.33.20 > --advertise_port=5050 --cluster=mesut-ozil --logging_level=ERROR > This is log mesos-master: zookeeper refaused to connect the mesos-master. > 2016-08-16 03:29:09,052:2107(0x7f93e2de4700):ZOO_INFO@log_env@747: Client >
[jira] [Commented] (SPARK-17070) Zookeeper server refused to accept the client (mesos-master)
[ https://issues.apache.org/jira/browse/SPARK-17070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15425847#comment-15425847 ] Anh Nguyen commented on SPARK-17070: hi [~c...@koeninger.org] and [~srowen] if you free time. Please, help me resolve this issue. > Zookeeper server refused to accept the client (mesos-master) > > > Key: SPARK-17070 > URL: https://issues.apache.org/jira/browse/SPARK-17070 > Project: Spark > Issue Type: Bug >Reporter: Anh Nguyen > > I started zookepper server: ./bin/zkServer.sh start-foreground > conf/cnf_zoo.cfg > and this is zookeeper server log : > ZooKeeper JMX enabled by default > Using config: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,767 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading > configuration from: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,778 [myid:] - INFO [main:DatadirCleanupManager@78] - > autopurge.snapRetainCount set to 3 > 2016-08-16 02:59:20,779 [myid:] - INFO [main:DatadirCleanupManager@79] - > autopurge.purgeInterval set to 0 > 2016-08-16 02:59:20,780 [myid:] - INFO [main:DatadirCleanupManager@101] - > Purge task is not scheduled. > 2016-08-16 02:59:20,781 [myid:] - WARN [main:QuorumPeerMain@113] - Either no > config or no quorum defined in config, running in standalone mode > 2016-08-16 02:59:20,803 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading > configuration from: conf/cnf_zoo.cfg > 2016-08-16 02:59:20,804 [myid:] - INFO [main:ZooKeeperServerMain@95] - > Starting server > 2016-08-16 02:59:20,826 [myid:] - INFO [main:Environment@100] - Server > environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT > 2016-08-16 02:59:20,828 [myid:] - INFO [main:Environment@100] - Server > environment:host.name=zookeeper > 2016-08-16 02:59:20,829 [myid:] - INFO [main:Environment@100] - Server > environment:java.version=1.8.0_91 > 2016-08-16 02:59:20,830 [myid:] - INFO [main:Environment@100] - Server > environment:java.vendor=Oracle Corporation > 2016-08-16 02:59:20,831 [myid:] - INFO [main:Environment@100] - Server > environment:java.home=/usr/lib/jvm/java-8-oracle/jre > 2016-08-16 02:59:20,834 [myid:] - INFO [main:Environment@100] - Server > environment:java.class.path=/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/classes:/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/netty-3.7.0.Final.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/log4j-1.2.16.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../src/java/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../conf: > 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server > environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib > 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server > environment:java.io.tmpdir=/tmp > 2016-08-16 02:59:20,836 [myid:] - INFO [main:Environment@100] - Server > environment:java.compiler= > 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server > environment:os.name=Linux > 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server > environment:os.arch=amd64 > 2016-08-16 02:59:20,844 [myid:] - INFO [main:Environment@100] - Server > environment:os.version=3.13.0-92-generic > 2016-08-16 02:59:20,845 [myid:] - INFO [main:Environment@100] - Server > environment:user.name=vagrant > 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server > environment:user.home=/home/vagrant > 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server > environment:user.dir=/home/vagrant/mesos/zookeeper-3.4.8 > 2016-08-16 02:59:20,861 [myid:] - INFO [main:ZooKeeperServer@787] - tickTime > set to 2000 > 2016-08-16 02:59:20,863 [myid:] - INFO [main:ZooKeeperServer@796] - > minSessionTimeout set to -1 > 2016-08-16 02:59:20,864 [myid:] - INFO [main:ZooKeeperServer@805] - > maxSessionTimeout set to -1 > 2016-08-16 02:59:20,883 [myid:] - INFO [main:NIOServerCnxnFactory@89] - > binding to port 0.0.0.0/0.0.0.0:2181 > But, when I start mesos-master: mesos-master > --zk=zk://192.168.33.10:2128/home/vagrant/mesos/zookeeper-3.4.8 --port=5050 > --ip=192.168.33.20 --quorum=1 --work_dir=/home/vagrant/mesos/master1/work > --log_dir=/home/vagrant/mesos/master1/logs --advertise_ip=192.168.33.20 > --advertise_port=5050 --cluster=mesut-ozil --logging_level=ERROR > This is log mesos-master: zookeeper refaused to connect the mesos-master. > 2016-08-16 03:29:09,052:2107(0x7f93e2de4700):ZOO_INFO@log_env@747: Client >
[jira] [Updated] (SPARK-17070) Zookeeper server refused to accept the client (mesos-master)
[ https://issues.apache.org/jira/browse/SPARK-17070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anh Nguyen updated SPARK-17070: --- Description: I started zookepper server: ./bin/zkServer.sh start-foreground conf/cnf_zoo.cfg and this is zookeeper server log : ZooKeeper JMX enabled by default Using config: conf/cnf_zoo.cfg 2016-08-16 02:59:20,767 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: conf/cnf_zoo.cfg 2016-08-16 02:59:20,778 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2016-08-16 02:59:20,779 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2016-08-16 02:59:20,780 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2016-08-16 02:59:20,781 [myid:] - WARN [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running in standalone mode 2016-08-16 02:59:20,803 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: conf/cnf_zoo.cfg 2016-08-16 02:59:20,804 [myid:] - INFO [main:ZooKeeperServerMain@95] - Starting server 2016-08-16 02:59:20,826 [myid:] - INFO [main:Environment@100] - Server environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT 2016-08-16 02:59:20,828 [myid:] - INFO [main:Environment@100] - Server environment:host.name=zookeeper 2016-08-16 02:59:20,829 [myid:] - INFO [main:Environment@100] - Server environment:java.version=1.8.0_91 2016-08-16 02:59:20,830 [myid:] - INFO [main:Environment@100] - Server environment:java.vendor=Oracle Corporation 2016-08-16 02:59:20,831 [myid:] - INFO [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-8-oracle/jre 2016-08-16 02:59:20,834 [myid:] - INFO [main:Environment@100] - Server environment:java.class.path=/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/classes:/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/netty-3.7.0.Final.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/log4j-1.2.16.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../src/java/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../conf: 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp 2016-08-16 02:59:20,836 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler= 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64 2016-08-16 02:59:20,844 [myid:] - INFO [main:Environment@100] - Server environment:os.version=3.13.0-92-generic 2016-08-16 02:59:20,845 [myid:] - INFO [main:Environment@100] - Server environment:user.name=vagrant 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/home/vagrant 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/home/vagrant/mesos/zookeeper-3.4.8 2016-08-16 02:59:20,861 [myid:] - INFO [main:ZooKeeperServer@787] - tickTime set to 2000 2016-08-16 02:59:20,863 [myid:] - INFO [main:ZooKeeperServer@796] - minSessionTimeout set to -1 2016-08-16 02:59:20,864 [myid:] - INFO [main:ZooKeeperServer@805] - maxSessionTimeout set to -1 2016-08-16 02:59:20,883 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181 But, when I start mesos-master: mesos-master --zk=zk://192.168.33.10:2128/home/vagrant/mesos/zookeeper-3.4.8 --port=5050 --ip=192.168.33.20 --quorum=1 --work_dir=/home/vagrant/mesos/master1/work --log_dir=/home/vagrant/mesos/master1/logs --advertise_ip=192.168.33.20 --advertise_port=5050 --cluster=mesut-ozil --logging_level=ERROR This is log mesos-master: zookeeper refaused to connect the mesos-master. 2016-08-16 03:29:09,052:2107(0x7f93e2de4700):ZOO_INFO@log_env@747: Client environment:user.name=vagrant 2016-08-16 03:29:09,055:2107(0x7f93e2de4700):ZOO_INFO@log_env@755: Client environment:user.home=/home/vagrant 2016-08-16 03:29:09,055:2107(0x7f93e2de4700):ZOO_INFO@log_env@767: Client environment:user.dir=/home/vagrant 2016-08-16 03:29:09,055:2107(0x7f93e2de4700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=192.168.33.10:2128 sessionTimeout=1 watcher=0x7f93ec335340 sessionId=0 sessionPasswd= context=0x7f93c8000930 flags=0 2016-08-16 03:29:09,052:2107(0x7f93e3de6700):ZOO_INFO@log_env@747: Client
[jira] [Created] (SPARK-17070) Zookeeper server refused to accept the client (mesos-master)
Anh Nguyen created SPARK-17070: -- Summary: Zookeeper server refused to accept the client (mesos-master) Key: SPARK-17070 URL: https://issues.apache.org/jira/browse/SPARK-17070 Project: Spark Issue Type: Bug Reporter: Anh Nguyen I started zookepper server: ./bin/zkServer.sh start-foreground conf/cnf_zoo.cfg and this is zookeeper server log : ZooKeeper JMX enabled by default Using config: conf/cnf_zoo.cfg 2016-08-16 02:59:20,767 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: conf/cnf_zoo.cfg 2016-08-16 02:59:20,778 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3 2016-08-16 02:59:20,779 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0 2016-08-16 02:59:20,780 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled. 2016-08-16 02:59:20,781 [myid:] - WARN [main:QuorumPeerMain@113] - Either no config or no quorum defined in config, running in standalone mode 2016-08-16 02:59:20,803 [myid:] - INFO [main:QuorumPeerConfig@103] - Reading configuration from: conf/cnf_zoo.cfg 2016-08-16 02:59:20,804 [myid:] - INFO [main:ZooKeeperServerMain@95] - Starting server 2016-08-16 02:59:20,826 [myid:] - INFO [main:Environment@100] - Server environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT 2016-08-16 02:59:20,828 [myid:] - INFO [main:Environment@100] - Server environment:host.name=zookeeper 2016-08-16 02:59:20,829 [myid:] - INFO [main:Environment@100] - Server environment:java.version=1.8.0_91 2016-08-16 02:59:20,830 [myid:] - INFO [main:Environment@100] - Server environment:java.vendor=Oracle Corporation 2016-08-16 02:59:20,831 [myid:] - INFO [main:Environment@100] - Server environment:java.home=/usr/lib/jvm/java-8-oracle/jre 2016-08-16 02:59:20,834 [myid:] - INFO [main:Environment@100] - Server environment:java.class.path=/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/classes:/home/vagrant/mesos/zookeeper-3.4.8/bin/../build/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-log4j12-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/slf4j-api-1.6.1.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/netty-3.7.0.Final.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/log4j-1.2.16.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../lib/jline-0.9.94.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../zookeeper-3.4.8.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../src/java/lib/*.jar:/home/vagrant/mesos/zookeeper-3.4.8/bin/../conf: 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib 2016-08-16 02:59:20,835 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=/tmp 2016-08-16 02:59:20,836 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler= 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Linux 2016-08-16 02:59:20,842 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64 2016-08-16 02:59:20,844 [myid:] - INFO [main:Environment@100] - Server environment:os.version=3.13.0-92-generic 2016-08-16 02:59:20,845 [myid:] - INFO [main:Environment@100] - Server environment:user.name=vagrant 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server environment:user.home=/home/vagrant 2016-08-16 02:59:20,846 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=/home/vagrant/mesos/zookeeper-3.4.8 2016-08-16 02:59:20,861 [myid:] - INFO [main:ZooKeeperServer@787] - tickTime set to 2000 2016-08-16 02:59:20,863 [myid:] - INFO [main:ZooKeeperServer@796] - minSessionTimeout set to -1 2016-08-16 02:59:20,864 [myid:] - INFO [main:ZooKeeperServer@805] - maxSessionTimeout set to -1 2016-08-16 02:59:20,883 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181 But, when I start mesos-master: mesos-master --zk=zk://192.168.33.10:2128/home/vagrant/mesos/zookeeper-3.4.8 --port=5050 --ip=192.168.33.20 --quorum=1 --work_dir=/home/vagrant/mesos/master1/work --log_dir=/home/vagrant/mesos/master1/logs --advertise_ip=192.168.33.20 --advertise_port=5050 --cluster=mesut-ozil --logging_level=ERROR 2016-08-16 03:29:09,053:2107(0x7f93e25e3700):ZOO_INFO@zookeeper_init@800: Initiating client connection, host=192.168.33.10:2128 sessionTimeout=1 watcher=0x7f93ec335340 sessionId=0 sessionPasswd= context=0x7f93d80047c0 flags=0 2016-08-16 03:29:09,051:2107(0x7f93e35e5700):ZOO_INFO@log_env@747: Client environment:user.name=vagrant 2016-08-16 03:29:09,054:2107(0x7f93e35e5700):ZOO_INFO@log_env@755: Client environment:user.home=/home/vagrant 2016-08-16 03:29:09,054:2107(0x7f93e35e5700):ZOO_INFO@log_env@767: Client environment:user.dir=/home/vagrant 2016-08-16
[jira] [Closed] (SPARK-16762) spark hanging when action method print
[
https://issues.apache.org/jira/browse/SPARK-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anh Nguyen closed SPARK-16762.
--
this issue relate resoure for master and slave
> spark hanging when action method print
> --
>
> Key: SPARK-16762
> URL: https://issues.apache.org/jira/browse/SPARK-16762
> Project: Spark
> Issue Type: Bug
> Components: Deploy
>Reporter: Anh Nguyen
> Attachments: Screen Shot 2016-07-28 at 12.33.35 PM.png, Screen Shot
> 2016-07-28 at 12.34.22 PM.png
>
>
> I write code spark Streaming (consumer) intergate kafaka and deploy on mesos
> FW:
> import org.apache.spark.SparkConf
> import org.apache.spark.streaming._
> import org.apache.spark.streaming.kafka._
> import org.apache.spark.streaming.kafka._
> object consumer {
> def main(args: Array[String]) {
> if (args.length < 4) {
> System.err.println("Usage: KafkaWordCount
> ")
> System.exit(1)
> }
> val Array(zkQuorum, group, topics, numThreads) = args
> val sparkConf = new
> SparkConf().setAppName("KafkaWordCount").set("spark.rpc.netty.dispatcher.numThreads","4")
> val ssc = new StreamingContext(sparkConf, Seconds(2))
> ssc.checkpoint("checkpoint")
> val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
> val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap).map(_._2)
> lines.print()
> val words = lines.flatMap(_.split(" "))
> val wordCounts = words.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _
> - _, Minutes(10), Seconds(2), 2)
> ssc.start()
> ssc.awaitTermination()
> }
> }
> This log:
> I0728 04:28:05.439469 4295 exec.cpp:143] Version: 0.28.2
> I0728 04:28:05.443464 4296 exec.cpp:217] Executor registered on slave
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/07/28 04:28:07 INFO CoarseGrainedExecutorBackend: Registered signal
> handlers for [TERM, HUP, INT]
> 16/07/28 04:28:08 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 16/07/28 04:28:09 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:11 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:12 INFO Slf4jLogger: Slf4jLogger started
> 16/07/28 04:28:12 INFO Remoting: Starting remoting
> 16/07/28 04:28:13 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutorActorSystem@slave1:57380]
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'sparkExecutorActorSystem' on port 57380.
> 16/07/28 04:28:13 INFO DiskBlockManager: Created local directory at
> /home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/blockmgr-4e94f4ea-f074-4dbe-bfa1-34054e6e079c
> 16/07/28 04:28:13 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> spark://CoarseGrainedScheduler@192.168.33.30:36179
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Successfully registered
> with driver
> 16/07/28 04:28:13 INFO Executor: Starting executor ID
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0 on host slave1
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48672.
> 16/07/28 04:28:13 INFO NettyBlockTransferService: Server created on 48672
> 16/07/28 04:28:13 INFO BlockManagerMaster: Trying to register BlockManager
> 16/07/28 04:28:13 INFO BlockManagerMaster: Registered BlockManager
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Got assigned task 0
> 16/07/28 04:28:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 16/07/28 04:28:13 INFO Executor: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> with timestamp 1469680084686
> 16/07/28 04:28:14 INFO Utils: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> to
>
[jira] [Commented] (SPARK-16762) spark hanging when action method print
[
https://issues.apache.org/jira/browse/SPARK-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15399152#comment-15399152
]
Anh Nguyen commented on SPARK-16762:
Thank [~c...@koeninger.org] and [~srowen]
I understood this issue.
It relate distribution to slave and master.
It resolved.
> spark hanging when action method print
> --
>
> Key: SPARK-16762
> URL: https://issues.apache.org/jira/browse/SPARK-16762
> Project: Spark
> Issue Type: Bug
> Components: Deploy
>Reporter: Anh Nguyen
> Attachments: Screen Shot 2016-07-28 at 12.33.35 PM.png, Screen Shot
> 2016-07-28 at 12.34.22 PM.png
>
>
> I write code spark Streaming (consumer) intergate kafaka and deploy on mesos
> FW:
> import org.apache.spark.SparkConf
> import org.apache.spark.streaming._
> import org.apache.spark.streaming.kafka._
> import org.apache.spark.streaming.kafka._
> object consumer {
> def main(args: Array[String]) {
> if (args.length < 4) {
> System.err.println("Usage: KafkaWordCount
> ")
> System.exit(1)
> }
> val Array(zkQuorum, group, topics, numThreads) = args
> val sparkConf = new
> SparkConf().setAppName("KafkaWordCount").set("spark.rpc.netty.dispatcher.numThreads","4")
> val ssc = new StreamingContext(sparkConf, Seconds(2))
> ssc.checkpoint("checkpoint")
> val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
> val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap).map(_._2)
> lines.print()
> val words = lines.flatMap(_.split(" "))
> val wordCounts = words.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _
> - _, Minutes(10), Seconds(2), 2)
> ssc.start()
> ssc.awaitTermination()
> }
> }
> This log:
> I0728 04:28:05.439469 4295 exec.cpp:143] Version: 0.28.2
> I0728 04:28:05.443464 4296 exec.cpp:217] Executor registered on slave
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/07/28 04:28:07 INFO CoarseGrainedExecutorBackend: Registered signal
> handlers for [TERM, HUP, INT]
> 16/07/28 04:28:08 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 16/07/28 04:28:09 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:11 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:12 INFO Slf4jLogger: Slf4jLogger started
> 16/07/28 04:28:12 INFO Remoting: Starting remoting
> 16/07/28 04:28:13 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutorActorSystem@slave1:57380]
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'sparkExecutorActorSystem' on port 57380.
> 16/07/28 04:28:13 INFO DiskBlockManager: Created local directory at
> /home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/blockmgr-4e94f4ea-f074-4dbe-bfa1-34054e6e079c
> 16/07/28 04:28:13 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> spark://CoarseGrainedScheduler@192.168.33.30:36179
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Successfully registered
> with driver
> 16/07/28 04:28:13 INFO Executor: Starting executor ID
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0 on host slave1
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48672.
> 16/07/28 04:28:13 INFO NettyBlockTransferService: Server created on 48672
> 16/07/28 04:28:13 INFO BlockManagerMaster: Trying to register BlockManager
> 16/07/28 04:28:13 INFO BlockManagerMaster: Registered BlockManager
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Got assigned task 0
> 16/07/28 04:28:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 16/07/28 04:28:13 INFO Executor: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> with timestamp 1469680084686
> 16/07/28 04:28:14 INFO Utils: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> to
>
[jira] [Updated] (SPARK-16762) spark hanging when action method print
[
https://issues.apache.org/jira/browse/SPARK-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anh Nguyen updated SPARK-16762:
---
Attachment: Screen Shot 2016-07-28 at 12.34.22 PM.png
> spark hanging when action method print
> --
>
> Key: SPARK-16762
> URL: https://issues.apache.org/jira/browse/SPARK-16762
> Project: Spark
> Issue Type: Bug
> Components: Deploy
>Reporter: Anh Nguyen
> Attachments: Screen Shot 2016-07-28 at 12.33.35 PM.png, Screen Shot
> 2016-07-28 at 12.34.22 PM.png
>
>
> I write code spark Streaming (consumer) intergate kafaka and deploy on mesos
> FW:
> import org.apache.spark.SparkConf
> import org.apache.spark.streaming._
> import org.apache.spark.streaming.kafka._
> import org.apache.spark.streaming.kafka._
> object consumer {
> def main(args: Array[String]) {
> if (args.length < 4) {
> System.err.println("Usage: KafkaWordCount
> ")
> System.exit(1)
> }
> val Array(zkQuorum, group, topics, numThreads) = args
> val sparkConf = new
> SparkConf().setAppName("KafkaWordCount").set("spark.rpc.netty.dispatcher.numThreads","4")
> val ssc = new StreamingContext(sparkConf, Seconds(2))
> ssc.checkpoint("checkpoint")
> val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
> val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap).map(_._2)
> lines.print()
> val words = lines.flatMap(_.split(" "))
> val wordCounts = words.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _
> - _, Minutes(10), Seconds(2), 2)
> ssc.start()
> ssc.awaitTermination()
> }
> }
> This log:
> I0728 04:28:05.439469 4295 exec.cpp:143] Version: 0.28.2
> I0728 04:28:05.443464 4296 exec.cpp:217] Executor registered on slave
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/07/28 04:28:07 INFO CoarseGrainedExecutorBackend: Registered signal
> handlers for [TERM, HUP, INT]
> 16/07/28 04:28:08 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 16/07/28 04:28:09 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:11 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:12 INFO Slf4jLogger: Slf4jLogger started
> 16/07/28 04:28:12 INFO Remoting: Starting remoting
> 16/07/28 04:28:13 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutorActorSystem@slave1:57380]
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'sparkExecutorActorSystem' on port 57380.
> 16/07/28 04:28:13 INFO DiskBlockManager: Created local directory at
> /home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/blockmgr-4e94f4ea-f074-4dbe-bfa1-34054e6e079c
> 16/07/28 04:28:13 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> spark://CoarseGrainedScheduler@192.168.33.30:36179
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Successfully registered
> with driver
> 16/07/28 04:28:13 INFO Executor: Starting executor ID
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0 on host slave1
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48672.
> 16/07/28 04:28:13 INFO NettyBlockTransferService: Server created on 48672
> 16/07/28 04:28:13 INFO BlockManagerMaster: Trying to register BlockManager
> 16/07/28 04:28:13 INFO BlockManagerMaster: Registered BlockManager
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Got assigned task 0
> 16/07/28 04:28:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 16/07/28 04:28:13 INFO Executor: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> with timestamp 1469680084686
> 16/07/28 04:28:14 INFO Utils: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> to
>
[jira] [Updated] (SPARK-16762) spark hanging when action method print
[
https://issues.apache.org/jira/browse/SPARK-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anh Nguyen updated SPARK-16762:
---
Attachment: Screen Shot 2016-07-28 at 12.33.35 PM.png
> spark hanging when action method print
> --
>
> Key: SPARK-16762
> URL: https://issues.apache.org/jira/browse/SPARK-16762
> Project: Spark
> Issue Type: Bug
> Components: Deploy
>Reporter: Anh Nguyen
> Attachments: Screen Shot 2016-07-28 at 12.33.35 PM.png
>
>
> I write code spark Streaming (consumer) intergate kafaka and deploy on mesos
> FW:
> import org.apache.spark.SparkConf
> import org.apache.spark.streaming._
> import org.apache.spark.streaming.kafka._
> import org.apache.spark.streaming.kafka._
> object consumer {
> def main(args: Array[String]) {
> if (args.length < 4) {
> System.err.println("Usage: KafkaWordCount
> ")
> System.exit(1)
> }
> val Array(zkQuorum, group, topics, numThreads) = args
> val sparkConf = new
> SparkConf().setAppName("KafkaWordCount").set("spark.rpc.netty.dispatcher.numThreads","4")
> val ssc = new StreamingContext(sparkConf, Seconds(2))
> ssc.checkpoint("checkpoint")
> val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
> val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
> topicMap).map(_._2)
> lines.print()
> val words = lines.flatMap(_.split(" "))
> val wordCounts = words.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _
> - _, Minutes(10), Seconds(2), 2)
> ssc.start()
> ssc.awaitTermination()
> }
> }
> This log:
> I0728 04:28:05.439469 4295 exec.cpp:143] Version: 0.28.2
> I0728 04:28:05.443464 4296 exec.cpp:217] Executor registered on slave
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0
> Using Spark's default log4j profile:
> org/apache/spark/log4j-defaults.properties
> 16/07/28 04:28:07 INFO CoarseGrainedExecutorBackend: Registered signal
> handlers for [TERM, HUP, INT]
> 16/07/28 04:28:08 WARN NativeCodeLoader: Unable to load native-hadoop library
> for your platform... using builtin-java classes where applicable
> 16/07/28 04:28:09 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:09 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:11 INFO SecurityManager: Changing view acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: Changing modify acls to: vagrant
> 16/07/28 04:28:11 INFO SecurityManager: SecurityManager: authentication
> disabled; ui acls disabled; users with view permissions: Set(vagrant); users
> with modify permissions: Set(vagrant)
> 16/07/28 04:28:12 INFO Slf4jLogger: Slf4jLogger started
> 16/07/28 04:28:12 INFO Remoting: Starting remoting
> 16/07/28 04:28:13 INFO Remoting: Remoting started; listening on addresses
> :[akka.tcp://sparkExecutorActorSystem@slave1:57380]
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'sparkExecutorActorSystem' on port 57380.
> 16/07/28 04:28:13 INFO DiskBlockManager: Created local directory at
> /home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/blockmgr-4e94f4ea-f074-4dbe-bfa1-34054e6e079c
> 16/07/28 04:28:13 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
> spark://CoarseGrainedScheduler@192.168.33.30:36179
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Successfully registered
> with driver
> 16/07/28 04:28:13 INFO Executor: Starting executor ID
> 8e00b1ea-3f70-428a-8125-fb0eed88aede-S0 on host slave1
> 16/07/28 04:28:13 INFO Utils: Successfully started service
> 'org.apache.spark.network.netty.NettyBlockTransferService' on port 48672.
> 16/07/28 04:28:13 INFO NettyBlockTransferService: Server created on 48672
> 16/07/28 04:28:13 INFO BlockManagerMaster: Trying to register BlockManager
> 16/07/28 04:28:13 INFO BlockManagerMaster: Registered BlockManager
> 16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Got assigned task 0
> 16/07/28 04:28:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
> 16/07/28 04:28:13 INFO Executor: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> with timestamp 1469680084686
> 16/07/28 04:28:14 INFO Utils: Fetching
> http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
> to
>
[jira] [Updated] (SPARK-16762) spark hanging when action method print
[
https://issues.apache.org/jira/browse/SPARK-16762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anh Nguyen updated SPARK-16762:
---
Description:
I write code spark Streaming (consumer) intergate kafaka and deploy on mesos FW:
import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka._
import org.apache.spark.streaming.kafka._
object consumer {
def main(args: Array[String]) {
if (args.length < 4) {
System.err.println("Usage: KafkaWordCount
")
System.exit(1)
}
val Array(zkQuorum, group, topics, numThreads) = args
val sparkConf = new
SparkConf().setAppName("KafkaWordCount").set("spark.rpc.netty.dispatcher.numThreads","4")
val ssc = new StreamingContext(sparkConf, Seconds(2))
ssc.checkpoint("checkpoint")
val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
topicMap).map(_._2)
lines.print()
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _ -
_, Minutes(10), Seconds(2), 2)
ssc.start()
ssc.awaitTermination()
}
}
This log:
I0728 04:28:05.439469 4295 exec.cpp:143] Version: 0.28.2
I0728 04:28:05.443464 4296 exec.cpp:217] Executor registered on slave
8e00b1ea-3f70-428a-8125-fb0eed88aede-S0
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
16/07/28 04:28:07 INFO CoarseGrainedExecutorBackend: Registered signal handlers
for [TERM, HUP, INT]
16/07/28 04:28:08 WARN NativeCodeLoader: Unable to load native-hadoop library
for your platform... using builtin-java classes where applicable
16/07/28 04:28:09 INFO SecurityManager: Changing view acls to: vagrant
16/07/28 04:28:09 INFO SecurityManager: Changing modify acls to: vagrant
16/07/28 04:28:09 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(vagrant); users
with modify permissions: Set(vagrant)
16/07/28 04:28:11 INFO SecurityManager: Changing view acls to: vagrant
16/07/28 04:28:11 INFO SecurityManager: Changing modify acls to: vagrant
16/07/28 04:28:11 INFO SecurityManager: SecurityManager: authentication
disabled; ui acls disabled; users with view permissions: Set(vagrant); users
with modify permissions: Set(vagrant)
16/07/28 04:28:12 INFO Slf4jLogger: Slf4jLogger started
16/07/28 04:28:12 INFO Remoting: Starting remoting
16/07/28 04:28:13 INFO Remoting: Remoting started; listening on addresses
:[akka.tcp://sparkExecutorActorSystem@slave1:57380]
16/07/28 04:28:13 INFO Utils: Successfully started service
'sparkExecutorActorSystem' on port 57380.
16/07/28 04:28:13 INFO DiskBlockManager: Created local directory at
/home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/blockmgr-4e94f4ea-f074-4dbe-bfa1-34054e6e079c
16/07/28 04:28:13 INFO MemoryStore: MemoryStore started with capacity 517.4 MB
16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Connecting to driver:
spark://CoarseGrainedScheduler@192.168.33.30:36179
16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Successfully registered
with driver
16/07/28 04:28:13 INFO Executor: Starting executor ID
8e00b1ea-3f70-428a-8125-fb0eed88aede-S0 on host slave1
16/07/28 04:28:13 INFO Utils: Successfully started service
'org.apache.spark.network.netty.NettyBlockTransferService' on port 48672.
16/07/28 04:28:13 INFO NettyBlockTransferService: Server created on 48672
16/07/28 04:28:13 INFO BlockManagerMaster: Trying to register BlockManager
16/07/28 04:28:13 INFO BlockManagerMaster: Registered BlockManager
16/07/28 04:28:13 INFO CoarseGrainedExecutorBackend: Got assigned task 0
16/07/28 04:28:13 INFO Executor: Running task 0.0 in stage 0.0 (TID 0)
16/07/28 04:28:13 INFO Executor: Fetching
http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar
with timestamp 1469680084686
16/07/28 04:28:14 INFO Utils: Fetching
http://192.168.33.30:52977/jars/spaka-1.0-SNAPSHOT-jar-with-dependencies.jar to
/home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/spark-b833a8b6-b16c-46c6-92a7-89a8f0b3f713/fetchFileTemp4218016376984854956.tmp
16/07/28 04:28:14 INFO Utils: Copying
/home/vagrant/mesos/slave1/work/slaves/8e00b1ea-3f70-428a-8125-fb0eed88aede-S0/frameworks/3b189004-daea-47a3-9ba5-dfac5143f4ab-/executors/0/runs/0433234f-332d-4858-9527-64558fe7fb90/spark-b833a8b6-b16c-46c6-92a7-89a8f0b3f713/-10744548291469680084686_cache
to
[jira] [Created] (SPARK-16762) spark hanging when action method print
Anh Nguyen created SPARK-16762:
--
Summary: spark hanging when action method print
Key: SPARK-16762
URL: https://issues.apache.org/jira/browse/SPARK-16762
Project: Spark
Issue Type: Bug
Components: Deploy
Reporter: Anh Nguyen
I write code spark Streaming (consumer) intergate kafaka and deploy on mesos FW:
import org.apache.spark.SparkConf
import org.apache.spark.streaming._
import org.apache.spark.streaming.kafka._
import org.apache.spark.streaming.kafka._
object consumer {
def main(args: Array[String]) {
if (args.length < 4) {
System.err.println("Usage: KafkaWordCount
")
System.exit(1)
}
val Array(zkQuorum, group, topics, numThreads) = args
val sparkConf = new
SparkConf().setAppName("KafkaWordCount").set("spark.rpc.netty.dispatcher.numThreads","4")
val ssc = new StreamingContext(sparkConf, Seconds(2))
ssc.checkpoint("checkpoint")
val topicMap = topics.split(",").map((_, numThreads.toInt)).toMap
val lines = KafkaUtils.createStream(ssc, zkQuorum, group,
topicMap).map(_._2)
lines.print()
val words = lines.flatMap(_.split(" "))
val wordCounts = words.map(x => (x, 1L)).reduceByKeyAndWindow(_ + _, _ -
_, Minutes(10), Seconds(2), 2)
ssc.start()
ssc.awaitTermination()
}
}
when deploy, spark hanging at method action lines.print()
Please help me, explain this issue
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
