Re: [VOTE] Release Apache Mesos 0.23.0 (rc1)

[Weitao]

+1

发自我的 iPhone

> 在 2015年7月4日，09:41，Marco Massenzio  写道：
> 
> +1
> 
> Marco Massenzio
> Distributed Systems Engineer
> 
>> On Fri, Jul 3, 2015 at 12:25 PM, Adam Bordelon  wrote:
>> Hello Mesos community,
>> 
>> Please vote on releasing the following candidate as Apache Mesos 0.23.0.
>> 
>> 0.23.0 includes the following:
>> 
>> - Per-container network isolation
>> - Upgraded minimum required compilers to GCC 4.8+ or clang 3.5+.
>> - Dockerized slaves will properly recover Docker containers upon failover.
>> 
>> as well as experimental support for:
>> - Fetcher Caching
>> - Revocable Resources
>> - SSL encryption
>> - Persistent Volumes
>> - Dynamic Reservations
>> 
>> The CHANGELOG for the release is available at:
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.23.0-rc1
>> 
>> 
>> The candidate for Mesos 0.23.0 release is available at:
>> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz
>> 
>> The tag to be voted on is 0.23.0-rc1:
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.23.0-rc1
>> 
>> The MD5 checksum of the tarball can be found at:
>> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.md5
>> 
>> The signature of the tarball can be found at:
>> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.asc
>> 
>> The PGP key used to sign the release is here:
>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>> 
>> The JAR is up in Maven in a staging repository here:
>> https://repository.apache.org/content/repositories/orgapachemesos-1056
>> 
>> Please vote on releasing this package as Apache Mesos 0.23.0!
>> 
>> The vote is open until Fri July 10th, 12:00 PDT 2015 and passes if a 
>> majority of at least 3 +1 PMC votes are cast.
>> 
>> [ ] +1 Release this package as Apache Mesos 0.23.0
>> [ ] -1 Do not release this package because ...
>> 
>> Thanks,
>> -Adam-
> 

Re: [VOTE] Release Apache Mesos 0.23.0 (rc1)

[适兕]

+1

2015-07-04 9:41 GMT+08:00 Marco Massenzio :

> +1
>
> *Marco Massenzio*
> *Distributed Systems Engineer*
>
> On Fri, Jul 3, 2015 at 12:25 PM, Adam Bordelon  wrote:
>
>> Hello Mesos community,
>>
>> Please vote on releasing the following candidate as Apache Mesos 0.23.0.
>>
>> 0.23.0 includes the following:
>>
>> 
>> - Per-container network isolation
>> - Upgraded minimum required compilers to GCC 4.8+ or clang 3.5+.
>> - Dockerized slaves will properly recover Docker containers upon failover.
>>
>> as well as experimental support for:
>> - Fetcher Caching
>> - Revocable Resources
>> - SSL encryption
>> - Persistent Volumes
>> - Dynamic Reservations
>>
>> The CHANGELOG for the release is available at:
>>
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.23.0-rc1
>>
>> 
>>
>> The candidate for Mesos 0.23.0 release is available at:
>>
>> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz
>>
>> The tag to be voted on is 0.23.0-rc1:
>> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.23.0-rc1
>>
>> The MD5 checksum of the tarball can be found at:
>>
>> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.md5
>>
>> The signature of the tarball can be found at:
>>
>> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.asc
>>
>> The PGP key used to sign the release is here:
>> https://dist.apache.org/repos/dist/release/mesos/KEYS
>>
>> The JAR is up in Maven in a staging repository here:
>> https://repository.apache.org/content/repositories/orgapachemesos-1056
>>
>> Please vote on releasing this package as Apache Mesos 0.23.0!
>>
>> The vote is open until Fri July 10th, 12:00 PDT 2015 and passes if a
>> majority of at least 3 +1 PMC votes are cast.
>>
>> [ ] +1 Release this package as Apache Mesos 0.23.0
>> [ ] -1 Do not release this package because ...
>>
>> Thanks,
>> -Adam-
>>
>
>


-- 
独立之思想，自由之精神。
--陈寅恪

Re: [VOTE] Release Apache Mesos 0.23.0 (rc1)

[Marco Massenzio]

+1

*Marco Massenzio*
*Distributed Systems Engineer*

On Fri, Jul 3, 2015 at 12:25 PM, Adam Bordelon  wrote:

> Hello Mesos community,
>
> Please vote on releasing the following candidate as Apache Mesos 0.23.0.
>
> 0.23.0 includes the following:
>
> 
> - Per-container network isolation
> - Upgraded minimum required compilers to GCC 4.8+ or clang 3.5+.
> - Dockerized slaves will properly recover Docker containers upon failover.
>
> as well as experimental support for:
> - Fetcher Caching
> - Revocable Resources
> - SSL encryption
> - Persistent Volumes
> - Dynamic Reservations
>
> The CHANGELOG for the release is available at:
>
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.23.0-rc1
>
> 
>
> The candidate for Mesos 0.23.0 release is available at:
> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz
>
> The tag to be voted on is 0.23.0-rc1:
> https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.23.0-rc1
>
> The MD5 checksum of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.md5
>
> The signature of the tarball can be found at:
>
> https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.asc
>
> The PGP key used to sign the release is here:
> https://dist.apache.org/repos/dist/release/mesos/KEYS
>
> The JAR is up in Maven in a staging repository here:
> https://repository.apache.org/content/repositories/orgapachemesos-1056
>
> Please vote on releasing this package as Apache Mesos 0.23.0!
>
> The vote is open until Fri July 10th, 12:00 PDT 2015 and passes if a
> majority of at least 3 +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Mesos 0.23.0
> [ ] -1 Do not release this package because ...
>
> Thanks,
> -Adam-
>

Re: Mesos Slave Port Change Fails Recovery

[Philippe Laflamme]

Awesome!

We've reverted to the previous port and all our slaves have recovered
nicely.

Thanks for looking into this,
Philippe

On Fri, Jul 3, 2015 at 3:27 PM, Vinod Kone  wrote:

> Looks like this is due to a bug in versions < 23.0, where slave recovery
> didn't check for changes in 'port' when considering compatibility
> .
> It has since been fixed in the upcoming 0.23.0 release.
>
> On Thu, Jul 2, 2015 at 8:45 PM, Philippe Laflamme 
> wrote:
>
>> Checkpointing has been enabled since 0.18 on these slaves. The only other
>> setting that changed during the upgrade was that we added --gc_delay=1days.
>> Otherwise, it's an in-place upgrade without any changes to the work
>> directory...
>>
>> Philippe
>>
>> On Thu, Jul 2, 2015 at 8:59 PM, Vinod Kone  wrote:
>>
>>> It is surprising that the slave didn't bail out during the initial phase
>>> of recovery when the port changed. I'm assuming you enabled checkpointing
>>> in 0.20.0 and that you didn't wipe the meta data directory or anything when
>>> upgrading to 21.0?
>>>
>>> On Thu, Jul 2, 2015 at 3:06 PM, Philippe Laflamme 
>>> wrote:
>>>
 Here you are:

 https://gist.github.com/plaflamme/9cd056dc959e0597fb1c

 You can see in the mesos-master.INFO log that it re-registers the slave
 using port :5050 (line 9) and fails the health checks on port :5051 (line
 10). So it might be the slave that re-uses the old configuration?

 Thanks,
 Philippe

 On Thu, Jul 2, 2015 at 5:54 PM, Vinod Kone  wrote:

> Can you paste some logs?
>
> On Thu, Jul 2, 2015 at 2:23 PM, Philippe Laflamme  > wrote:
>
>> Ok, that's reasonable, but I'm not sure why it would successfully
>> re-register with the master if it's not supposed to in the first place. I
>> think changing the resources (for example) will dump the old 
>> configuration
>> in the logs and tell you why recovery is bailing out. It's not doing that
>> in this case.
>>
>> I looks as though this doesn't work only because the master can't
>> ping the slave on the old port, because the whole recovery process was
>> successful otherwise.
>>
>> I'm not sure if the slave could have picked up its configuration
>> change and failed the recovery early, but that would definitely be a 
>> better
>> experience.
>>
>> Philippe
>>
>> On Thu, Jul 2, 2015 at 5:15 PM, Vinod Kone 
>> wrote:
>>
>>> For slave recovery to work, it is expected to not change its config.
>>>
>>> On Thu, Jul 2, 2015 at 2:10 PM, Philippe Laflamme <
>>> phili...@hopper.com> wrote:
>>>
 Hi,

 I'm trying to roll out an upgrade from 0.20.0 to 0.21.0 with slaves
 configured with checkpointing and with "reconnect" recovery.

 I was investigating why the slaves would successfully re-register
 with the master and recover, but would subsequently be asked to 
 shutdown
 ("health check timeout").

 It turns out that our slaves had been unintentionally configured to
 use port 5050 in the previous configuration. We decided to fix that 
 during
 the upgrade and have them use the default 5051 port.

 This change seems to make the health checks fail and eventually
 kills the slave due to inactivity.

 I've confirmed that leaving the port to what it was in the previous
 configuration makes the slave successfully re-register and is not 
 asked to
 shutdown later on.

 Is this a known issue? I haven't been able to find a JIRA ticket
 for this. Maybe it's the expected behaviour? Should I create a ticket?

 Thanks,
 Philippe

>>>
>>>
>>
>

>>>
>>
>

Re: Mesos Slave Port Change Fails Recovery

[Vinod Kone]

Looks like this is due to a bug in versions < 23.0, where slave recovery
didn't check for changes in 'port' when considering compatibility
.
It has since been fixed in the upcoming 0.23.0 release.

On Thu, Jul 2, 2015 at 8:45 PM, Philippe Laflamme 
wrote:

> Checkpointing has been enabled since 0.18 on these slaves. The only other
> setting that changed during the upgrade was that we added --gc_delay=1days.
> Otherwise, it's an in-place upgrade without any changes to the work
> directory...
>
> Philippe
>
> On Thu, Jul 2, 2015 at 8:59 PM, Vinod Kone  wrote:
>
>> It is surprising that the slave didn't bail out during the initial phase
>> of recovery when the port changed. I'm assuming you enabled checkpointing
>> in 0.20.0 and that you didn't wipe the meta data directory or anything when
>> upgrading to 21.0?
>>
>> On Thu, Jul 2, 2015 at 3:06 PM, Philippe Laflamme 
>> wrote:
>>
>>> Here you are:
>>>
>>> https://gist.github.com/plaflamme/9cd056dc959e0597fb1c
>>>
>>> You can see in the mesos-master.INFO log that it re-registers the slave
>>> using port :5050 (line 9) and fails the health checks on port :5051 (line
>>> 10). So it might be the slave that re-uses the old configuration?
>>>
>>> Thanks,
>>> Philippe
>>>
>>> On Thu, Jul 2, 2015 at 5:54 PM, Vinod Kone  wrote:
>>>
 Can you paste some logs?

 On Thu, Jul 2, 2015 at 2:23 PM, Philippe Laflamme 
 wrote:

> Ok, that's reasonable, but I'm not sure why it would successfully
> re-register with the master if it's not supposed to in the first place. I
> think changing the resources (for example) will dump the old configuration
> in the logs and tell you why recovery is bailing out. It's not doing that
> in this case.
>
> I looks as though this doesn't work only because the master can't ping
> the slave on the old port, because the whole recovery process was
> successful otherwise.
>
> I'm not sure if the slave could have picked up its configuration
> change and failed the recovery early, but that would definitely be a 
> better
> experience.
>
> Philippe
>
> On Thu, Jul 2, 2015 at 5:15 PM, Vinod Kone 
> wrote:
>
>> For slave recovery to work, it is expected to not change its config.
>>
>> On Thu, Jul 2, 2015 at 2:10 PM, Philippe Laflamme <
>> phili...@hopper.com> wrote:
>>
>>> Hi,
>>>
>>> I'm trying to roll out an upgrade from 0.20.0 to 0.21.0 with slaves
>>> configured with checkpointing and with "reconnect" recovery.
>>>
>>> I was investigating why the slaves would successfully re-register
>>> with the master and recover, but would subsequently be asked to shutdown
>>> ("health check timeout").
>>>
>>> It turns out that our slaves had been unintentionally configured to
>>> use port 5050 in the previous configuration. We decided to fix that 
>>> during
>>> the upgrade and have them use the default 5051 port.
>>>
>>> This change seems to make the health checks fail and eventually
>>> kills the slave due to inactivity.
>>>
>>> I've confirmed that leaving the port to what it was in the previous
>>> configuration makes the slave successfully re-register and is not asked 
>>> to
>>> shutdown later on.
>>>
>>> Is this a known issue? I haven't been able to find a JIRA ticket for
>>> this. Maybe it's the expected behaviour? Should I create a ticket?
>>>
>>> Thanks,
>>> Philippe
>>>
>>
>>
>

>>>
>>
>

[VOTE] Release Apache Mesos 0.23.0 (rc1)

[Adam Bordelon]

Hello Mesos community,

Please vote on releasing the following candidate as Apache Mesos 0.23.0.

0.23.0 includes the following:

- Per-container network isolation
- Upgraded minimum required compilers to GCC 4.8+ or clang 3.5+.
- Dockerized slaves will properly recover Docker containers upon failover.

as well as experimental support for:
- Fetcher Caching
- Revocable Resources
- SSL encryption
- Persistent Volumes
- Dynamic Reservations

The CHANGELOG for the release is available at:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=0.23.0-rc1


The candidate for Mesos 0.23.0 release is available at:
https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz

The tag to be voted on is 0.23.0-rc1:
https://git-wip-us.apache.org/repos/asf?p=mesos.git;a=commit;h=0.23.0-rc1

The MD5 checksum of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.md5

The signature of the tarball can be found at:
https://dist.apache.org/repos/dist/dev/mesos/0.23.0-rc1/mesos-0.23.0.tar.gz.asc

The PGP key used to sign the release is here:
https://dist.apache.org/repos/dist/release/mesos/KEYS

The JAR is up in Maven in a staging repository here:
https://repository.apache.org/content/repositories/orgapachemesos-1056

Please vote on releasing this package as Apache Mesos 0.23.0!

The vote is open until Fri July 10th, 12:00 PDT 2015 and passes if a
majority of at least 3 +1 PMC votes are cast.

[ ] +1 Release this package as Apache Mesos 0.23.0
[ ] -1 Do not release this package because ...

Thanks,
-Adam-

Re: Hadoop on Mesos. HDFS question.

[Kk Bk]

Thanks guys for the response.

1) I use trusty. Seems like CDH4 does not have support for Trusty.

2) Followed instructions as per link https://github.com/mesosphere/hdfs.
Able to build "hdfs-mesos-*.tgz"

Should i copy this file to all nodes (i have multi-node mesos cluster) or
just the master node of mesos where i plan to keep the namenode for hadoop





On Fri, Jul 3, 2015 at 8:34 AM, Tom Arnfeld  wrote:

>  It might be worth taking a look at the install documentation on the
> Hadoop on Mesos product here; https://github.com/mesos/hadoop
>
> For our installations I don't think we really do much more than installing
> the apt packages you mentioned and then installing the hadoop-mesos jars..
> plus adding the appropriate configuration.
>
> On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk , wrote:
>
>> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
>> instruction as per link
>> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>>
>> Step-2 of link says to install HDFS using as per link
>> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
>> .
>>
>> Question: Is it sufficient to run following commands
>>
>> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
>> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
>> hadoop-hdfs-datanode
>>
>> Or just follow the instructions on the mesosphere link that installs HDFS
>> ?
>>
>>
>>
>>
>>

Re: Running storm over mesos

[Pradeep Chhetri]

Hello Tim,

I guess I found the issue. The memory which was allocated to both the
vagrant vms where i was running mesos-slaves was less than the minimum
memory required for spawning a supervisor i guess. I just changed the VM
limits and respawned one of them. Now, the supervisor ran but it failed due
to slave going offline. Here are some of the logs:

I0703 18:19:39.588933  3177 slave.cpp:1144] Got assigned task node2-31000
for framework 20150703-151956-169978048-5050-12019-0001
I0703 18:19:39.589437  3177 gc.cpp:84] Unscheduling
'/tmp/mesos/slaves/20150703-173159-169978048-5050-23609-S0/frameworks/20150703-151956-169978048-5050-12019-0001'
from gc
I0703 18:19:39.589656  3177 gc.cpp:84] Unscheduling
'/tmp/mesos/slaves/20150703-173159-169978048-5050-23609-S0/frameworks/20150703-151956-169978048-5050-12019-0001/executors/myTopo-1-1435938251'
from gc
I0703 18:19:39.589779  3177 slave.cpp:1254] Launching task node2-31000 for
framework 20150703-151956-169978048-5050-12019-0001
I0703 18:19:39.600589  3177 slave.cpp:4208] Launching executor
myTopo-1-1435938251 of framework 20150703-151956-169978048-5050-12019-0001
in work directory
'/tmp/mesos/slaves/20150703-173159-169978048-5050-23609-S0/frameworks/20150703-151956-169978048-5050-12019-0001/executors/myTopo-1-1435938251/runs/693f59db-7b74-411a-ba4d-c83f32ee2619'
I0703 18:19:39.600769  3177 slave.cpp:1401] Queuing task 'node2-31000' for
executor myTopo-1-1435938251 of framework
'20150703-151956-169978048-5050-12019-0001
I0703 18:19:39.600838  3177 slave.cpp:1144] Got assigned task node2-31001
for framework 20150703-151956-169978048-5050-12019-0001
I0703 18:19:39.603096  3177 gc.cpp:84] Unscheduling
'/tmp/mesos/slaves/20150703-173159-169978048-5050-23609-S0/frameworks/20150703-151956-169978048-5050-12019-0001/executors/againTopo-2-1435942258'
from gc
I0703 18:19:39.603212  3177 slave.cpp:1254] Launching task node2-31001 for
framework 20150703-151956-169978048-5050-12019-0001
I0703 18:19:39.601094  3178 containerizer.cpp:484] Starting container
'693f59db-7b74-411a-ba4d-c83f32ee2619' for executor 'myTopo-1-1435938251'
of framework '20150703-151956-169978048-5050-12019-0001'
I0703 18:19:39.606645  3178 launcher.cpp:130] Forked child with pid '3187'
for container '693f59db-7b74-411a-ba4d-c83f32ee2619'
I0703 18:19:39.619261  3175 fetcher.cpp:238] Fetching URIs using command
'/usr/libexec/mesos/mesos-fetcher'
I0703 18:19:39.619355  3177 slave.cpp:4208] Launching executor
againTopo-2-1435942258 of framework
20150703-151956-169978048-5050-12019-0001 in work directory
'/tmp/mesos/slaves/20150703-173159-169978048-5050-23609-S0/frameworks/20150703-151956-169978048-5050-12019-0001/executors/againTopo-2-1435942258/runs/2cf39864-16e5-45bf-98a1-3e2e6fffdac9'
I0703 18:19:39.629672  3177 slave.cpp:1401] Queuing task 'node2-31001' for
executor againTopo-2-1435942258 of framework
'20150703-151956-169978048-5050-12019-0001
I0703 18:19:39.629956  3175 containerizer.cpp:484] Starting container
'2cf39864-16e5-45bf-98a1-3e2e6fffdac9' for executor
'againTopo-2-1435942258' of framework
'20150703-151956-169978048-5050-12019-0001'
I0703 18:19:39.633669  3175 launcher.cpp:130] Forked child with pid '3191'
for container '2cf39864-16e5-45bf-98a1-3e2e6fffdac9'
I0703 18:19:39.660353  3175 fetcher.cpp:238] Fetching URIs using command
'/usr/libexec/mesos/mesos-fetcher'
I0703 18:19:57.502316  3176 slave.cpp:3165] Monitoring executor
'myTopo-1-1435938251' of framework
'20150703-151956-169978048-5050-12019-0001' in container
'693f59db-7b74-411a-ba4d-c83f32ee2619'
I0703 18:19:57.603291  3181 containerizer.cpp:1123] Executor for container
'693f59db-7b74-411a-ba4d-c83f32ee2619' has exited
I0703 18:19:57.603431  3181 containerizer.cpp:918] Destroying container
'693f59db-7b74-411a-ba4d-c83f32ee2619'
I0703 18:19:57.617431  3174 slave.cpp:3223] Executor 'myTopo-1-1435938251'
of framework 20150703-151956-169978048-5050-12019-0001 exited with status 1
I0703 18:19:57.618479  3174 slave.cpp:2531] Handling status update
TASK_LOST (UUID: da7cce06-4089-4366-b156-4cd9d527a903) for task node2-31000
of framework 20150703-151956-169978048-5050-12019-0001 from @0.0.0.0:0
W0703 18:19:57.619213  3174 containerizer.cpp:814] Ignoring update for
unknown container: 693f59db-7b74-411a-ba4d-c83f32ee2619
I0703 18:19:57.619920  3174 status_update_manager.cpp:317] Received status
update TASK_LOST (UUID: da7cce06-4089-4366-b156-4cd9d527a903) for task
node2-31000 of framework 20150703-151956-169978048-5050-12019-0001
I0703 18:19:57.620934  3174 slave.cpp:2776] Forwarding the update TASK_LOST
(UUID: da7cce06-4089-4366-b156-4cd9d527a903) for task node2-31000 of
framework 20150703-151956-169978048-5050-12019-0001 to
master@192.168.33.10:5050
I0703 18:19:57.628343  3179 statu

Re: Running storm over mesos

[Pradeep Chhetri]

Hello Tim,

The nimbus logs says :

2015-07-03 17:48:47 o.a.z.ClientCnxn [INFO] Session establishment complete
on server node1/192.168.33.10:2181, sessionid = 0x14e548011e40024,
negotiated timeout = 2
2015-07-03 17:48:47 b.s.d.nimbus [INFO] Starting Nimbus server...
2015-07-03 17:48:48 s.m.MesosNimbus [INFO] Currently have 2 offers buffered
2015-07-03 17:48:48 s.m.MesosNimbus [INFO] Topologies that need
assignments: #{"newTopo-3-1435942676" "o-5-1435944054"
"againTopo-2-1435942258" "test-4-1435943781" "myTopo-1-1435938251"}
2015-07-03 17:48:48 s.m.MesosNimbus [INFO] Number of available slots: 0
2015-07-03 17:48:58 s.m.MesosNimbus [INFO] Currently have 2 offers buffered
2015-07-03 17:48:58 s.m.MesosNimbus [INFO] Topologies that need
assignments: #{"newTopo-3-1435942676" "o-5-1435944054"
"againTopo-2-1435942258" "test-4-1435943781" "myTopo-1-1435938251"}
2015-07-03 17:48:58 s.m.MesosNimbus [INFO] Number of available slots: 0
2015-07-03 17:49:08 s.m.MesosNimbus [INFO] Currently have 2 offers buffered
2015-07-03 17:49:08 s.m.MesosNimbus [INFO] Topologies that need
assignments: #{"newTopo-3-1435942676" "o-5-1435944054"
"againTopo-2-1435942258" "test-4-1435943781" "myTopo-1-1435938251"}
2015-07-03 17:49:08 s.m.MesosNimbus [INFO] Number of available slots: 0
2015-07-03 17:49:18 s.m.MesosNimbus [INFO] Currently have 2 offers buffered
2015-07-03 17:49:18 s.m.MesosNimbus [INFO] Topologies that need
assignments: #{"newTopo-3-1435942676" "o-5-1435944054"
"againTopo-2-1435942258" "test-4-1435943781" "myTopo-1-1435938251"}
2015-07-03 17:49:18 s.m.MesosNimbus [INFO] Number of available slots: 0


There is something interesting in mesos-slave logs saying:

W0703 17:48:46.204479 23660 slave.cpp:1934] Ignoring updating pid for
framework 20150703-151956-169978048-5050-12019-0001 because it does not
exist

The framework id is that of storm.

Mesos-master logs says:

lave(1)@192.168.33.10:5051 (node1) for framework
20150703-151956-169978048-5050-12019-0001 (Storm!!!) at
scheduler-4bc2ee4e-7d62-4ef9-b04c-14fb92ca3ee1@192.168.33.10:38445
I0703 17:54:21.237983 23627 master.cpp:2273] Processing ACCEPT call for
offers: [ 20150703-173159-169978048-5050-23609-O230 ] on slave
20150703-151956-169978048-5050-12019-S3 at slave(1)@192.168.33.11:5051
(node2) for framework 20150703-151956-169978048-5050-12019-0001 (Storm!!!)
at scheduler-4bc2ee4e-7d62-4ef9-b04c-14fb92ca3ee1@192.168.33.10:38445
I0703 17:54:21.239336 23627 hierarchical.hpp:648] Recovered cpus(*):1;
mem(*):229; disk(*):34260; ports(*):[31000-32000] (total allocatable:
cpus(*):1; mem(*):229; disk(*):34260; ports(*):[31000-32000]) on slave
20150703-151956-169978048-5050-12019-S0 from framework
20150703-151956-169978048-5050-12019-0001
I0703 17:54:21.240054 23627 hierarchical.hpp:648] Recovered cpus(*):2;
mem(*):497; disk(*):34260; ports(*):[31000-32000] (total allocatable:
cpus(*):2; mem(*):497; disk(*):34260; ports(*):[31000-32000]) on slave
20150703-151956-169978048-5050-12019-S3 from framework
20150703-151956-169978048-5050-12019-0001
I0703 17:54:22.108049 23621 http.cpp:516] HTTP request for
'/master/state.json'
I0703 17:54:24.108803 23621 http.cpp:516] HTTP request for
'/master/state.json'
I0703 17:54:26.965812 23621 master.cpp:3760] Sending 2 offers to framework
20150703-151956-169978048-5050-12019-0001 (Storm!!!) at
scheduler-4bc2ee4e-7d62-4ef9-b04c-14fb92ca3ee1@192.168.33.10:38445
I0703 17:54:33.117069 23622 http.cpp:516] HTTP request for
'/master/state.json'
I0703 17:54:35.118489 23622 http.cpp:516] HTTP request for
'/master/state.json'
I0703 17:54:41.238107 23622 master.cpp:2273] Processing ACCEPT call for
offers: [ 20150703-173159-169978048-5050-23609-O231 ] on slave
20150703-151956-169978048-5050-12019-S3 at slave(1)@192.168.33.11:5051
(node2) for framework 20150703-151956-169978048-5050-12019-0001 (Storm!!!)
at scheduler-4bc2ee4e-7d62-4ef9-b04c-14fb92ca3ee1@192.168.33.10:38445
I0703 17:54:41.238258 23622 master.cpp:2273] Processing ACCEPT call for
offers: [ 20150703-173159-169978048-5050-23609-O232 ] on slave
20150703-151956-169978048-5050-12019-S0 at slave(1)@192.168.33.10:5051
(node1) for framework 20150703-151956-169978048-5050-12019-0001 (Storm!!!)
at scheduler-4bc2ee4e-7d62-4ef9-b04c-14fb92ca3ee1@192.168.33.10:38445
I0703 17:54:41.239572 23622 hierarchical.hpp:648] Recovered cpus(*):2;
mem(*):497; disk(*):34260; ports(*):[31000-32000] (total allocatable:
cpus(*):2; mem(*):497; disk(*):34260; ports(*):[31000-32000]) on slave
20150703-151956-169978048-5050-12019-S3 from framework
20150703-151956-169978048-5050-12019-0001
I0703 17:54:41.240078 23622 hierarchical.hpp:648] Recovered cpus(*):1;
mem(*):229; disk(*):34260; ports(*):[31000-32000] (total allocatable:
cpus(*):1; mem(*):229; disk(*):34260; ports(*

Re: Running storm over mesos

[CCAAT]

On 07/03/2015 12:30 PM, Tim Chen wrote:

Hi Pradeep,

Without any more information it's quite impossible to know what's going on.
What's in the slave logs and storm framework logs?
Tim

On Fri, Jul 3, 2015 at 10:06 AM, Pradeep Chhetri
mailto:pradeep.chhetr...@gmail.com>> wrote:

Hello all,

I am trying to run Storm over Mesos using the tutorial
(http://open.mesosphere.com/tutorials/run-storm-on-mesos) over
vagrant. When I am trying to submit a sample topology, it is not
spawning any storm supervisors over the mesos-slaves. I didn't find
anything interesting in the logs as well. Can someone help in
figuring out the problem.
Pradeep Chhetri


Sometimes it helps to just read about some of the various ways to use 
storm. Here are some links for reading at what others have done::



http://tutorials.github.io/pages/creating-a-production-storm-cluster.html?ts=1340499018#.VM67mz5VHxg

https://storm.apache.org/documentation/Setting-up-a-Storm-cluster.html


And of coarse this reference, just to be complete.
https://storm.canonical.com/

Re: Running storm over mesos

[Tim Chen]

Hi Pradeep,

Without any more information it's quite impossible to know what's going on.

What's in the slave logs and storm framework logs?

Tim

On Fri, Jul 3, 2015 at 10:06 AM, Pradeep Chhetri <
pradeep.chhetr...@gmail.com> wrote:

> Hello all,
>
> I am trying to run Storm over Mesos using the tutorial (
> http://open.mesosphere.com/tutorials/run-storm-on-mesos) over vagrant.
> When I am trying to submit a sample topology, it is not spawning any storm
> supervisors over the mesos-slaves. I didn't find anything interesting in
> the logs as well. Can someone help in figuring out the problem.
>
> Thank you.
>
> --
> Pradeep Chhetri
>
>

Running storm over mesos

[Pradeep Chhetri]

Hello all,

I am trying to run Storm over Mesos using the tutorial (
http://open.mesosphere.com/tutorials/run-storm-on-mesos) over vagrant. When
I am trying to submit a sample topology, it is not spawning any storm
supervisors over the mesos-slaves. I didn't find anything interesting in
the logs as well. Can someone help in figuring out the problem.

Thank you.

-- 
Pradeep Chhetri

About Hadoop mr1 on Mesos 0.22.1.

[Wang Chun]

Hi,
I'm a fresh people of Mesos usage, and recently I tried to follow the
instructions in
https://github.com/mesos/hadoop/blob/master/README.md
to setup the Hadoop running on Mesos.
Now I get a problem that the Hadoop is not displayed under list of Mesos
Web UI "Active Frameworks".  I want to know what I have missing to get the
framework registered with Mesos, or is it necessary in this case?

Here is some env information:

Hadoop running with following services:
$ jps
12375 JobTracker
12839 TaskTracker
11325 DataNode
11267 NameNode
12895 Jps

The log from jobtracker is showing Mesos is doing some job:
I0704 00:33:24.501077 12460 master.cpp:2273] Processing ACCEPT call for
offers: [ 20150704-001951-2397939904-59631-12375-O160 ] on slave
20150704-001951-2397939904-59631-12375-S0 at slave(1)@192.168.237.142:59631
(kilo) for framework 20150704-001951-2397939904-59631-12375- (hadoop)
at scheduler-0ab6c7e5-fb4f-4951-9086-cc9efc3c27a4@192.168.237.142:59631
I0704 00:33:24.501871 12457 hierarchical.hpp:648] Recovered cpus(*):2;
mem(*):2921; disk(*):10618; ports(*):[31000-32000] (total allocatable:
cpus(*):2; mem(*):2921; disk(*):10618; ports(*):[31000-32000]) on slave
20150704-001951-2397939904-59631-12375-S0 from framework
20150704-001951-2397939904-59631-12375-
I0704 00:33:29.508990 12457 master.cpp:3760] Sending 1 offers to framework
20150704-001951-2397939904-59631-12375- (hadoop) at
scheduler-0ab6c7e5-fb4f-4951-9086-cc9efc3c27a4@192.168.237.142:59631
15/07/04 00:33:29 INFO mapred.ResourcePolicy: JobTracker Status
  Pending Map Tasks: 0
   Pending Reduce Tasks: 0
  Running Map Tasks: 0
   Running Reduce Tasks: 0
 Idle Map Slots: 0
  Idle Reduce Slots: 0
 Inactive Map Slots: 0 (launched but no hearbeat yet)
  Inactive Reduce Slots: 0 (launched but no hearbeat yet)
   Needed Map Slots: 0
Needed Reduce Slots: 0
 Unhealthy Trackers: 0

Thanks a lot.

Re: Hadoop on Mesos. HDFS question.

[Tom Arnfeld]

It might be worth taking a look at the install documentation on the Hadoop on 
Mesos product here; https://github.com/mesos/hadoop



For our installations I don't think we really do much more than installing the 
apt packages you mentioned and then installing the hadoop-mesos jars.. plus 
adding the appropriate configuration.






On Friday, Jul 3, 2015 at 3:52 pm, Kk Bk , wrote:

I am trying to install Hadoop on Mesos on ubuntu servers, So followed 
instruction as per link 
https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.




Step-2 of link says to install HDFS using as per link 
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html.




Question: Is it sufficient to run following commands




1) On Namenode: sudo apt-get install hadoop-hdfs-namenode

2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker 
hadoop-hdfs-datanode




Or just follow the instructions on the mesosphere link that installs HDFS ?

Re: Hadoop on Mesos. HDFS question.

[haosdent]

You just need install HDFS through sudo apt-get install
hadoop-hdfs-namenode hadoop-hdfs-secondarynamenode
hadoop-hdfs-datanode hadoop-client
And then continue to follow the mesosphere link step. Mesosphere link don't
contain instructions to install HDFS.

On Fri, Jul 3, 2015 at 10:51 PM, Kk Bk  wrote:

> I am trying to install Hadoop on Mesos on ubuntu servers, So followed
> instruction as per link
> https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.
>
> Step-2 of link says to install HDFS using as per link
> http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
> .
>
> Question: Is it sufficient to run following commands
>
> 1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
> 2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
> hadoop-hdfs-datanode
>
> Or just follow the instructions on the mesosphere link that installs HDFS ?
>
>
>
>
>


-- 
Best Regards,
Haosdent Huang

Hadoop on Mesos. HDFS question.

[Kk Bk]

I am trying to install Hadoop on Mesos on ubuntu servers, So followed
instruction as per link
https://open.mesosphere.com/tutorials/run-hadoop-on-mesos/#step-2.

Step-2 of link says to install HDFS using as per link
http://www.cloudera.com/content/cloudera/en/documentation/cdh4/latest/CDH4-Installation-Guide/cdh4ig_topic_4_4.html
.

Question: Is it sufficient to run following commands

1) On Namenode: sudo apt-get install hadoop-hdfs-namenode
2) On Datanode: sudo apt-get install hadoop-0.20-mapreduce-tasktracker
hadoop-hdfs-datanode

Or just follow the instructions on the mesosphere link that installs HDFS ?