回复: Mesos CI extension for s390x

[Yu Wei]

Hi,
Do your mean you want to port mesos to 390 system?

Thx,
Jared
Interested in cloud computing,big data processing,linux

2017年4月28日 01:40于 Nayana Thorat 写道：

Hello,
This is about Mesos CI extension for s390x.
Can Mesos CI extended to support s390x systems? We can provide s390x vm to
add in CI pool.

Let us know the hardware/software requirements if any.

Regards,
Nayana

Re: Question about extending authentication modules

[Yu Wei]

I found the solution now. Using MESOS_ as prefix.


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Yu Wei <yu20...@hotmail.com>
Sent: Wednesday, March 29, 2017 7:04:45 PM
To: dev@mesos.apache.org
Subject: Question about extending authentication modules

Hi guys,

I developed a customized authentication module.

For agent registration with master, parameters could be passed via "--modules".

For framework registration such as spark, kafka and so on, how could I set 
similar parameters?



Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

Question about extending authentication modules

[Yu Wei]

Hi guys,

I developed a customized authentication module.

For agent registration with master, parameters could be passed via "--modules".

For framework registration such as spark, kafka and so on, how could I set 
similar parameters?



Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

One question about src/Makefile.am

[Yu Wei]

Hi guys,


In src/Makefile.am, there is two lines as below,

MESOS_CPPFLAGS += -I$(top_srcdir)/include
MESOS_CPPFLAGS += -I../include


In current code structure, it seems they references the same directory.


Is this redundant code? Or any story about this?


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

How could I setup mesos developement environment?

[Yu Wei]

Hi guys,

I want to setup development environment to debug mesos.

Is there any document about this?


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

Re: Is there any plan to enhance authentication and authorization feature?

[Yu Wei]

Got that. Thank you.


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Greg Mann <g...@mesosphere.io>
Sent: Wednesday, February 15, 2017 12:24:42 AM
To: dev
Subject: Re: Is there any plan to enhance authentication and authorization 
feature?

Hi Jared,
As Vinod said, if you're using the default Mesos authenticators, the only
way to update the dictionary of credentials is to restart the master/agent
process. Note, however, that the authentication system is modular, so you
could also write a custom authenticator which permits dynamic modification
of the credentials. Docs for authentication and for the modules system can
be found here:

http://mesos.apache.org/documentation/latest/authentication/
http://mesos.apache.org/documentation/latest/modules/

Cheers,
Greg



On Tue, Feb 14, 2017 at 2:40 AM, Yu Wei <yu20...@hotmail.com> wrote:

> Hi Guys,
>
> Is there any plan to enhance authentication and authorization
> functionality?
>
> For example, after agent register successfully, then password of the agent
> is changed, is that possible to make master and agent aware of such changes?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>

Is there any plan to enhance authentication and authorization feature?

[Yu Wei]

Hi Guys,

Is there any plan to enhance authentication and authorization functionality?

For example, after agent register successfully, then password of the agent is 
changed, is that possible to make master and agent aware of such changes?


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

Re: Question about mesos authentication and authorization?

[Yu Wei]

Hi Greg,


Thanks for the information.


Jared


From: Greg Mann <g...@mesosphere.io>
Sent: Wednesday, December 14, 2016 9:18:53 AM
To: dev
Subject: Re: Question about mesos authentication and authorization?

Hi Jared,
The default authenticator modules included with Mesos only support storing
the credentials in a file on disk. However, Mesos exposes an authenticator
module interface which can be used to implement more complex authentication
scenarios. Note that there are authenticator module interfaces for both the
V0 API [1] and the HTTP V1 API [2]. You can find an introduction to Mesos
modules in the docs [3].

Some Mesos users have implemented support for systems like LDAP and
Kerberos. DC/OS Enterprise is one example of a system with LDAP support,
but that functionality is proprietary so the module source-code isn't
freely available.

Cheers,
Greg

[1] https://github.com/apache/mesos/blob/master/include/meso
s/authentication/authenticator.hpp
[2] https://github.com/apache/mesos/blob/master/3rdparty/lib
process/include/process/authenticator.hpp
[3] http://mesos.apache.org/documentation/latest/modules/



On Mon, Dec 12, 2016 at 6:39 PM, Yu Wei <yu20...@hotmail.com> wrote:

> Hi folks,
>
>
> One question about mesos authentication and authorization.
>
> Where could I store the credentials? In documentation, it seems the
> credentials is stored in local disk.
>
> Is there any other approach to store and manage credentials via LDAP or
> other 3rd party software?
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>

Question about mesos authentication and authorization?

[Yu Wei]

Hi folks,


One question about mesos authentication and authorization.

Where could I store the credentials? In documentation, it seems the credentials 
is stored in local disk.

Is there any other approach to store and manage credentials via LDAP or other 
3rd party software?


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

[Review request] Kafka mesos framework pull request #259: Issue 197: --bind-address for broker seems to be ignored

[Yu Wei]

Hi,


I created pull request kafka mesos framework #259 to fix issue 197 which was 
marked as bug by @joe.st...@stealth.ly.


Please help to review the change.


Any comments please let me know.


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

How to commit code change to kafka mesos framework to fix known issues?

[Yu Wei]

Hi Guys,


I  want to fix a problem in kafka mesos framework.

https://github.com/mesos/kafka/issues/197


How could I do that? Is there any process about this?


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: Two questions about running spark on mesos

[Yu Wei]

Hi Joseph,

I read the document you pointed. However, I didn't find anything about that, so 
I turn to community for help and guidance.

For the 2nd question, I already know how to do that.


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux



From: Joseph Wu <jos...@mesosphere.io>
Sent: Tuesday, November 15, 2016 3:40 AM
To: dev
Cc: dev
Subject: Re: Two questions about running spark on mesos

1) You should read through this page:
https://spark.apache.org/docs/latest/running-on-mesos.html
I (Mesos person) can't answer any questions that aren't already answered on
that page :)

2) Your normal spark commands (whatever they are) should still work
regardless of the backend.

On Mon, Nov 14, 2016 at 2:58 AM, Yu Wei <yu20...@hotmail.com> wrote:

> Hi Guys,
>
>
> Two questions about running spark on mesos.
>
> 1, Does spark configuration of conf/slaves still work when running spark
> on mesos?
>
> According to my observations, it seemed that conf/slaves still took
> effect when running spark-shell.
>
> However, it doesn't take effect when deploying in cluster mode.
>
> Is this expected behavior?
>
>Or did I miss anything?
>
>
> 2, Could I kill submitted jobs when running spark on mesos in cluster mode?
>
> I launched spark on mesos in cluster mode. Then submitted a long
> running job succeeded.
>
> Then I want to kill the job.
>
> How could I do that? Is there any similar commands as launching spark
> on yarn?
>
>
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>

Two questions about running spark on mesos

[Yu Wei]

Hi Guys,


Two questions about running spark on mesos.

1, Does spark configuration of conf/slaves still work when running spark on 
mesos?

According to my observations, it seemed that conf/slaves still took effect 
when running spark-shell.

However, it doesn't take effect when deploying in cluster mode.

Is this expected behavior?

   Or did I miss anything?


2, Could I kill submitted jobs when running spark on mesos in cluster mode?

I launched spark on mesos in cluster mode. Then submitted a long running 
job succeeded.

Then I want to kill the job.

How could I do that? Is there any similar commands as launching spark on 
yarn?




Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: Failed to launch spark jobs on mesos due to "hadoop" not found

[Yu Wei]

A little interesting.



From: tommy xiao <xia...@gmail.com>
Sent: Friday, November 11, 2016 2:19 PM
To: dev
Subject: Re: Failed to launch spark jobs on mesos due to "hadoop" not found

this is pain. if the mesos support proxy user feature like hadoop,then
every thing can easy handle.

2016-11-11 14:06 GMT+08:00 Yu Wei <yu20...@hotmail.com>:

> I fixed the problem by adding environment variables HADOOP_HOME and
> JAVA_HOME to launch mesos-agent as below,
>
> sudo HADOOP_HOME=/home/dcos/repo/bigdata/hadoop-2.7.3/
> JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el7_2.x86_64/jre/
> /usr/local/sbin/mesos-agent --work_dir=/var/lib/mesos/agent  ...
>
> It seemed that this is caused by sudo secure_path.
>
>
> Is there any better solution to fix this problem?
>
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>
> 
> From: tommy xiao <xia...@gmail.com>
> Sent: Friday, November 11, 2016 1:52:54 PM
> To: dev
> Subject: Re: Failed to launch spark jobs on mesos due to "hadoop" not found
>
> it seem like in same user mode.
>
> 2016-11-10 22:37 GMT+08:00 Joris Van Remoortere <jo...@mesosphere.io>:
>
> > Switch to the user that the agent is running as, and the directory from
> > which it is executing and ensure that you can find hadoop. For example by
> > running
> > `which hadoop`. This may be a PATH or JAVA_HOME issue?
> >
> > -
> > *Joris Van Remoortere*
> > Mesosphere
> >
> > On Thu, Nov 10, 2016 at 6:29 AM, Yu Wei <yu20...@hotmail.com> wrote:
> >
> > > Hi Guys,
> > >
> > > I failed to launch spark jobs on mesos. Actually I submitted the job to
> > > cluster successfully.
> > >
> > > But the job failed to run.
> > >
> > > I1110 18:25:11.095507   301 fetcher.cpp:498] Fetcher Info:
> > > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/
> > 1f8e621b-3cbf-4b86-a1c1-
> > > 9e2cf77265ee-S7\/root","items":[{"action":"BYPASS_CACHE","
> > > uri":{"extract":true,"value":"hdfs:\/\/192.168.111.74:9090\/
> > > bigdata\/package\/spark-examples_2.11-2.0.1.jar"}}],"
> > > sandbox_directory":"\/var\/lib\/mesos\/agent\/slaves\/
> > > 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-S7\/frameworks\/
> > > 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-0002\/executors\/
> > > driver-20161110182510-0001\/runs\/b561328e-9110-4583-b740-
> > > 98f9653e7fc2","user":"root"}
> > > I1110 18:25:11.099799   301 fetcher.cpp:409] Fetching URI 'hdfs://
> > > 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
> > > I1110 18:25:11.099820   301 fetcher.cpp:250] Fetching directly into the
> > > sandbox directory
> > > I1110 18:25:11.099862   301 fetcher.cpp:187] Fetching URI 'hdfs://
> > > 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
> > > E1110 18:25:11.101842   301 shell.hpp:106] Command 'hadoop version
> 2>&1'
> > > failed; this is the output:
> > > sh: hadoop: command not found
> > > Failed to fetch 'hdfs://192.168.111.74:9090/bigdata/package/spark-
> > > examples_2.11-2.0.1.jar': Failed to create HDFS client: Failed to
> execute
> > > 'hadoop version 2>&1'; the command was either not found or exited with
> a
> > > non-zero exit status: 127
> > > Failed to synchronize with agent (it's probably exited
> > >
> > >
> > > Actually I installed hadoop on each agent node.
> > >
> > >
> > > Any advice?
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Jared, (??)
> > > Software developer
> > > Interested in open source software, big data, Linux
> > >
> >
>
>
>
> --
> Deshi Xiao
> Twitter: xds2000
> E-mail: xiaods(AT)gmail.com
>



--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Re: Failed to launch spark jobs on mesos due to "hadoop" not found

[Yu Wei]

I fixed the problem by adding environment variables HADOOP_HOME and JAVA_HOME 
to launch mesos-agent as below,

sudo HADOOP_HOME=/home/dcos/repo/bigdata/hadoop-2.7.3/ 
JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk-1.8.0.111-1.b15.el7_2.x86_64/jre/ 
/usr/local/sbin/mesos-agent --work_dir=/var/lib/mesos/agent  ...

It seemed that this is caused by sudo secure_path.


Is there any better solution to fix this problem?



Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: tommy xiao <xia...@gmail.com>
Sent: Friday, November 11, 2016 1:52:54 PM
To: dev
Subject: Re: Failed to launch spark jobs on mesos due to "hadoop" not found

it seem like in same user mode.

2016-11-10 22:37 GMT+08:00 Joris Van Remoortere <jo...@mesosphere.io>:

> Switch to the user that the agent is running as, and the directory from
> which it is executing and ensure that you can find hadoop. For example by
> running
> `which hadoop`. This may be a PATH or JAVA_HOME issue?
>
> ―
> *Joris Van Remoortere*
> Mesosphere
>
> On Thu, Nov 10, 2016 at 6:29 AM, Yu Wei <yu20...@hotmail.com> wrote:
>
> > Hi Guys,
> >
> > I failed to launch spark jobs on mesos. Actually I submitted the job to
> > cluster successfully.
> >
> > But the job failed to run.
> >
> > I1110 18:25:11.095507   301 fetcher.cpp:498] Fetcher Info:
> > {"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/
> 1f8e621b-3cbf-4b86-a1c1-
> > 9e2cf77265ee-S7\/root","items":[{"action":"BYPASS_CACHE","
> > uri":{"extract":true,"value":"hdfs:\/\/192.168.111.74:9090\/
> > bigdata\/package\/spark-examples_2.11-2.0.1.jar"}}],"
> > sandbox_directory":"\/var\/lib\/mesos\/agent\/slaves\/
> > 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-S7\/frameworks\/
> > 1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-0002\/executors\/
> > driver-20161110182510-0001\/runs\/b561328e-9110-4583-b740-
> > 98f9653e7fc2","user":"root"}
> > I1110 18:25:11.099799   301 fetcher.cpp:409] Fetching URI 'hdfs://
> > 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
> > I1110 18:25:11.099820   301 fetcher.cpp:250] Fetching directly into the
> > sandbox directory
> > I1110 18:25:11.099862   301 fetcher.cpp:187] Fetching URI 'hdfs://
> > 192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
> > E1110 18:25:11.101842   301 shell.hpp:106] Command 'hadoop version 2>&1'
> > failed; this is the output:
> > sh: hadoop: command not found
> > Failed to fetch 'hdfs://192.168.111.74:9090/bigdata/package/spark-
> > examples_2.11-2.0.1.jar': Failed to create HDFS client: Failed to execute
> > 'hadoop version 2>&1'; the command was either not found or exited with a
> > non-zero exit status: 127
> > Failed to synchronize with agent (it's probably exited
> >
> >
> > Actually I installed hadoop on each agent node.
> >
> >
> > Any advice?
> >
> >
> >
> > Thanks,
> >
> > Jared, (??)
> > Software developer
> > Interested in open source software, big data, Linux
> >
>



--
Deshi Xiao
Twitter: xds2000
E-mail: xiaods(AT)gmail.com

Failed to launch spark jobs on mesos due to "hadoop" not found

[Yu Wei]

Hi Guys,

I failed to launch spark jobs on mesos. Actually I submitted the job to cluster 
successfully.

But the job failed to run.

I1110 18:25:11.095507   301 fetcher.cpp:498] Fetcher Info: 
{"cache_directory":"\/tmp\/mesos\/fetch\/slaves\/1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-S7\/root","items":[{"action":"BYPASS_CACHE","uri":{"extract":true,"value":"hdfs:\/\/192.168.111.74:9090\/bigdata\/package\/spark-examples_2.11-2.0.1.jar"}}],"sandbox_directory":"\/var\/lib\/mesos\/agent\/slaves\/1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-S7\/frameworks\/1f8e621b-3cbf-4b86-a1c1-9e2cf77265ee-0002\/executors\/driver-20161110182510-0001\/runs\/b561328e-9110-4583-b740-98f9653e7fc2","user":"root"}
I1110 18:25:11.099799   301 fetcher.cpp:409] Fetching URI 
'hdfs://192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
I1110 18:25:11.099820   301 fetcher.cpp:250] Fetching directly into the sandbox 
directory
I1110 18:25:11.099862   301 fetcher.cpp:187] Fetching URI 
'hdfs://192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar'
E1110 18:25:11.101842   301 shell.hpp:106] Command 'hadoop version 2>&1' 
failed; this is the output:
sh: hadoop: command not found
Failed to fetch 
'hdfs://192.168.111.74:9090/bigdata/package/spark-examples_2.11-2.0.1.jar': 
Failed to create HDFS client: Failed to execute 'hadoop version 2>&1'; the 
command was either not found or exited with a non-zero exit status: 127
Failed to synchronize with agent (it's probably exited


Actually I installed hadoop on each agent node.


Any advice?



Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

Which service discovery mechanism could be used with mesos?

[Yu Wei]

Hi Guys,


I deployed a mesos cluster with zookeeper.

Which methods could be used for service discovery and load balance?


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: Mesos UI problem

[Yu Wei]

@haosdent<mailto:haosd...@gmail.com>
It works well. Thank you.


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: haosdent <haosd...@gmail.com>
Sent: Friday, November 4, 2016 11:49:35 AM
To: dev
Cc: Pankaj Saha
Subject: Re: Mesos UI problem

A workaround to avoid edit /etc/hosts is use

--ip=192.168.111.65 --hostname=192.168.111.65 to start agent.

On Fri, Nov 4, 2016 at 11:42 AM, Yu Wei <yu20...@hotmail.com> wrote:

> Hi,
>
> Command I used as below,
>
>
> sudo /usr/local/sbin/mesos-master  --work_dir=/var/lib/mesos/master
> --zk=zk://192.168.111.53:2181,192.168.111.55:2181,192.168.
> 111.54:2181/mesos --quorum=2 --cluster="cluster 0.1" --ip=192.168.111.53
>
> sudo /usr/local/sbin/mesos-agent --work_dir=/var/lib/mesos/agent
> --master=zk://192.168.111.53:2181,192.168.111.55:2181,192.
> 168.111.54:2181/mesos --ip=192.168.111.65 --hostname=`hostname`
>
>
> However, after adding all master/agent nodes to /etc/hosts in which
> browser was launched, it worked well now.
>
>
> I think it's DNS problem.
>
> Any thoughts about this?
>
>
> Thanks,
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
> 
> From: Pankaj Saha <psa...@binghamton.edu>
> Sent: Thursday, November 3, 2016 10:55:54 PM
> To: dev@mesos.apache.org
> Subject: Re: Mesos UI problem
>
> is your slave name appears as your masters's name and master's name is
> blank?
> then your mesos master is not able to connect with mesos agent correctly.
> can you send the command line parameters you set while running your mesos
> master and agent?
>
> On Wed, Nov 2, 2016 at 11:32 PM, haosdent <haosd...@gmail.com> wrote:
>
> > Hi, @yuwei Mailing list doesn't support send images. May you add me in
> the
> > recipients? I would like to help check.
> >
> > On Thu, Nov 3, 2016 at 2:26 PM, Yu Wei <yu20...@hotmail.com> wrote:
> >
> > > Hi Guys,
> > >
> > > I setup Mesos cluster with zookeeper enabled.
> > >
> > > I found UI problem when checking agent status.
> > >
> > > As showed in below snapshot, the agent name is not correct. And master
> > > name is empty.
> > >
> > > Actually the master name should be "mesos2".
> > >
> > >
> > > Any advice about this?
> > >
> > >
> > >
> > >
> > > Thanks,
> > >
> > > Jared, (韦煜）
> > > Software developer
> > > Interested in open source software, big data, Linux
> > >
> >
> >
> >
> > --
> > Best Regards,
> > Haosdent Huang
> >
>



--
Best Regards,
Haosdent Huang

Re: Mesos UI problem

[Yu Wei]

Hi,

Command I used as below,


sudo /usr/local/sbin/mesos-master  --work_dir=/var/lib/mesos/master 
--zk=zk://192.168.111.53:2181,192.168.111.55:2181,192.168.111.54:2181/mesos 
--quorum=2 --cluster="cluster 0.1" --ip=192.168.111.53

sudo /usr/local/sbin/mesos-agent --work_dir=/var/lib/mesos/agent 
--master=zk://192.168.111.53:2181,192.168.111.55:2181,192.168.111.54:2181/mesos 
--ip=192.168.111.65 --hostname=`hostname`


However, after adding all master/agent nodes to /etc/hosts in which browser was 
launched, it worked well now.


I think it's DNS problem.

Any thoughts about this?


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Pankaj Saha <psa...@binghamton.edu>
Sent: Thursday, November 3, 2016 10:55:54 PM
To: dev@mesos.apache.org
Subject: Re: Mesos UI problem

is your slave name appears as your masters's name and master's name is
blank?
then your mesos master is not able to connect with mesos agent correctly.
can you send the command line parameters you set while running your mesos
master and agent?

On Wed, Nov 2, 2016 at 11:32 PM, haosdent <haosd...@gmail.com> wrote:

> Hi, @yuwei Mailing list doesn't support send images. May you add me in the
> recipients? I would like to help check.
>
> On Thu, Nov 3, 2016 at 2:26 PM, Yu Wei <yu20...@hotmail.com> wrote:
>
> > Hi Guys,
> >
> > I setup Mesos cluster with zookeeper enabled.
> >
> > I found UI problem when checking agent status.
> >
> > As showed in below snapshot, the agent name is not correct. And master
> > name is empty.
> >
> > Actually the master name should be "mesos2".
> >
> >
> > Any advice about this?
> >
> >
> >
> >
> > Thanks,
> >
> > Jared, (韦煜）
> > Software developer
> > Interested in open source software, big data, Linux
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Mesos UI problem

[Yu Wei]

Hi Guys,

I setup Mesos cluster with zookeeper enabled.

I found UI problem when checking agent status.

As showed in below snapshot, the agent name is not correct. And master name is 
empty.

Actually the master name should be "mesos2".


Any advice about this?


[cid:4de372d2-c9fd-4490-b18f-1918d313335b]


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: mesos 1.0.1 build error on CentOS 7.2

[Yu Wei]

It seems that it's python problem. It failed to get correct result when 
comparing "pytz 2012d" and "pytz 2010".

After update "pytz>=2010d" to "pytz>=2010d" in 
mesos-1.0.1/build/3rdparty/protobuf-2.6.1/python/.eggs/google_apputils-0.4.2-py2.7.egg/EGG-INFOrequires.txt,
 build succeeded.


Thanks,

Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: haosdent <haosd...@gmail.com>
Sent: Tuesday, November 1, 2016 2:30:54 PM
To: dev
Subject: Re: mesos 1.0.1 build error on CentOS 7.2

Have you install pytz in your operate system? What's the output of `pip
freeze|grep pytz`?

On Tue, Nov 1, 2016 at 2:17 PM, Yu Wei <yu20...@hotmail.com> wrote:

> Hi Guys,
>
> I tried to build mesos 1.0.1 on centos 7.2 and met following error.
>
> Building protobuf Python egg ...
> cd ../3rdparty/protobuf-2.6.1/python &&\
>   CC="gcc"\
>   CXX="g++"\
>   CFLAGS="-g1 -O0 -Wno-unused-local-typedefs"\
>   CXXFLAGS="-g1 -O0 -Wno-unused-local-typedefs -std=c++11"\
>   PYTHONPATH=/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0
>   \
>   /usr/bin/python setup.py build bdist_egg
>
> Installed /home/dcos/repo/mesos-1.0.1/build/3rdparty/protobuf-2.6.1/
> python/.eggs/google_apputils-0.4.2-py2.7.egg
> Traceback (most recent call last):
>   File "setup.py", line 200, in 
> "Protocol Buffers are Google's data interchange format.",
>   File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
> _setup_distribution = dist = klass(attrs)
>   File 
> "/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/setuptools/dist.py",
> line 269, in __init__
> self.fetch_build_eggs(attrs['setup_requires'])
>   File 
> "/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/setuptools/dist.py",
> line 313, in fetch_build_eggs
> replace_conflicting=True,
>   File "/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.
> 9.0/pkg_resources/__init__.py", line 826, in resolve
> dist = best[req.key] = env.best_match(req, ws, installer)
>   File "/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.
> 9.0/pkg_resources/__init__.py", line 1085, in best_match
> dist = working_set.find(req)
>   File "/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.
> 9.0/pkg_resources/__init__.py", line 695, in find
> raise VersionConflict(dist, req)
> pkg_resources.VersionConflict: (pytz 2012d (/usr/lib/python2.7/site-packages),
> Requirement.parse('pytz>=2010'))
> make[2]: *** [../3rdparty/protobuf-2.6.1/python/dist/protobuf-2.6.1-py2.7.egg]
> Error 1
> make[2]: Leaving directory `/home/dcos/repo/mesos-1.0.1/build/src'
> make[1]: *** [all] Error 2
> make[1]: Leaving directory `/home/dcos/repo/mesos-1.0.1/build/src'
> make: *** [all-recursive] Error 1
>
>
> Any thoughts about this?
>
>
> Thanks,
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>



--
Best Regards,
Haosdent Huang

mesos 1.0.1 build error on CentOS 7.2

[Yu Wei]

Hi Guys,

I tried to build mesos 1.0.1 on centos 7.2 and met following error.

Building protobuf Python egg ...
cd ../3rdparty/protobuf-2.6.1/python &&\
  CC="gcc"\
  CXX="g++"\
  CFLAGS="-g1 -O0 -Wno-unused-local-typedefs"\
  CXXFLAGS="-g1 -O0 -Wno-unused-local-typedefs -std=c++11"\
  PYTHONPATH=/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0\
  /usr/bin/python setup.py build bdist_egg

Installed 
/home/dcos/repo/mesos-1.0.1/build/3rdparty/protobuf-2.6.1/python/.eggs/google_apputils-0.4.2-py2.7.egg
Traceback (most recent call last):
  File "setup.py", line 200, in 
"Protocol Buffers are Google's data interchange format.",
  File "/usr/lib64/python2.7/distutils/core.py", line 112, in setup
_setup_distribution = dist = klass(attrs)
  File 
"/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/setuptools/dist.py",
 line 269, in __init__
self.fetch_build_eggs(attrs['setup_requires'])
  File 
"/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/setuptools/dist.py",
 line 313, in fetch_build_eggs
replace_conflicting=True,
  File 
"/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/pkg_resources/__init__.py",
 line 826, in resolve
dist = best[req.key] = env.best_match(req, ws, installer)
  File 
"/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/pkg_resources/__init__.py",
 line 1085, in best_match
dist = working_set.find(req)
  File 
"/home/dcos/repo/mesos-1.0.1/build/3rdparty/setuptools-20.9.0/pkg_resources/__init__.py",
 line 695, in find
raise VersionConflict(dist, req)
pkg_resources.VersionConflict: (pytz 2012d (/usr/lib/python2.7/site-packages), 
Requirement.parse('pytz>=2010'))
make[2]: *** [../3rdparty/protobuf-2.6.1/python/dist/protobuf-2.6.1-py2.7.egg] 
Error 1
make[2]: Leaving directory `/home/dcos/repo/mesos-1.0.1/build/src'
make[1]: *** [all] Error 2
make[1]: Leaving directory `/home/dcos/repo/mesos-1.0.1/build/src'
make: *** [all-recursive] Error 1


Any thoughts about this?


Thanks,

Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: How to select Containerizer? Mesos containerizer or Docker?

[Yu Wei]

Hi Qian,


Thanks for your explanation.


Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Qian Zhang <zhq527...@gmail.com>
Sent: Wednesday, August 24, 2016 10:56:07 AM
To: user
Cc: dev@mesos.apache.org
Subject: Re: How to select Containerizer? Mesos containerizer or Docker?

For port mapping, we are actively working on it now, see
https://issues.apache.org/jira/browse/MESOS-6014 for details.

Mesos containerizer is the long term choice which we will keep enhancing
and adding new features to, it supports multiple image formats rather than
just Docker, and easily to be customized. And I believe Docker
containerizer will be eventually deprecated sometime in future.


Thanks,
Qian Zhang

On Tue, Aug 23, 2016 at 5:42 PM, Olivier Sallou <olivier.sal...@irisa.fr>
wrote:

>
>
> On 08/23/2016 11:36 AM, Yu Wei wrote:
> > Hi,
> >
> >
> > Which containerizer should be used? Mesos, Docker or other?
> >
> > Is there any principles to help making decision?
> with unified containerizer, mesos pushes for Mesos containerizer.
> However, it does not support for the moment port mapping. So if you need
> port mapping, you should go to Docker one.
>
> Olivier
> >
> >
> > Thanks,
> >
> >
> > Jared, (??)
> > Software developer
> > Interested in open source software, big data, Linux
> >
>
> --
>
>
> gpg key id: 4096R/326D8438  (keyring.debian.org)
> Key fingerprint = 5FB4 6F83 D3B9 5204 6335  D26D 78DC 68DB 326D 8438
>
>

How to select Containerizer? Mesos containerizer or Docker?

[Yu Wei]

Hi,


Which containerizer should be used? Mesos, Docker or other?

Is there any principles to help making decision?


Thanks,


Jared, (??)
Software developer
Interested in open source software, big data, Linux

launch marathon app but no leave "waiting" on mesos

[Yu Wei]

Hi guys,

I'm running a simple marathon application "An Inline Shell Script".

After deploying, the app doesn't leave "waiting".


In mesos/state, I found following stuff.

"root_submissions":"true","user_sorter":"drf","version":"false","webui_dir":"\/usr\/local\/share\/mesos\/webui","work_dir":"\/opensrc\/cloud\/workbench","zk_session_timeout":"10secs"},"slaves":[{"id":"5c79ff49-8180-4d0b-8fcd-82eaf47c5a72-S0","pid":"slave(1)@127.0.1.1:5051","hostname":"jared-QiTianM4550-D680","registered_time":1471247912.3384,"reregistered_time":1471250972.04209,"resources":{"disk":516395.0,"mem":14970.0,"gpus":0.0,"cpus":8.0,"ports":"[31000-32000]"},"used_resources":{"disk":0.0,"mem":0.0,"gpus":0.0,"cpus":0.0},"offered_resources":{"disk":0.0,"mem":0.0,"gpus":0.0,"cpus":0.0},"reserved_resources":{},"unreserved_resources":{"disk":516395.0,"mem":14970.0,"gpus":0.0,"cpus":8.0,"ports":"[31000-32000]"},"attributes":{},"active":true,"version":"1.1.0"}],"frameworks":[],"completed_frameworks":[],"orphan_tasks":[],"unregistered_frameworks":[]}


I started mesos master and agent as below,

sudo mesos-master --work_dir=/opensrc/cloud/workbench

sudo mesos-agent --master=127.0.1.1:5050 --containerizers=docker,mesos 
--isolation=docker/runtime,filesystem/linux --work_dir=/opensrc/cloud/workbench 
--image_providers=docker


Then I launched marathon as below,

bin/start --master localhost:2181/mesos --zk zk://localhost:2181/marathon 
--logging_level DEBUG


Then create app to execute simple shell command, it doesn't leave waiting.  Do 
I need to specify image for docker/container?


Any advice?


Jared, (??)
Software developer
Interested in open source software, big data, Linux

what's the difference between mesos and yarn?

[Yu Wei]

Hi guys,


What's the difference between yarn and mesos in practice?


If using yarn, does container still needed?


Thanks,


Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: Configure error when try to build mesos from source on ubuntu

[Yu Wei]

Good advice.

Thanks very much.


Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Marco Massenzio <m.massen...@gmail.com>
Sent: Thursday, August 11, 2016 1:57:40 AM
To: dev
Subject: Re: Configure error when try to build mesos from source on ubuntu

As a follow up,  for those who may not need Java or Python support in their
build, you can use --disable-java and --disable-python flags; this is how I
usually run it:

mkdir build && cd build
../configure --prefix /my/path/to/local --disable-java --disable-python
make -j 10 V=0

(I use the --prefix so that the generated libraries during the make install
step won't "pollute" my system's /usr/local/bin).

You'll miss the build of the Java/Python protobuf, and probably other
stuff, but it's usually good enough to experiment with C++ frameworks,
features, etc.

YMMV


--
*Marco Massenzio*
http://codetrips.com

On Sat, Aug 6, 2016 at 9:54 AM, Yu Wei <yu20...@hotmail.com> wrote:

> Yes, you're right. My java_home is not correct.
>
> Now the problem is fixed.
>
>
> Thanks very much.
>
>
> Jared, (韦煜）
> Software developer
> Interested in open source software, big data, Linux
>
> 
> From: haosdent <haosd...@gmail.com>
> Sent: Sunday, August 7, 2016 12:39:09 AM
> To: dev
> Subject: Re: Configure error when try to build mesos from source on ubuntu
>
> Hi, @Yu Yei Mesos requires JDK not only JRE. Please follow
> https://github.com/apache/mesos/blob/master/docs/
> getting-started.md#ubuntu-1404
> to getting start
>
> ```
> # Install the latest OpenJDK.
> $ sudo apt-get install -y openjdk-7-jdk
> ```
>
> On Sun, Aug 7, 2016 at 12:30 AM, Yu Wei <yu20...@hotmail.com> wrote:
>
> > Hi guys,
> >
> >
> > I tried to build mesos 1.0.0 from source on Ubuntu.
> >
> > I met following problem when tried to configure.
> >
> > checking for curl_global_init in -lcurl... yes
> > configure: error: failed to determine linker flags for using Java (bad
> > JAVA_HOME or missing support for your architecture?)
> >
> >
> > My JAVA_HOME is actually set to  "/usr/lib/jvm/java-8-oracle/jre".
> >
> >
> > Any hints about this issue?
> >
> >
> > Thx,
> >
> >
> > Jared, (??)
> > Software developer
> > Interested in open source software, big data, Linux
> >
>
>
>
> --
> Best Regards,
> Haosdent Huang
>

Re: How could Mesos work with docker?

[Yu Wei]

Hi Tomek,


Good point.
Thanks very much for your help.


Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: Tomek Janiszewski <jani...@gmail.com>
Sent: Tuesday, August 9, 2016 10:06:28 PM
To: dev@mesos.apache.org; u...@mesos.apache.org
Subject: Re: How could Mesos work with docker?

Hi

Docker support is described here 
http://mesos.apache.org/documentation/latest/docker-containerizer/

Best
Tomek

wt., 9.08.2016 o 16:04 u?ytkownik Yu Wei 
<yu20...@hotmail.com<mailto:yu20...@hotmail.com>> napisa?:

Hi guys,

I start learning mesos.

How could mesos work with docker? Is there any document about this?


Any other advice?


Thx,


Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux

How could Mesos work with docker?

[Yu Wei]

Hi guys,

I start learning mesos.

How could mesos work with docker? Is there any document about this?


Any other advice?


Thx,


Jared, (??)
Software developer
Interested in open source software, big data, Linux

Re: Configure error when try to build mesos from source on ubuntu

[Yu Wei]

Yes, you're right. My java_home is not correct.

Now the problem is fixed.


Thanks very much.


Jared, (韦煜）
Software developer
Interested in open source software, big data, Linux


From: haosdent <haosd...@gmail.com>
Sent: Sunday, August 7, 2016 12:39:09 AM
To: dev
Subject: Re: Configure error when try to build mesos from source on ubuntu

Hi, @Yu Yei Mesos requires JDK not only JRE. Please follow
https://github.com/apache/mesos/blob/master/docs/getting-started.md#ubuntu-1404
to getting start

```
# Install the latest OpenJDK.
$ sudo apt-get install -y openjdk-7-jdk
```

On Sun, Aug 7, 2016 at 12:30 AM, Yu Wei <yu20...@hotmail.com> wrote:

> Hi guys,
>
>
> I tried to build mesos 1.0.0 from source on Ubuntu.
>
> I met following problem when tried to configure.
>
> checking for curl_global_init in -lcurl... yes
> configure: error: failed to determine linker flags for using Java (bad
> JAVA_HOME or missing support for your architecture?)
>
>
> My JAVA_HOME is actually set to  "/usr/lib/jvm/java-8-oracle/jre".
>
>
> Any hints about this issue?
>
>
> Thx,
>
>
> Jared, (??)
> Software developer
> Interested in open source software, big data, Linux
>



--
Best Regards,
Haosdent Huang

Configure error when try to build mesos from source on ubuntu

[Yu Wei]

Hi guys,


I tried to build mesos 1.0.0 from source on Ubuntu.

I met following problem when tried to configure.

checking for curl_global_init in -lcurl... yes
configure: error: failed to determine linker flags for using Java (bad 
JAVA_HOME or missing support for your architecture?)


My JAVA_HOME is actually set to  "/usr/lib/jvm/java-8-oracle/jre".


Any hints about this issue?


Thx,


Jared, (??)
Software developer
Interested in open source software, big data, Linux