You need to ensure there is no overlap between these 3 things: 1. Static ports for mesos agent/master, e.g. 5050, 5051 by default. 2. Linux ephemeral port range (32768-61000 by default). 3. Mesos view of ports as resources (31000-32000 by default).
The behavior initially described here sounds to me like a scheduler may have been offered an already used port on the agent host. I answered a related question before: http://unix.stackexchange.com/a/237543 - Erik On Thursday, September 1, 2016, haosdent <[email protected]> wrote: > If you use Linux, you could execute follow command on every Mesos Agent to > make the ephemeral port assigned during 5000~6000 > > echo "5000 6000" > /proc/sys/net/ipv4/ip_local_port_range > > On Thu, Sep 1, 2016 at 3:12 PM, Vinod Kone <[email protected] > <javascript:;>> wrote: > > > AFAICT, your agent is listening on port 8082 and not the default 5051. > > --------- > > I0829 14:24:21.750063 2679 slave.cpp:193] Slave started on 1)@ > > 128.226.116.69:8082 > > ------------ > > > > The fact that agent is receiving a task from the master means that the > > firewall on the agent allows incoming connections to 8082. So I'm > surprised > > that a local connection from the executor to the agent is being denied. > > What exactly are your firewall rules on the agent? > > > > Also, can you share the stderr/stdout of an example executor? > > > > > > On Wed, Aug 31, 2016 at 6:18 PM, Pankaj Saha <[email protected] > <javascript:;>> > > wrote: > > > > > I think the executor wants to get registered by communicating with > > > mesos master and it fails due to network restriction. > > > How can I change the /tmp/ path? I have mentioned /var/lib/mesos as my > > > work_dir. > > > > > > > > > *I am explaining my setup here:* > > > I have a Mesos setup where master and slave both are running on the > same > > > network of my university campus. Mesos agent node is situated under a > > > firewall and only port: 5000 to port:6000 are open for incoming traffic > > > whereas Mesos master has no such restrictions. I am running master > > service > > > on master:5050 and agent is running on agent:5051 as default. > > > > > > I can see agent is communicating correctly to master and offering the > > > available resources. I have mentioned the available ports for agents > are > > > ports:[5001-6000] in *src/slave/constants.cpp* file so that framework > can > > > communicate only through those ports which are open for my agent system > > > behind the firewall. > > > > > > Now when I am launching jobs through Mesosphere marathon framework, I > can > > > see all jobs are connected to mesos-agent through those mentioned port > > > ranges[5001-6000]. But my jobs are not getting submitted. So I started > > > debugging and realised that when launching jobs mesos slaves create and > > > launch an executor (*/erc/executor/executor.cpp*) which communicates to > > the > > > mesos master through a random port. Which is outside my available range > > of > > > 5000-6000 open ports. Now as through those ports my agent machine can > not > > > take any requests so executor is getting timed out and restarting the > > > executor again and again after every 1 min of time limit. > > > > > > I could not find out where exactly that random port is assigned. Is > there > > > any socket connection that we can change to get executor connection > > happen > > > on desired range of ports? Please let me know if my understanding is > > > correct and how can I change those ports for executor registration. > > > > > > > > > > > > > > > On Wed, Aug 31, 2016 at 3:09 AM, haosdent <[email protected] > <javascript:;>> wrote: > > > > > > > >I0829 14:27:38.322805 2700 slave.cpp:4307] *Terminating executor > > > > ''test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' of framework > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' because it did not > register > > > > within 1mins > > > > > > > > This log looks wired. Could you find anything in the stdout/stderr of > > the > > > > executor. For the executor 'test.1fb85a35-6e16-11e6-bec9- > c27afc834a0c' > > > > above, it should be under the folder '/tmp/mesos/slaves/d6f0e3e2- > > > > d144-4275-9d38-82327408622b-S12/frameworks/c796100f-9ecb- > > > > 46fa-90a2-72ad649c5dd3-0000/executors/test.1fb85a35-6e16- > > > > 11e6-bec9-c27afc834a0c/runs/dff399f0-beb1-4c49-bd8e-c19621de2f71/' > > > > > > > > Apart from that, run mesos under '/tmp' is not recommended. > > > > > > > > On Tue, Aug 30, 2016 at 2:32 AM, Pankaj Saha <[email protected] > <javascript:;>> > > > > wrote: > > > > > > > > > here is the log: > > > > > > > > > > > > > > > > > > > > I0829 14:24:21.727960 2679 main.cpp:223] Build: 2016-08-28 13:39:46 > > by > > > > > root > > > > > I0829 14:24:21.728159 2679 main.cpp:225] Version: 0.28.2 > > > > > I0829 14:24:21.733256 2679 containerizer.cpp:149] Using isolation: > > > > > posix/cpu,posix/mem,filesystem/posix > > > > > I0829 14:24:21.738895 2679 linux_launcher.cpp:101] Using > > > > > /sys/fs/cgroup/freezer as the freezer hierarchy for the Linux > > launcher > > > > > I0829 14:24:21.748019 2679 main.cpp:328] Starting Mesos slave > > > > > I0829 14:24:21.750063 2679 slave.cpp:193] Slave started on 1)@ > > > > > 128.226.116.69:8082 > > > > > I0829 14:24:21.750114 2679 slave.cpp:194] Flags at startup: > > > > > --advertise_ip="128.226.116.69" --appc_simple_discovery_uri_ > > > > > prefix="http://"; > > > > > --appc_store_dir="/tmp/mesos/store/appc" --authenticatee="crammd5" > > > > > --cgroups_cpu_enable_pids_and_tids_count="false" > > > > > --cgroups_enable_cfs="false" --cgroups_hierarchy="/sys/fs/cgroup" > > > > > --cgroups_limit_swap="false" --cgroups_root="mesos" > > > > > --container_disk_watch_interval="15secs" --containerizers="mesos" > > > > > --default_role="*" --disk_watch_interval="1mins" --docker="docker" > > > > > --docker_kill_orphans="true" --docker_registry="https:// > > > > > registry-1.docker.io" > > > > > --docker_remove_delay="6hrs" --docker_socket="/var/run/ > docker.sock" > > > > > --docker_stop_timeout="0ns" --docker_store_dir="/tmp/ > > > mesos/store/docker" > > > > > --enforce_container_disk_quota="false" > > > > > --executor_registration_timeout="1mins" > > > > > --executor_shutdown_grace_period="5secs" > > > > > --fetcher_cache_dir="/tmp/mesos/fetch" --fetcher_cache_size="2GB" > > > > > --frameworks_home="" --gc_delay="1weeks" --gc_disk_headroom="0.1" > > > > > --hadoop_home="" --help="false" --hostname_lookup="true" > > > > > --image_provisioner_backend="copy" --initialize_driver_logging=" > > true" > > > > > --isolation="posix/cpu,posix/mem" > > > > > --launcher_dir="/home/pankaj/mesos-0.28.2/build/src" > > --logbufsecs="0" > > > > > --logging_level="INFO" --master="129.114.110.143:5050" > > > > > --oversubscribed_resources_interval="15secs" > > --perf_duration="10secs" > > > > > --perf_interval="1mins" --port="8082" > --qos_correction_interval_min= > > > > "0ns" > > > > > --quiet="false" --recover="reconnect" --recovery_timeout="15mins" > > > > > --registration_backoff_factor="1secs" > --revocable_cpu_low_priority=" > > > > true" > > > > > --sandbox_directory="/mnt/mesos/sandbox" --strict="true" > > > > > --switch_user="true" --systemd_enable_support="true" > > > > > --systemd_runtime_directory="/run/systemd/system" > --version="false" > > > > > --work_dir="/tmp/mesos" > > > > > I0829 14:24:21.753572 2679 slave.cpp:464] Slave resources: > > cpus(*):2; > > > > > mem(*):2855; disk(*):84691; ports(*):[8081-8081] > > > > > I0829 14:24:21.753706 2679 slave.cpp:472] Slave attributes: [ ] > > > > > I0829 14:24:21.753762 2679 slave.cpp:477] Slave hostname: > > > > > venom.cs.binghamton.edu > > > > > I0829 14:24:21.770992 2696 state.cpp:58] Recovering state from > > > > > '/tmp/mesos/meta' > > > > > I0829 14:24:21.771304 2696 state.cpp:698] No checkpointed > resources > > > > found > > > > > at '/tmp/mesos/meta/resources/resources.info' > > > > > I0829 14:24:21.771644 2696 state.cpp:101] Failed to find the > latest > > > > slave > > > > > from '/tmp/mesos/meta' > > > > > I0829 14:24:21.772583 2696 status_update_manager.cpp:200] > Recovering > > > > > status update manager > > > > > I0829 14:24:21.773082 2698 containerizer.cpp:407] Recovering > > > > containerizer > > > > > I0829 14:24:21.777489 2702 provisioner.cpp:245] Provisioner > recovery > > > > > complete > > > > > I0829 14:24:21.778149 2699 slave.cpp:4550] Finished recovery > > > > > I0829 14:24:21.779564 2699 slave.cpp:796] New master detected at > > > > > [email protected]:5050 > > > > > I0829 14:24:21.779742 2697 status_update_manager.cpp:174] Pausing > > > > sending > > > > > status updates > > > > > I0829 14:24:21.780607 2699 slave.cpp:821] No credentials provided. > > > > > Attempting to register without authentication > > > > > I0829 14:24:21.781394 2699 slave.cpp:832] Detecting new master > > > > > I0829 14:24:22.698812 2702 slave.cpp:971] Registered with master > > > > > [email protected]:5050; given slave ID > > > > > d6f0e3e2-d144-4275-9d38-82327408622b-S12 > > > > > I0829 14:24:22.699113 2698 status_update_manager.cpp:181] Resuming > > > > sending > > > > > status updates > > > > > I0829 14:24:22.700258 2702 slave.cpp:1030] Forwarding total > > > > oversubscribed > > > > > resources > > > > > I0829 14:24:43.638958 2695 http.cpp:190] HTTP GET for > > /slave(1)/state > > > > from > > > > > 128.226.119.78:59261 with User-Agent='Mozilla/5.0 (X11; Linux > > x86_64) > > > > > AppleWebKit/537.36 (KHTML, like Gecko) Chrome/50.0.2661.86 > > > Safari/537.36' > > > > > I0829 14:25:21.764268 2702 slave.cpp:4359] Current disk usage > 9.67%. > > > Max > > > > > allowed age: 5.622868987169502days > > > > > I0829 14:26:21.778849 2695 slave.cpp:4359] Current disk usage > 9.67%. > > > Max > > > > > allowed age: 5.622860462326585days > > > > > I0829 14:26:38.271085 2698 slave.cpp:1361] Got assigned task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c for framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:26:38.311063 2698 slave.cpp:1480] Launching task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c for framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:26:38.314755 2698 paths.cpp:528] Trying to chown > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > executors/test.1fb85a35-6e16-11e6-bec9-c27afc834a0c/runs/ > > > > > dff399f0-beb1-4c49-bd8e-c19621de2f71' > > > > > to user 'root' > > > > > I0829 14:26:38.320300 2698 slave.cpp:5352] Launching executor > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 with resources > > cpus(*):0.1; > > > > > mem(*):32 in work directory > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > executors/test.1fb85a35-6e16-11e6-bec9-c27afc834a0c/runs/ > > > > > dff399f0-beb1-4c49-bd8e-c19621de2f71' > > > > > I0829 14:26:38.321523 2702 containerizer.cpp:666] Starting > container > > > > > 'dff399f0-beb1-4c49-bd8e-c19621de2f71' for executor > > > > > 'test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' of framework > > > > > 'c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > I0829 14:26:38.322588 2698 slave.cpp:1698] Queuing task > > > > > 'test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' for executor > > > > > 'test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:26:38.358906 2702 linux_launcher.cpp:304] Cloning child > > > process > > > > > with flags = > > > > > I0829 14:26:38.366492 2702 containerizer.cpp:1179] Checkpointing > > > > > executor's forked pid 2758 to > > > > > '/tmp/mesos/meta/slaves/d6f0e3e2-d144-4275-9d38- > > > > > 82327408622b-S12/frameworks/c796100f-9ecb-46fa-90a2- > > > > > 72ad649c5dd3-0000/executors/test.1fb85a35-6e16-11e6-bec9- > > > > > c27afc834a0c/runs/dff399f0-beb1-4c49-bd8e-c19621de2f71/ > > > pids/forked.pid' > > > > > I0829 14:27:21.779755 2701 slave.cpp:4359] Current disk usage > 9.67%. > > > Max > > > > > allowed age: 5.622850415190289days > > > > > I0829 14:27:38.322805 2700 slave.cpp:4307] > > > > > *Terminating executor ''test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' > > of > > > > > framework c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' because it > did > > > not > > > > > register within 1minsI0829 14:27:38.323226 2700 > > containerizer.cpp:1453] > > > > > Destroying container 'dff399f0-beb1-4c49-bd8e-c19621de2f71'* > > > > > I0829 14:27:38.329186 2702 cgroups.cpp:2427] Freezing cgroup > > > > > /sys/fs/cgroup/freezer/mesos/dff399f0-beb1-4c49-bd8e-c19621de2f71 > > > > > I0829 14:27:38.331509 2699 cgroups.cpp:1409] Successfully froze > > cgroup > > > > > /sys/fs/cgroup/freezer/mesos/dff399f0-beb1-4c49-bd8e-c19621de2f71 > > > after > > > > > 2.19392ms > > > > > I0829 14:27:38.334520 2698 cgroups.cpp:2445] Thawing cgroup > > > > > /sys/fs/cgroup/freezer/mesos/dff399f0-beb1-4c49-bd8e-c19621de2f71 > > > > > I0829 14:27:38.337821 2698 cgroups.cpp:1438] Successfullly thawed > > > cgroup > > > > > /sys/fs/cgroup/freezer/mesos/dff399f0-beb1-4c49-bd8e-c19621de2f71 > > > after > > > > > 3.194112ms > > > > > I0829 14:27:38.435214 2696 containerizer.cpp:1689] Executor for > > > > container > > > > > 'dff399f0-beb1-4c49-bd8e-c19621de2f71' has exited > > > > > I0829 14:27:38.441556 2695 provisioner.cpp:306] Ignoring destroy > > > request > > > > > for unknown container dff399f0-beb1-4c49-bd8e-c19621de2f71 > > > > > I0829 14:27:38.442186 2695 slave.cpp:3871] Executor > > > > > 'test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 terminated with signal > > > Killed > > > > > I0829 14:27:38.445689 2695 slave.cpp:3012] Handling status update > > > > > TASK_FAILED (UUID: 154a67ea-f6d5-4e9c-ad3d-9b09161ba34d) for task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 from @0.0.0.0:0 > > > > > W0829 14:27:38.447599 2702 containerizer.cpp:1295] Ignoring update > > for > > > > > unknown container: dff399f0-beb1-4c49-bd8e-c19621de2f71 > > > > > I0829 14:27:38.448391 2702 status_update_manager.cpp:320] Received > > > > status > > > > > update TASK_FAILED (UUID: 154a67ea-f6d5-4e9c-ad3d-9b09161ba34d) > for > > > task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:38.449525 2702 status_update_manager.cpp:824] > > > Checkpointing > > > > > UPDATE for status update TASK_FAILED (UUID: > > > > > 154a67ea-f6d5-4e9c-ad3d-9b09161ba34d) for task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:38.523027 2696 slave.cpp:3410] Forwarding the update > > > > > TASK_FAILED (UUID: 154a67ea-f6d5-4e9c-ad3d-9b09161ba34d) for task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 to > > > [email protected]:5050 > > > > > I0829 14:27:38.627722 2698 status_update_manager.cpp:392] Received > > > > status > > > > > update acknowledgement (UUID: 154a67ea-f6d5-4e9c-ad3d- > 9b09161ba34d) > > > for > > > > > task test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:38.627943 2698 status_update_manager.cpp:824] > > > Checkpointing > > > > > ACK for status update TASK_FAILED (UUID: > > > > > 154a67ea-f6d5-4e9c-ad3d-9b09161ba34d) for task > > > > > test.1fb85a35-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:38.698822 2701 slave.cpp:3975] Cleaning up executor > > > > > 'test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:38.699582 2698 gc.cpp:55] Scheduling > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > executors/test.1fb85a35-6e16-11e6-bec9-c27afc834a0c/runs/ > > > > > dff399f0-beb1-4c49-bd8e-c19621de2f71' > > > > > for gc 6.99999190486222days in the future > > > > > I0829 14:27:38.700202 2698 gc.cpp:55] Scheduling > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > executors/test.1fb85a35-6e16-11e6-bec9-c27afc834a0c' > > > > > for gc 6.99999190029037days in the future > > > > > I0829 14:27:38.700382 2701 slave.cpp:4063] Cleaning up framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:38.700443 2698 gc.cpp:55] Scheduling > > > > > '/tmp/mesos/meta/slaves/d6f0e3e2-d144-4275-9d38- > > > > > 82327408622b-S12/frameworks/c796100f-9ecb-46fa-90a2- > > > > > 72ad649c5dd3-0000/executors/test.1fb85a35-6e16-11e6-bec9- > > > > > c27afc834a0c/runs/dff399f0-beb1-4c49-bd8e-c19621de2f71' > > > > > for gc 6.99999189796148days in the future > > > > > I0829 14:27:38.700649 2698 gc.cpp:55] Scheduling > > > > > '/tmp/mesos/meta/slaves/d6f0e3e2-d144-4275-9d38- > > > > > 82327408622b-S12/frameworks/c796100f-9ecb-46fa-90a2- > > > > > 72ad649c5dd3-0000/executors/test.1fb85a35-6e16-11e6-bec9- > > c27afc834a0c' > > > > > for gc 6.99999189622815days in the future > > > > > I0829 14:27:38.700845 2698 gc.cpp:55] Scheduling > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > for gc 6.99999189143704days in the future > > > > > I0829 14:27:38.701015 2701 status_update_manager.cpp:282] Closing > > > status > > > > > update streams for framework c796100f-9ecb-46fa-90a2- > > 72ad649c5dd3-0000 > > > > > I0829 14:27:38.701161 2698 gc.cpp:55] Scheduling > > > > > '/tmp/mesos/meta/slaves/d6f0e3e2-d144-4275-9d38- > > > > > 82327408622b-S12/frameworks/c796100f-9ecb-46fa-90a2- > > 72ad649c5dd3-0000' > > > > > for gc 6.9999918900237days in the future > > > > > I0829 14:27:39.651463 2697 slave.cpp:1361] Got assigned task > > > > > test.445696e6-6e16-11e6-bec9-c27afc834a0c for framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:39.655815 2696 gc.cpp:83] Unscheduling > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > from gc > > > > > I0829 14:27:39.656445 2696 gc.cpp:83] Unscheduling > > > > > '/tmp/mesos/meta/slaves/d6f0e3e2-d144-4275-9d38- > > > > > 82327408622b-S12/frameworks/c796100f-9ecb-46fa-90a2- > > 72ad649c5dd3-0000' > > > > > from gc > > > > > I0829 14:27:39.656855 2702 slave.cpp:1480] Launching task > > > > > test.445696e6-6e16-11e6-bec9-c27afc834a0c for framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:39.660585 2702 paths.cpp:528] Trying to chown > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > executors/test.445696e6-6e16-11e6-bec9-c27afc834a0c/runs/ > > > > > ea676570-0a2a-49c3-a75c-14e045eb842b' > > > > > to user 'root' > > > > > I0829 14:27:39.666008 2702 slave.cpp:5352] Launching executor > > > > > test.445696e6-6e16-11e6-bec9-c27afc834a0c of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 with resources > > cpus(*):0.1; > > > > > mem(*):32 in work directory > > > > > '/tmp/mesos/slaves/d6f0e3e2-d144-4275-9d38-82327408622b- > > > > > S12/frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > executors/test.445696e6-6e16-11e6-bec9-c27afc834a0c/runs/ > > > > > ea676570-0a2a-49c3-a75c-14e045eb842b' > > > > > I0829 14:27:39.667603 2702 slave.cpp:1698] Queuing task > > > > > 'test.445696e6-6e16-11e6-bec9-c27afc834a0c' for executor > > > > > 'test.445696e6-6e16-11e6-bec9-c27afc834a0c' of framework > > > > > c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > I0829 14:27:39.668207 2702 containerizer.cpp:666] Starting > container > > > > > 'ea676570-0a2a-49c3-a75c-14e045eb842b' for executor > > > > > 'test.445696e6-6e16-11e6-bec9-c27afc834a0c' of framework > > > > > 'c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > I0829 14:27:39.678665 2702 linux_launcher.cpp:304] Cloning child > > > process > > > > > with flags = > > > > > I0829 14:27:39.681824 2702 containerizer.cpp:1179] Checkpointing > > > > > executor's forked pid 2799 to > > > > > '/tmp/mesos/meta/slaves/d6f0e3e2-d144-4275-9d38- > > > > > 82327408622b-S12/frameworks/c796100f-9ecb-46fa-90a2- > > > > > 72ad649c5dd3-0000/executors/test.445696e6-6e16-11e6-bec9- > > > > > c27afc834a0c/runs/ea676570-0a2a-49c3-a75c-14e045eb842b/ > > > pids/forked.pid' > > > > > > > > > > Thanks > > > > > Pankaj > > > > > > > > > > > > > > > On Mon, Aug 29, 2016 at 7:25 AM, haosdent <[email protected] > <javascript:;>> > > wrote: > > > > > > > > > > > Hi, @Pankaj, Could you provide logs during " the job is getting > > > > restarted > > > > > > and a new container is created with a new process id. ". The logs > > you > > > > > > provided looks normal. > > > > > > > > > > > > On Mon, Aug 29, 2016 at 5:26 AM, Pankaj Saha < > > [email protected] <javascript:;>> > > > > > > wrote: > > > > > > > > > > > > > Hi > > > > > > > I am facing an issue with a launched jobs into my mesos > agents. I > > > am > > > > > > trying > > > > > > > to launch a job through marathon framework and job is staying > in > > > > > stagged > > > > > > > state and not running. > > > > > > > I could see the log message at the agent console as below: > > > > > > > > > > > > > > Scheduling > > > > > > > '/var/lib/mesos-8082/meta/slaves/d6f0e3e2-d144-4275- > > > > > > 9d38-82327408622b-S8/ > > > > > > > frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > > > for gc 6.99999884239407days in the future > > > > > > > I0828 16:20:36.053483 28512 slave.cpp:1361] *Got assigned task > > > > > > > test-crixus*.eb66a42b-6d5c-11e6-bec9-c27afc834a0c > > > > > > > for framework c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > > > I0828 16:20:36.056224 28510 gc.cpp:83] Unscheduling > > > > > > > '/var/lib/mesos-8082/slaves/d6f0e3e2-d144-4275-9d38- > > > > > > > 82327408622b-S8/frameworks/c796100f-9ecb-46fa-90a2- > > > > 72ad649c5dd3-0000' > > > > > > > from gc > > > > > > > I0828 16:20:36.056715 28510 gc.cpp:83] Unscheduling > > > > > > > '/var/lib/mesos-8082/meta/slaves/d6f0e3e2-d144-4275- > > > > > > 9d38-82327408622b-S8/ > > > > > > > frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > > > from gc > > > > > > > I0828 16:20:36.057231 28509 slave.cpp:1480] *Launching task > > > > > > > test-crixus*.eb66a42b-6d5c-11e6-bec9-c27afc834a0c > > > > > > > for framework c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > > > I0828 16:20:36.058661 28509 paths.cpp:528]* Trying to chown* > > > > > > > '/var/lib/mesos-8082/slaves/d6f0e3e2-d144-4275-9d38- > > > > > > > 82327408622b-S8/frameworks/c796100f-9ecb-46fa-90a2- > > > > > > > 72ad649c5dd3-0000/executors/test-crixus.eb66a42b-6d5c- > > > > > > > 11e6-bec9-c27afc834a0c/runs/99620406-87b5-406c-a88b- > > 13adb145c12d' > > > > > > > to user 'root' > > > > > > > I0828 16:20:36.067807 28509 slave.cpp:5352]* Launching executor > > > > > > > test-crixus*.eb66a42b-6d5c-11e6-bec9-c27afc834a0c > > > > > > > of framework c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 with > > > > resources > > > > > > > cpus(*):0.1; mem(*):32 in work directory > > > > > > > '/var/lib/mesos-8082/slaves/d6f0e3e2-d144-4275-9d38- > > > > > > > 82327408622b-S8/frameworks/c796100f-9ecb-46fa-90a2- > > > > > > > 72ad649c5dd3-0000/executors/test-crixus.eb66a42b-6d5c- > > > > > > > 11e6-bec9-c27afc834a0c/runs/99620406-87b5-406c-a88b- > > 13adb145c12d' > > > > > > > I0828 16:20:36.069314 28509 slave.cpp:1698] *Queuing task > > > > > > > 'test-crixus.*eb66a42b-6d5c-11e6-bec9-c27afc834a0c' > > > > > > > for executor 'test-crixus.eb66a42b-6d5c- > 11e6-bec9-c27afc834a0c' > > of > > > > > > > framework c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000 > > > > > > > I0828 16:20:36.069902 28509 containerizer.cpp:666] *Starting > > > > container* > > > > > > > '99620406-87b5-406c-a88b-13adb145c12d' for executor > > > > > > > 'test-crixus.eb66a42b-6d5c-11e6-bec9-c27afc834a0c' of > framework > > > > > > > 'c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000' > > > > > > > I0828 16:20:36.080713 28509 linux_launcher.cpp:304] *Cloning > > child > > > > > > process* > > > > > > > with flags = > > > > > > > I0828 16:20:36.084738 28509 containerizer.cpp:1179] > > *Checkpointing > > > > > > > executor's forked pid 29629* to > > > > > > > '/var/lib/mesos-8082/meta/slaves/d6f0e3e2-d144-4275- > > > > > > 9d38-82327408622b-S8/ > > > > > > > frameworks/c796100f-9ecb-46fa-90a2-72ad649c5dd3-0000/ > > > > > > > executors/test-crixus.eb66a42b-6d5c-11e6-bec9- > > > > > > c27afc834a0c/runs/99620406- > > > > > > > 87b5-406c-a88b-13adb145c12d/pids/forked.pid' > > > > > > > > > > > > > > > > > > > > > But after that, the job is getting restarted and a new > container > > is > > > > > > created > > > > > > > with a new process id. It happening infinitely which is keeping > > the > > > > job > > > > > > in > > > > > > > stagged state to mesos-master. > > > > > > > > > > > > > > This job is nothing but a simle echo "hello world" kind of > shell > > > > > command. > > > > > > > Can anyone please point out where its failing or I am doing > > wrong. > > > > > > > > > > > > > > > > > > > > > > > > > > > > Thanks > > > > > > > Pankaj > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Best Regards, > > > > > > Haosdent Huang > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > Best Regards, > > > > Haosdent Huang > > > > > > > > > > > > > -- > Best Regards, > Haosdent Huang >
