The suggested info would be needed to triage this. Thanks, Vinod
> On Dec 24, 2019, at 11:32 PM, Harjinder Singh Mistry > <harjindersing...@flipkart.com> wrote: > > > We have been encountering an intermittent issue where Chronos stops getting > resource offers from Mesos master and the scheduled jobs get stuck in 'Queued' > state at Chronos. > > The sequence of observed events is as follows: > 1. Chronos jobs are not executed by Mesos and status of jobs on Chronos > dashboard is ‘Queued’. > 2. Mesos master dashboard no longer shows agents i.e. slaves. > 3. Mesos master logs show that master has not been sending resource offers to > framework i.e. Chronos. But master keeps getting update from slaves for old > tasks. > 4. Zookeeper and slaves are not down. They are working fine. > 5. After restarting Zookeeper, the system starts working fine. Chronos jobs > start getting executed. > > Please suggest a solution if this problem is known. > > Can you please help us with the steps/info required for investigation ? We > plan > to collect following when the issue happens next time: > > 1. Logs from Chronos, Mesos Master, Mesos Slaves and Zookeeper nodes. > 2. Check Mesos UI: http://mesos-master:5050 and see if any agents are listed > and note status of jobs. > 3. Hit the endpoint http://mesos-master:5050/state and save its output. > 4. Check if Mesos masters and Zookeeper nodes are reachable (i.e. ping) from > Mesos slaves. > 5. From output of step 3, determine the leader in Mesos master and check if > is > sending offers: tail -f /var/log/mesos-log/mesos-master.INFO | grep -i > sending > > Thanks, > Harjinder > > >> ----------------------------------------------------------------------------------------- >> This email and any files transmitted with it are confidential and intended >> solely for the use of the individual or entity to whom they are addressed. >> If you have received this email in error, please notify the system manager. >> This message contains confidential information and is intended only for the >> individual named. If you are not the named addressee, you should not >> disseminate, distribute or copy this email. Please notify the sender >> immediately by email if you have received this email by mistake and delete >> this email from your system. If you are not the intended recipient, you are >> notified that disclosing, copying, distributing or taking any action in >> reliance on the contents of this information is strictly prohibited. >> >> Any views or opinions presented in this email are solely those of the author >> and do not necessarily represent those of the organization. Any information >> on shares, debentures or similar instruments, recommended product pricing, >> valuations and the like are for information purposes only. It is not meant >> to be an instruction or recommendation, as the case may be, to buy or to >> sell securities, products, services nor an offer to buy or sell securities, >> products or services unless specifically stated to be so on behalf of the >> Flipkart group. Employees of the Flipkart group of companies are expressly >> required not to make defamatory statements and not to infringe or authorise >> any infringement of copyright or any other legal right by email >> communications. Any such communication is contrary to organizational policy >> and outside the scope of the employment of the individual concerned. The >> organization will not accept any liability in respect of such communication, >> and the employee responsible will be personally liable for any damages or >> other liability arising. >> >> Our organization accepts no liability for the content of this email, or for >> the consequences of any actions taken on the basis of the information >> provided, unless that information is subsequently confirmed in writing. If >> you are not the intended recipient, you are notified that disclosing, >> copying, distributing or taking any action in reliance on the contents of >> this information is strictly prohibited. >> -----------------------------------------------------------------------------------------
