Re: Problem with scheduler (stops after ten executions)

[moon soo Lee]

Appreciate for sharing your investigation.
I tried to schedule some paragraphs returning  no value, but couldn't
reproduce the problem. Could you share some code snippets to reproduce the
problem?

Thanks,
moon

On Mon, Nov 14, 2016 at 11:36 AM Florian Schulz  wrote:

> Hi,
>
> sorry for my late reponse! I experimented a lot with this in the last days
> and I think I have fixed this now, but I'm not sure, what exactly the
> problem was. I think it has something to do with functions, which return no
> value (type == Unit). I changed all of them to return anything (e.g. a
> Boolean), so the REPL receives any return value.
> What confuses me a little bit is the fact that also the usage of the %sql
> function / the spark sql context is sometimes responsible for the scheduler
> blocks. I do nothing more than querying some cached temp tables. As soon as
> I change the query a little bit, the problem seems to be away. Jstack helps
> a lot to find these issues.
>
>
> Best regards
> Florian
>

Problem with scheduler (stops after ten executions)

[Florian Schulz]

Hi,

 

sorry for my late reponse! I experimented a lot with this in the last days and I think I have fixed this now, but I'm not sure, what exactly the problem was. I think it has something to do with functions, which return no value (type == Unit). I changed all of them to return anything (e.g. a Boolean), so the REPL receives any return value.

What confuses me a little bit is the fact that also the usage of the %sql function / the spark sql context is sometimes responsible for the scheduler blocks. I do nothing more than querying some cached temp tables. As soon as I change the query a little bit, the problem seems to be away. Jstack helps a lot to find these issues.

 

 

Best regards

Florian

Re: Problem with scheduler (stops after ten executions)

[Florian Schulz]

Hi Prasad,

thanks for your fast response! I use a lot of JDBC-Queries, so your assumption is right. Here is the jstack output regarding the scheduler:

 

"DefaultQuartzScheduler_QuartzSchedulerThread" #32 prio=5 os_prio=31 tid=0x7ffee8cd1000 nid=0x7403 in Object.wait() [0x72502000]

   java.lang.Thread.State: TIMED_WAITING (on object monitor)

at java.lang.Object.wait(Native Method)

at org.quartz.simpl.SimpleThreadPool.blockForAvailableThreads(SimpleThreadPool.java:452)

- locked <0x000780420438> (a java.lang.Object)

at org.quartz.core.QuartzSchedulerThread.run(QuartzSchedulerThread.java:263)

 

"DefaultQuartzScheduler_Worker-10" #31 prio=5 os_prio=31 tid=0x7ffee9dee800 nid=0x7203 waiting on condition [0x723ff000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x000780234e60> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-9" #30 prio=5 os_prio=31 tid=0x7ffee8cce000 nid=0x7003 waiting on condition [0x722fc000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x00078023cbc0> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-8" #29 prio=5 os_prio=31 tid=0x7ffee91e5000 nid=0x6e03 waiting on condition [0x721f9000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x0007802441b8> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-7" #28 prio=5 os_prio=31 tid=0x7ffee91e4000 nid=0x6c03 waiting on condition [0x720f6000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x000780235000> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-6" #27 prio=5 os_prio=31 tid=0x7ffee9dfc800 nid=0x6a03 waiting on condition [0x71ff3000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x0007802351a0> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-5" #26 prio=5 os_prio=31 tid=0x7ffee92f4800 nid=0x6803 waiting on condition [0x71ef]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x000780244358> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-4" #25 prio=5 os_prio=31 tid=0x7ffee92ef000 nid=0x6603 waiting on condition [0x71ded000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x000780235340> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-3" #24 prio=5 os_prio=31 tid=0x7ffee8ce7000 nid=0x6403 waiting on condition [0x71cea000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

- locked <0x0007802444f8> (a java.lang.Object)

 

"DefaultQuartzScheduler_Worker-2" #23 prio=5 os_prio=31 tid=0x7ffee8cee800 nid=0x6203 waiting on condition [0x71be7000]

   java.lang.Thread.State: TIMED_WAITING (sleeping)

at java.lang.Thread.sleep(Native Method)

at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:645)

at org.quartz.core.JobRunShell.run(JobRunShell.java:202)

at 

Problem with scheduler (stops after ten executions)

[Florian Schulz]

Hi, 

 

I have a problem with the scheduler. I have a notebook and I execute the spark code in it every minute (with the cron scheduler). The scheduler starts ten times and then it stops and do nothing anymore (in no notebook at all). I use version 0.6.2 of Apache Zeppelin. Do you have any idea, why this happens? I can reproduce this without a problem, but I can't find anything in the logs. Hope you can help me.

 

 

Best regards

Florian