[jira] [Commented] (GRIFFIN-71) Failure to submit multiple timing tasks with Livy
[ https://issues.apache.org/jira/browse/GRIFFIN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16283115#comment-16283115 ] Yang commented on GRIFFIN-71: - We don't submit many tasks in a single batch at the moment, but we submitted at different time points. Now we can all run successfully, but this is just a temporary processing method. Next we need to study the operation of the thread pool or the parameters under debugging. > Failure to submit multiple timing tasks with Livy > - > > Key: GRIFFIN-71 > URL: https://issues.apache.org/jira/browse/GRIFFIN-71 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Yang >Assignee: William Guo > Attachments: hs_err_pid56468.log, spark10err.txt > > > In the client server, using the Livy timer submit 100 monitoring rules, found > that Livy will be submitted to the spark cluster on the part of the > monitoring task and other tasks fail .We view the Livy log, JVM can't start > up, there may be insufficient memory, client log specific memory 32G, please > see the attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GRIFFIN-71) Failure to submit multiple timing tasks with Livy
[ https://issues.apache.org/jira/browse/GRIFFIN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16273853#comment-16273853 ] William Guo commented on GRIFFIN-71: Fixed? > Failure to submit multiple timing tasks with Livy > - > > Key: GRIFFIN-71 > URL: https://issues.apache.org/jira/browse/GRIFFIN-71 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Yang >Assignee: William Guo > Attachments: hs_err_pid56468.log, spark10err.txt > > > In the client server, using the Livy timer submit 100 monitoring rules, found > that Livy will be submitted to the spark cluster on the part of the > monitoring task and other tasks fail .We view the Livy log, JVM can't start > up, there may be insufficient memory, client log specific memory 32G, please > see the attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GRIFFIN-71) Failure to submit multiple timing tasks with Livy
[ https://issues.apache.org/jira/browse/GRIFFIN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270411#comment-16270411 ] William Guo commented on GRIFFIN-71: Use 64 bit Java on a 64 bit OS Decrease Java heap size (-Xmx/-Xms) Decrease number of Java threads Decrease Java thread stack sizes (-Xss) > Failure to submit multiple timing tasks with Livy > - > > Key: GRIFFIN-71 > URL: https://issues.apache.org/jira/browse/GRIFFIN-71 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Yang > Attachments: hs_err_pid56468.log, spark10err.txt > > > In the client server, using the Livy timer submit 100 monitoring rules, found > that Livy will be submitted to the spark cluster on the part of the > monitoring task and other tasks fail .We view the Livy log, JVM can't start > up, there may be insufficient memory, client log specific memory 32G, please > see the attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GRIFFIN-71) Failure to submit multiple timing tasks with Livy
[ https://issues.apache.org/jira/browse/GRIFFIN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270407#comment-16270407 ] William Guo commented on GRIFFIN-71: ok, if livy is down, Have you tried by possible solution mentioned in log? # There is insufficient memory for the Java Runtime Environment to continue. # Cannot create GC thread. Out of system resources. # Possible reasons: # The system is out of physical RAM or swap space # In 32 bit mode, the process size limit was hit # Possible solutions: # Reduce memory load on the system # Increase physical memory or swap space # Check if swap backing store is full # Use 64 bit Java on a 64 bit OS # Decrease Java heap size (-Xmx/-Xms) # Decrease number of Java threads # Decrease Java thread stack sizes (-Xss) # Set larger code cache with -XX:ReservedCodeCacheSize= # This output file may be truncated or incomplete. # # Out of Memory Error (gcTaskThread.cpp:48), pid=56468, tid=0x7f436fb38700 Thanks, William > Failure to submit multiple timing tasks with Livy > - > > Key: GRIFFIN-71 > URL: https://issues.apache.org/jira/browse/GRIFFIN-71 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Yang > Attachments: hs_err_pid56468.log, spark10err.txt > > > In the client server, using the Livy timer submit 100 monitoring rules, found > that Livy will be submitted to the spark cluster on the part of the > monitoring task and other tasks fail .We view the Livy log, JVM can't start > up, there may be insufficient memory, client log specific memory 32G, please > see the attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GRIFFIN-71) Failure to submit multiple timing tasks with Livy
[ https://issues.apache.org/jira/browse/GRIFFIN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270404#comment-16270404 ] Yang commented on GRIFFIN-71: - For example, 100 tasks are scheduled to run at 12 o'clock every day, of which Livy can submit 30 tasks to clusters, and the remaining 70 are not submitted to the cluster. Livy has hung up. > Failure to submit multiple timing tasks with Livy > - > > Key: GRIFFIN-71 > URL: https://issues.apache.org/jira/browse/GRIFFIN-71 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Yang > Attachments: hs_err_pid56468.log, spark10err.txt > > > In the client server, using the Livy timer submit 100 monitoring rules, found > that Livy will be submitted to the spark cluster on the part of the > monitoring task and other tasks fail .We view the Livy log, JVM can't start > up, there may be insufficient memory, client log specific memory 32G, please > see the attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (GRIFFIN-71) Failure to submit multiple timing tasks with Livy
[ https://issues.apache.org/jira/browse/GRIFFIN-71?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16270386#comment-16270386 ] William Guo commented on GRIFFIN-71: hi Yang, What do you mean by "on the part of the monitoring task" ? is Livy down or spark cluster is hanging? William > Failure to submit multiple timing tasks with Livy > - > > Key: GRIFFIN-71 > URL: https://issues.apache.org/jira/browse/GRIFFIN-71 > Project: Griffin (Incubating) > Issue Type: Bug >Reporter: Yang > Attachments: hs_err_pid56468.log, spark10err.txt > > > In the client server, using the Livy timer submit 100 monitoring rules, found > that Livy will be submitted to the spark cluster on the part of the > monitoring task and other tasks fail .We view the Livy log, JVM can't start > up, there may be insufficient memory, client log specific memory 32G, please > see the attachment. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
