Hi guys, So as Sumedha told me, the error that has come is a OOM perm gen error. And we suspect it's just because they've installed many features there in the IoT server, so lots of classes being loaded. So after increasing the perm gen size, Sumedha mentioned that the issue hasn't come yet.
Cheers, Anjana. On Dec 16, 2015 8:02 PM, "Niranda Perera" <nira...@wso2.com> wrote: > Hi Gihan, > > The memory can be set by using the conf parameters ie. " > spark.executor.memory" > > rgds > > On Wed, Dec 16, 2015 at 7:01 PM, Gihan Anuruddha <gi...@wso2.com> wrote: > >> Hi Niranda, >> >> So let say we have to run embedded DAS in a memory restricted >> environment. So where I can define the spark allocated memory configuration >> information? >> >> Regards, >> Gihan >> >> On Wed, Dec 16, 2015 at 6:55 PM, Niranda Perera <nira...@wso2.com> wrote: >> >>> Hi Sumedha, >>> >>> I checked the heapdump you provided, and the size of it is around 230MB. >>> I presume this was not a OOM scenario. >>> >>> As per the Spark memory usage, when you use spark in the local mode, the >>> processing will happen inside that JVM itself. So, we have to make sure >>> that we allocate enough memory for that >>> >>> Rgds >>> >>> On Wed, Dec 16, 2015 at 6:11 PM, Anjana Fernando <anj...@wso2.com> >>> wrote: >>> >>>> Hi Ayoma, >>>> >>>> Thanks for checking up on it, actually "getAllIndexedTables" doesn't >>>> return the Set here, it returns an array that was previously populated in >>>> the refresh operation, so no need to synchronize that method. >>>> >>>> Cheers, >>>> Anjana. >>>> >>>> On Wed, Dec 16, 2015 at 5:44 PM, Ayoma Wijethunga <ay...@wso2.com> >>>> wrote: >>>> >>>>> And, missed mentioning that when this this race condition / state >>>>> corruption happens all "get" operations performed on Set/Map get blocked >>>>> resulting in OOM situation. [1 >>>>> <http://mailinator.blogspot.gr/2009/06/beautiful-race-condition.html>] >>>>> has all that explained nicely. I have checked a heap dump in a similar >>>>> situation and if you take one, you will clearly see many threads waiting >>>>> to >>>>> access this Set instance. >>>>> >>>>> [1] >>>>> http://mailinator.blogspot.gr/2009/06/beautiful-race-condition.html >>>>> >>>>> On Wed, Dec 16, 2015 at 5:37 PM, Ayoma Wijethunga <ay...@wso2.com> >>>>> wrote: >>>>> >>>>>> Hi Anjana, >>>>>> >>>>>> Sorry, I didn't notice that you have already replied this thread. >>>>>> >>>>>> However, please consider my point on "getAllIndexedTables" as well. >>>>>> >>>>>> Thank you, >>>>>> Ayoma. >>>>>> >>>>>> On Wed, Dec 16, 2015 at 5:12 PM, Anjana Fernando <anj...@wso2.com> >>>>>> wrote: >>>>>> >>>>>>> Hi Sumedha, >>>>>>> >>>>>>> Thank you for reporting the issue. I've fixed the concurrent >>>>>>> modification exception issue, where, actually both the methods >>>>>>> "addIndexedTable" and "removeIndexedTable" needed to be synchronized, >>>>>>> since >>>>>>> they both work on the shared Set object there. >>>>>>> >>>>>>> As for the OOM issue, can you please share a heap dump when the OOM >>>>>>> happened. So we can see what is causing this. And also, I see there are >>>>>>> multiple scripts running at the same time, so this actually can be a >>>>>>> legitimate error also, where the server actually doesn't have enough >>>>>>> memory >>>>>>> to continue its operations. @Niranda, please share if there is any info >>>>>>> on >>>>>>> tuning Spark's memory requirements. >>>>>>> >>>>>>> Cheers, >>>>>>> Anjana. >>>>>>> >>>>>>> On Wed, Dec 16, 2015 at 3:32 PM, Sumedha Rubasinghe < >>>>>>> sume...@wso2.com> wrote: >>>>>>> >>>>>>>> We have DAS Lite included in IoT Server and several summarisation >>>>>>>> scripts deployed. Server is going OOM frequently with following >>>>>>>> exception. >>>>>>>> >>>>>>>> Shouldn't this[1] method be synchronised? >>>>>>>> >>>>>>>> [1] >>>>>>>> https://github.com/wso2/carbon-analytics/blob/master/components/analytics-core/org.wso2.carbon.analytics.dataservice.core/src/main/java/org/wso2/carbon/analytics/dataservice/core/indexing/AnalyticsIndexedTableStore.java#L45 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>>>>> >>>>>>>> [2015-12-16 15:11:00,004] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Light_Sensor_Script for tenant id: -1234 >>>>>>>> [2015-12-16 15:11:00,005] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Magnetic_Sensor_Script for tenant id: -1234 >>>>>>>> [2015-12-16 15:11:00,005] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Pressure_Sensor_Script for tenant id: -1234 >>>>>>>> [2015-12-16 15:11:00,006] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Proximity_Sensor_Script for tenant id: -1234 >>>>>>>> [2015-12-16 15:11:00,006] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Rotation_Sensor_Script for tenant id: -1234 >>>>>>>> [2015-12-16 15:11:00,007] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Temperature_Sensor_Script for tenant id: -1234 >>>>>>>> [2015-12-16 15:11:01,132] ERROR >>>>>>>> {org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter} - Error in >>>>>>>> executing task: null >>>>>>>> java.util.ConcurrentModificationException >>>>>>>> at java.util.HashMap$HashIterator.nextEntry(HashMap.java:922) >>>>>>>> at java.util.HashMap$KeyIterator.next(HashMap.java:956) >>>>>>>> at java.util.AbstractCollection.toArray(AbstractCollection.java:195) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsIndexedTableStore.refreshIndexedTableArray(AnalyticsIndexedTableStore.java:46) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.dataservice.core.indexing.AnalyticsIndexedTableStore.addIndexedTable(AnalyticsIndexedTableStore.java:37) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.refreshIndexedTableStoreEntry(AnalyticsDataServiceImpl.java:512) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.invalidateAnalyticsTableInfo(AnalyticsDataServiceImpl.java:525) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.checkAndInvalidateTableInfo(AnalyticsDataServiceImpl.java:504) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.dataservice.core.AnalyticsDataServiceImpl.setTableSchema(AnalyticsDataServiceImpl.java:495) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.spark.core.sources.AnalyticsRelation.insert(AnalyticsRelation.java:162) >>>>>>>> at >>>>>>>> org.apache.spark.sql.sources.InsertIntoDataSource.run(commands.scala:53) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult$lzycompute(commands.scala:57) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.ExecutedCommand.sideEffectResult(commands.scala:57) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.ExecutedCommand.doExecute(commands.scala:68) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.SparkPlan$$anonfun$execute$1.apply(SparkPlan.scala:88) >>>>>>>> at >>>>>>>> org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147) >>>>>>>> at >>>>>>>> org.apache.spark.sql.execution.SparkPlan.execute(SparkPlan.scala:87) >>>>>>>> at >>>>>>>> org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:950) >>>>>>>> at >>>>>>>> org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:950) >>>>>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:144) >>>>>>>> at org.apache.spark.sql.DataFrame.<init>(DataFrame.scala:128) >>>>>>>> at org.apache.spark.sql.DataFrame$.apply(DataFrame.scala:51) >>>>>>>> at org.apache.spark.sql.SQLContext.sql(SQLContext.scala:755) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQueryLocal(SparkAnalyticsExecutor.java:710) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.spark.core.internal.SparkAnalyticsExecutor.executeQuery(SparkAnalyticsExecutor.java:692) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeQuery(CarbonAnalyticsProcessorService.java:199) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.spark.core.CarbonAnalyticsProcessorService.executeScript(CarbonAnalyticsProcessorService.java:149) >>>>>>>> at >>>>>>>> org.wso2.carbon.analytics.spark.core.AnalyticsTask.execute(AnalyticsTask.java:57) >>>>>>>> at >>>>>>>> org.wso2.carbon.ntask.core.impl.TaskQuartzJobAdapter.execute(TaskQuartzJobAdapter.java:67) >>>>>>>> at org.quartz.core.JobRunShell.run(JobRunShell.java:213) >>>>>>>> at >>>>>>>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) >>>>>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>>>>> at >>>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>>>>> at java.lang.Thread.run(Thread.java:745) >>>>>>>> [2015-12-16 15:12:00,001] INFO >>>>>>>> {org.wso2.carbon.analytics.spark.core.AnalyticsTask} - Executing the >>>>>>>> schedule task for: Accelerometer_Sensor_Script for tenant id: -1234 >>>>>>>> >>>>>>>> -- >>>>>>>> /sumedha >>>>>>>> m: +94 773017743 >>>>>>>> b : bit.ly/sumedha >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Anjana Fernando* >>>>>>> Senior Technical Lead >>>>>>> WSO2 Inc. | http://wso2.com >>>>>>> lean . enterprise . middleware >>>>>>> >>>>>>> _______________________________________________ >>>>>>> Dev mailing list >>>>>>> Dev@wso2.org >>>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Ayoma Wijethunga >>>>>> Software Engineer >>>>>> WSO2, Inc.; http://wso2.com >>>>>> lean.enterprise.middleware >>>>>> >>>>>> Mobile : +94 (0) 719428123 <+94+(0)+719428123> >>>>>> Blog : http://www.ayomaonline.com >>>>>> LinkedIn: https://www.linkedin.com/in/ayoma >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Ayoma Wijethunga >>>>> Software Engineer >>>>> WSO2, Inc.; http://wso2.com >>>>> lean.enterprise.middleware >>>>> >>>>> Mobile : +94 (0) 719428123 <+94+(0)+719428123> >>>>> Blog : http://www.ayomaonline.com >>>>> LinkedIn: https://www.linkedin.com/in/ayoma >>>>> >>>> >>>> >>>> >>>> -- >>>> *Anjana Fernando* >>>> Senior Technical Lead >>>> WSO2 Inc. | http://wso2.com >>>> lean . enterprise . middleware >>>> >>> >>> >>> >>> -- >>> *Niranda Perera* >>> Software Engineer, WSO2 Inc. >>> Mobile: +94-71-554-8430 >>> Twitter: @n1r44 <https://twitter.com/N1R44> >>> https://pythagoreanscript.wordpress.com/ >>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >>> >> >> >> -- >> W.G. Gihan Anuruddha >> Senior Software Engineer | WSO2, Inc. >> M: +94772272595 >> > > > > -- > *Niranda Perera* > Software Engineer, WSO2 Inc. > Mobile: +94-71-554-8430 > Twitter: @n1r44 <https://twitter.com/N1R44> > https://pythagoreanscript.wordpress.com/ >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
