Here is the physical plan.
Also attaching the executor log from one of the executors. You can see that
memory consumption is slowly rising and then it is reaching around 10.5 GB.
There it is staying for around 5 minutes 06-50-36 to 06-55-00. Then this
executor is getting killed. ExecutorMemory configured is 10GB.
Regards,
Sourav
---------------
plan
------------------------------------------------------------------------------------------------------------------------------------------
== Parsed Logical Plan ==
'Project ['IKB_PROJECT_LIVE_DT,'FLOORPLAN_NM,'FLOORPLAN_DBKEY]
'Filter ('IKB_PROJECT_TYPE = CR)
'Join Inner, None
'UnresolvedRelation [IKB_FP_POG_PRE_EXT], Some(P)
'UnresolvedRelation [IKB_PROJECT_CALENDAR_EXT], Some(C)
== Analyzed Logical Plan ==
Project [IKB_PROJECT_LIVE_DT#31,FLOORPLAN_NM#20,FLOORPLAN_DBKEY#17]
Filter (IKB_PROJECT_TYPE#29 = CR)
Join Inner, None
MetastoreRelation sourav_ikb_hs, ikb_fp_pog_pre_ext, Some(P)
MetastoreRelation sourav_ikb_hs, ikb_project_calendar_ext, Some(C)
== Optimized Logical Plan ==
Project [IKB_PROJECT_LIVE_DT#31,FLOORPLAN_NM#20,FLOORPLAN_DBKEY#17]
Join Inner, None
Project [FLOORPLAN_NM#20,FLOORPLAN_DBKEY#17]
MetastoreRelation sourav_ikb_hs, ikb_fp_pog_pre_ext, Some(P)
Project [IKB_PROJECT_LIVE_DT#31]
Filter (IKB_PROJECT_TYPE#29 = CR)
MetastoreRelation sourav_ikb_hs, ikb_project_calendar_ext, Some(C)
== Physical Plan ==
Project [IKB_PROJECT_LIVE_DT#31,FLOORPLAN_NM#20,FLOORPLAN_DBKEY#17]
CartesianProduct
HiveTableScan [FLOORPLAN_NM#20,FLOORPLAN_DBKEY#17], (MetastoreRelation
sourav_ikb_hs, ikb_fp_pog_pre_ext, Some(P)), None
Project [IKB_PROJECT_LIVE_DT#31]
Filter (IKB_PROJECT_TYPE#29 = CR)
HiveTableScan [IKB_PROJECT_LIVE_DT#31,IKB_PROJECT_TYPE#29],
(MetastoreRelation sourav_ikb_hs, ikb_project_calendar_ext, Some(C)), None
Code Generation: false
== RDD ==
-------
On Wed, Jun 10, 2015 at 12:59 AM, Cheng Lian <lian.cs....@gmail.com> wrote:
> Would you mind to provide executor output so that we can check the
> reason why executors died?
>
> And you may run EXPLAIN EXTENDED to find out the physical plan of your
> query, something like:
>
> 0: jdbc:hive2://localhost:10000> explain extended select * from foo;
> +-------------------------------------------------------------------------+
> | plan |
> +-------------------------------------------------------------------------+
> | == Parsed Logical Plan == |
> | 'Project [*] |
> | 'UnresolvedRelation [foo], None |
> | |
> | == Analyzed Logical Plan == |
> | i: string |
> | Project [i#6] |
> | Subquery foo |
> | Relation[i#6] org.apache.spark.sql.parquet.ParquetRelation2@517574b8 |
> | |
> | == Optimized Logical Plan == |
> | Relation[i#6] org.apache.spark.sql.parquet.ParquetRelation2@517574b8 |
> | |
> | == Physical Plan == |
> | PhysicalRDD [i#6], MapPartitionsRDD[2] at |
> | |
> | Code Generation: false |
> | == RDD == |
> +-------------------------------------------------------------------------+
>
> On 6/10/15 1:28 PM, Sourav Mazumder wrote:
>
> From log file I noticed that the ExecutorLostFailure happens after the
> memory used by Executor becomes more than the Executor memory value.
> However, even if I increase the value of Executor Memory the Executor fails
> - only that it takes longer time.
>
> I'm wondering that for joining 2 Hive tables, one with 100 MB data
> (around 1 M rows) and another with 20 KB data (around 100 rows) why an
> executor is consuming so much of memory. Even if I increase the memory to
> 20 GB. The same failure happens.
>
> Regards,
> Sourav
>
> On Tue, Jun 9, 2015 at 12:58 PM, Sourav Mazumder <
> sourav.mazumde...@gmail.com> wrote:
>
>> Hi,
>>
>> I'm just doing a select statement which is supposed to return 10 MB data
>> maximum. The driver memory is 2G and executor memory is 20 G.
>>
>> The query I'm trying to run is something like
>>
>> SELECT PROJECT_LIVE_DT, FLOORPLAN_NM, FLOORPLAN_DB_KEY
>> FROM POG_PRE_EXT P, PROJECT_CALENDAR_EXT C
>> WHERE PROJECT_TYPE = 'CR'
>>
>> Not sure what exactly you mean by physical plan.
>>
>> Here is he stack trace from the machine where the thrift process is
>> running.
>>
>> Regards,
>> Sourav
>>
>> On Mon, Jun 8, 2015 at 11:18 PM, Cheng, Hao <hao.ch...@intel.com> wrote:
>>
>>> Is it the large result set return from the Thrift Server? And can you
>>> paste the SQL and physical plan?
>>>
>>>
>>>
>>> *From:* Ted Yu [mailto:yuzhih...@gmail.com]
>>> *Sent:* Tuesday, June 9, 2015 12:01 PM
>>> *To:* Sourav Mazumder
>>> *Cc:* user
>>> *Subject:* Re: Spark SQL with Thrift Server is very very slow and
>>> finally failing
>>>
>>>
>>>
>>> Which Spark release are you using ?
>>>
>>>
>>>
>>> Can you pastebin the stack trace w.r.t. ExecutorLostFailure ?
>>>
>>>
>>>
>>> Thanks
>>>
>>>
>>>
>>> On Mon, Jun 8, 2015 at 8:52 PM, Sourav Mazumder <
>>> sourav.mazumde...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am trying to run a SQL form a JDBC driver using Spark's Thrift Server.
>>>
>>> I'm doing a join between a Hive Table of size around 100 GB and another
>>> Hive Table with 10 KB, with a filter on a particular column
>>>
>>> The query takes more than 45 minutes and then I get ExecutorLostFailure.
>>> That is because of memory as once I increase the memory the failure happens
>>> but after a long time.
>>>
>>> I'm having executor memory 20 GB, Spark DRiver Memory 2 GB, Executor
>>> Instances 2 and Executor Core 2.
>>>
>>> Running the job using Yarn with master as 'yarn-client'.
>>>
>>> Any idea if I'm missing any other configuration ?
>>>
>>> Regards,
>>>
>>> Sourav
>>>
>>>
>>>
>>
>>
>
>
2015-06-10 06:50:00,258 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 5.0 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:03,309 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 5.0 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:06,365 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 5.9 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:09,415 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 5.9 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:12,525 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 6.9 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:15,580 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 6.9 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:18,630 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 6.9 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:21,687 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 7.8 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:24,737 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 7.9 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:27,788 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 8.0 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:30,843 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 8.7 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:33,900 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.2 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:36,970 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:40,031 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:43,094 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:46,164 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:49,214 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:52,283 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:55,343 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:50:58,406 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:01,475 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:04,525 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:07,587 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:10,643 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:13,706 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:16,787 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:19,852 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:22,913 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:25,982 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:29,044 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:32,124 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:35,172 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:38,234 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:41,303 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:44,353 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:47,429 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:50,499 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:53,566 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:56,641 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:51:59,709 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:02,769 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:05,831 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:08,895 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:11,965 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:15,032 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:18,095 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:21,162 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:24,225 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:27,293 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:30,357 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:33,418 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:36,484 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:39,545 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:42,611 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:45,670 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:48,748 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:51,815 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:54,873 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:52:57,951 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:01,010 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:04,070 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:07,145 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:10,205 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:13,266 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:13,266 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:16,334 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:19,399 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:22,453 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:25,514 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:28,578 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:31,649 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:34,712 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:37,784 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:40,847 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:43,910 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:47,007 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:50,070 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:53,140 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:56,202 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:53:59,265 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:02,335 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:05,395 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:08,462 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:11,523 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:14,584 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:17,651 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
015-06-10 06:54:20,711 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:23,781 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:26,866 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:29,928 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:33,002 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:36,068 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:39,128 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:42,199 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:45,262 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:48,330 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:51,396 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:54,458 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:54:57,525 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:55:00,588 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(408)) - Memory usage of ProcessTree 5366 for
container-id container_1433878270250_0029_02_000004: 10.5 GB of 19 GB physical
memory used; 11.3 GB of 95 GB virtual memory used
2015-06-10 06:55:01,139 INFO container.Container
(ContainerImpl.java:handle(999)) - Container
container_1433878270250_0029_02_000004 transitioned from RUNNING to KILLING
2015-06-10 06:55:01,139 INFO application.Application
(ApplicationImpl.java:handle(464)) - Application application_1433878270250_0029
transitioned from RUNNING to FINISHING_CONTAINERS_WAIT
2015-06-10 06:55:01,139 INFO launcher.ContainerLaunch
(ContainerLaunch.java:cleanupContainer(370)) - Cleaning up container
container_1433878270250_0029_02_000004
2015-06-10 06:55:01,148 WARN nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:launchContainer(223)) - Exit code from container
container_1433878270250_0029_02_000004 is : 143
2015-06-10 06:55:01,163 INFO container.Container
(ContainerImpl.java:handle(999)) - Container
container_1433878270250_0029_02_000004 transitioned from KILLING to
CONTAINER_CLEANEDUP_AFTER_KILL
2015-06-10 06:55:01,163 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/srv/hadoop/yarn/local/usercache/spark/appcache/application_1433878270250_0029/container_1433878270250_0029_02_000004
2015-06-10 06:55:01,163 INFO nodemanager.NMAuditLogger
(NMAuditLogger.java:logSuccess(89)) - USER=spark OPERATION=Container
Finished - Killed TARGET=ContainerImpl RESULT=SUCCESS
APPID=application_1433878270250_0029
CONTAINERID=container_1433878270250_0029_02_000004
2015-06-10 06:55:01,164 INFO container.Container
(ContainerImpl.java:handle(999)) - Container
container_1433878270250_0029_02_000004 transitioned from
CONTAINER_CLEANEDUP_AFTER_KILL to DONE
2015-06-10 06:55:01,164 INFO application.Application
(ApplicationImpl.java:transition(402)) - Removing
container_1433878270250_0029_02_000004 from application
application_1433878270250_0029
2015-06-10 06:55:01,164 INFO application.Application
(ApplicationImpl.java:handle(464)) - Application application_1433878270250_0029
transitioned from FINISHING_CONTAINERS_WAIT to APPLICATION_RESOURCES_CLEANINGUP
2015-06-10 06:55:01,164 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:startContainerLogAggregation(488)) - Considering
container container_1433878270250_0029_02_000004 for log-aggregation
2015-06-10 06:55:01,164 INFO containermanager.AuxServices
(AuxServices.java:handle(196)) - Got event CONTAINER_STOP for appId
application_1433878270250_0029
2015-06-10 06:55:01,164 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(457)) - Deleting absolute path :
/srv/hadoop/yarn/local/usercache/spark/appcache/application_1433878270250_0029
2015-06-10 06:55:01,164 INFO containermanager.AuxServices
(AuxServices.java:handle(196)) - Got event APPLICATION_STOP for appId
application_1433878270250_0029
2015-06-10 06:55:01,165 INFO application.Application
(ApplicationImpl.java:handle(464)) - Application application_1433878270250_0029
transitioned from APPLICATION_RESOURCES_CLEANINGUP to FINISHED
2015-06-10 06:55:01,165 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:finishLogAggregation(496)) - Application just
finished : application_1433878270250_0029
2015-06-10 06:55:01,177 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:doContainerLogAggregation(525)) - Uploading logs for
container container_1433878270250_0029_02_000004. Current good log dirs are
/srv/hadoop/yarn/log
2015-06-10 06:55:01,246 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:doContainerLogAggregation(525)) - Uploading logs for
container container_1433878270250_0029_01_000005. Current good log dirs are
/srv/hadoop/yarn/log
2015-06-10 06:55:01,246 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_02_000004/stdout
2015-06-10 06:55:01,247 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_02_000004/stderr
2015-06-10 06:55:01,316 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_01_000005/stdout
2015-06-10 06:55:01,317 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_01_000005/stderr
2015-06-10 06:55:01,318 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:doContainerLogAggregation(525)) - Uploading logs for
container container_1433878270250_0029_02_000003. Current good log dirs are
/srv/hadoop/yarn/log
2015-06-10 06:55:01,387 INFO logaggregation.AppLogAggregatorImpl
(AppLogAggregatorImpl.java:doContainerLogAggregation(525)) - Uploading logs for
container container_1433878270250_0029_01_000003. Current good log dirs are
/srv/hadoop/yarn/log
2015-06-10 06:55:01,387 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_02_000003/stderr
2015-06-10 06:55:01,389 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_02_000003/stdout
2015-06-10 06:55:01,440 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_01_000003/stdout
2015-06-10 06:55:01,440 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029/container_1433878270250_0029_01_000003/stderr
2015-06-10 06:55:01,478 INFO nodemanager.DefaultContainerExecutor
(DefaultContainerExecutor.java:deleteAsUser(466)) - Deleting path :
/srv/hadoop/yarn/log/application_1433878270250_0029
2015-06-10 06:55:03,588 INFO monitor.ContainersMonitorImpl
(ContainersMonitorImpl.java:run(356)) - Stopping resource-monitoring for
container_1433878270250_0029_02_000004
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
