[
https://issues.apache.org/jira/browse/AURORA-1155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338659#comment-14338659
]
Stephan Erb edited comment on AURORA-1155 at 2/26/15 5:08 PM:
--------------------------------------------------------------
Even with the latest git snapshot (including the related GC changes in
https://reviews.apache.org/r/30749/) the GC executor is still broken:
{code}
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0226 14:36:47.105795 25505 fetcher.cpp:76] Fetching URI
'/opt/thermos/bin/gc_executor.pex'
I0226 14:36:47.106696 25505 fetcher.cpp:179] Copying resource from
'/opt/thermos/bin/gc_executor.pex' to
'/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315'
twitter.common.app debug: Initializing: twitter.common.log (Logging subsystem.)
Writing log files to disk in executor_logs
I0226 14:36:47.898592 25518 exec.cpp:132] Version: 0.21.1
I0226 14:36:47.908756 25531 exec.cpp:206] Executor registered on slave
20150218-102226-1985259712-5050-12423-S6
Traceback (most recent call last):
File
"/.pex/install/twitter.common.exceptions-0.3.3-py2-none-any.whl.9bce1b9fc9e64880a8d0b5233b6f394bc9496bdb/twitter.common.exceptions-0.3.3-py2-none-any.whl/twitter/common/exceptions/__init__.py",
line 126, in _excepting_run
self.__real_run(*args, **kw)
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 497, in run
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 474, in _run_gc
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 435, in garbage_collect
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 408, in _gc
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/thermos/monitoring/garbage.py",
line 88, in erase_logs
AttributeError: 'TaskGarbageCollector' object has no attribute '_root'
{code}
was (Author: stephanerb):
Even with the latest git snapshot (including the related GC changes from
[~wickman] https://reviews.apache.org/r/30749/) the GC executor is still
broken:
{code}
WARNING: Logging before InitGoogleLogging() is written to STDERR
I0226 14:36:47.105795 25505 fetcher.cpp:76] Fetching URI
'/opt/thermos/bin/gc_executor.pex'
I0226 14:36:47.106696 25505 fetcher.cpp:179] Copying resource from
'/opt/thermos/bin/gc_executor.pex' to
'/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315'
twitter.common.app debug: Initializing: twitter.common.log (Logging subsystem.)
Writing log files to disk in executor_logs
I0226 14:36:47.898592 25518 exec.cpp:132] Version: 0.21.1
I0226 14:36:47.908756 25531 exec.cpp:206] Executor registered on slave
20150218-102226-1985259712-5050-12423-S6
Traceback (most recent call last):
File
"/.pex/install/twitter.common.exceptions-0.3.3-py2-none-any.whl.9bce1b9fc9e64880a8d0b5233b6f394bc9496bdb/twitter.common.exceptions-0.3.3-py2-none-any.whl/twitter/common/exceptions/__init__.py",
line 126, in _excepting_run
self.__real_run(*args, **kw)
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 497, in run
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 474, in _run_gc
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 435, in garbage_collect
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/aurora/executor/gc_executor.py",
line 408, in _gc
File
"/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/e2a9f631-0c14-4e78-8f56-127ef2104315/gc_executor.pex/apache/thermos/monitoring/garbage.py",
line 88, in erase_logs
AttributeError: 'TaskGarbageCollector' object has no attribute '_root'
{code}
> GC executor broken
> ------------------
>
> Key: AURORA-1155
> URL: https://issues.apache.org/jira/browse/AURORA-1155
> Project: Aurora
> Issue Type: Bug
> Reporter: Stephan Erb
> Priority: Critical
>
> Last week, we deployed git revision e0e3f2e
> (https://github.com/apache/incubator-aurora/tree/e0e3f2e) onto our test
> cluster.
> For this revision, all GC executor runs are failing:
> {code}
> WARNING: Logging before InitGoogleLogging() is written to STDERR
> I0224 15:00:23.165495 26402 fetcher.cpp:76] Fetching URI
> '/opt/thermos/bin/gc_executor.pex'
> I0224 15:00:23.165679 26402 fetcher.cpp:179] Copying resource from
> '/opt/thermos/bin/gc_executor.pex' to
> '/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc'
> Traceback (most recent call last):
> File
> "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/_pex/pex.py",
> line 272, in execute
> self.execute_entry(entry_point, args)
> File
> "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/_pex/pex.py",
> line 320, in execute_entry
> runner(entry_point)
> File
> "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/_pex/pex.py",
> line 342, in execute_pkg_resources
> runner = entry.load(require=False) # trust that the environment is sane
> File
> "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/.bootstrap/pkg_resources.py",
> line 2048, in load
> entry = __import__(self.module_name, globals(),globals(), ['__name__'])
> File
> "/var/lib/mesos/slaves/20150218-102226-1985259712-5050-12423-S6/frameworks/20140919-174559-16842879-5050-27194-0000/executors/aurora.gc/runs/3f1813a3-76f7-4716-bd75-261d6187a2fc/gc_executor.pex/apache/aurora/executor/bin/gc_executor_main.py",
> line 27, in <module>
> ImportError: No module named executor_detector
> {code}
> For me this is an indication that the test infrastructure is missing an
> integration or end to end tests which thoroughly tests the GC executor.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
