[jira] [Updated] (MESOS-1653) HealthCheckTest.GracePeriod is flaky.
[
https://issues.apache.org/jira/browse/MESOS-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gastón Kleiman updated MESOS-1653:
--
Shepherd: Alexander Rukletsov
Sprint: Mesosphere Sprint 44
Target Version/s: 1.1.0
> HealthCheckTest.GracePeriod is flaky.
> -
>
> Key: MESOS-1653
> URL: https://issues.apache.org/jira/browse/MESOS-1653
> Project: Mesos
> Issue Type: Bug
> Components: test
>Reporter: Benjamin Mahler
>Assignee: Gastón Kleiman
> Labels: flaky, health-check, mesosphere
>
> {noformat}
> [--] 3 tests from HealthCheckTest
> [ RUN ] HealthCheckTest.GracePeriod
> Using temporary directory '/tmp/HealthCheckTest_GracePeriod_d7zCPr'
> I0729 17:10:10.484951 1176 leveldb.cpp:176] Opened db in 28.883552ms
> I0729 17:10:10.499487 1176 leveldb.cpp:183] Compacted db in 13.674118ms
> I0729 17:10:10.500200 1176 leveldb.cpp:198] Created db iterator in 7394ns
> I0729 17:10:10.500692 1176 leveldb.cpp:204] Seeked to beginning of db in
> 2317ns
> I0729 17:10:10.501113 1176 leveldb.cpp:273] Iterated through 0 keys in the
> db in 1367ns
> I0729 17:10:10.501535 1176 replica.cpp:741] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0729 17:10:10.502233 1212 recover.cpp:425] Starting replica recovery
> I0729 17:10:10.502295 1212 recover.cpp:451] Replica is in EMPTY status
> I0729 17:10:10.502825 1212 replica.cpp:638] Replica in EMPTY status received
> a broadcasted recover request
> I0729 17:10:10.502877 1212 recover.cpp:188] Received a recover response from
> a replica in EMPTY status
> I0729 17:10:10.502980 1212 recover.cpp:542] Updating replica status to
> STARTING
> I0729 17:10:10.508482 1213 master.cpp:289] Master
> 20140729-171010-16842879-54701-1176 (trusty) started on 127.0.1.1:54701
> I0729 17:10:10.508607 1213 master.cpp:326] Master only allowing
> authenticated frameworks to register
> I0729 17:10:10.508632 1213 master.cpp:331] Master only allowing
> authenticated slaves to register
> I0729 17:10:10.508656 1213 credentials.hpp:36] Loading credentials for
> authentication from '/tmp/HealthCheckTest_GracePeriod_d7zCPr/credentials'
> I0729 17:10:10.509407 1213 master.cpp:360] Authorization enabled
> I0729 17:10:10.510030 1207 hierarchical_allocator_process.hpp:301]
> Initializing hierarchical allocator process with master :
> master@127.0.1.1:54701
> I0729 17:10:10.510113 1207 master.cpp:123] No whitelist given. Advertising
> offers for all slaves
> I0729 17:10:10.511699 1213 master.cpp:1129] The newly elected leader is
> master@127.0.1.1:54701 with id 20140729-171010-16842879-54701-1176
> I0729 17:10:10.512230 1213 master.cpp:1142] Elected as the leading master!
> I0729 17:10:10.512692 1213 master.cpp:960] Recovering from registrar
> I0729 17:10:10.513226 1210 registrar.cpp:313] Recovering registrar
> I0729 17:10:10.516006 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 12.946461ms
> I0729 17:10:10.516047 1212 replica.cpp:320] Persisted replica status to
> STARTING
> I0729 17:10:10.516129 1212 recover.cpp:451] Replica is in STARTING status
> I0729 17:10:10.516520 1212 replica.cpp:638] Replica in STARTING status
> received a broadcasted recover request
> I0729 17:10:10.516592 1212 recover.cpp:188] Received a recover response from
> a replica in STARTING status
> I0729 17:10:10.516767 1212 recover.cpp:542] Updating replica status to VOTING
> I0729 17:10:10.528376 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 11.537102ms
> I0729 17:10:10.528430 1212 replica.cpp:320] Persisted replica status to
> VOTING
> I0729 17:10:10.528501 1212 recover.cpp:556] Successfully joined the Paxos
> group
> I0729 17:10:10.528565 1212 recover.cpp:440] Recover process terminated
> I0729 17:10:10.528700 1212 log.cpp:656] Attempting to start the writer
> I0729 17:10:10.528960 1212 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0729 17:10:10.537821 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 8.830863ms
> I0729 17:10:10.537869 1212 replica.cpp:342] Persisted promised to 1
> I0729 17:10:10.540550 1209 coordinator.cpp:230] Coordinator attemping to
> fill missing position
> I0729 17:10:10.540856 1209 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0729 17:10:10.547430 1209 leveldb.cpp:343] Persisting action (8 bytes) to
> leveldb took 6.548344ms
> I0729 17:10:10.547471 1209 replica.cpp:676] Persisted action at 0
> I0729 17:10:10.547732 1209 replica.cpp:508] Replica received write request
> for position 0
> I0729 17:10:10.547765 1209 leveldb.cpp:438] Reading position from leveldb
> took 15676ns
> I0729 17:10:10.557169 1209 leveldb.cpp:343] Persisting action (
[jira] [Updated] (MESOS-1653) HealthCheckTest.GracePeriod is flaky.
[
https://issues.apache.org/jira/browse/MESOS-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haosdent updated MESOS-1653:
Assignee: Gastón Kleiman (was: haosdent)
> HealthCheckTest.GracePeriod is flaky.
> -
>
> Key: MESOS-1653
> URL: https://issues.apache.org/jira/browse/MESOS-1653
> Project: Mesos
> Issue Type: Bug
> Components: test
>Reporter: Benjamin Mahler
>Assignee: Gastón Kleiman
> Labels: flaky, health-check, mesosphere
>
> {noformat}
> [--] 3 tests from HealthCheckTest
> [ RUN ] HealthCheckTest.GracePeriod
> Using temporary directory '/tmp/HealthCheckTest_GracePeriod_d7zCPr'
> I0729 17:10:10.484951 1176 leveldb.cpp:176] Opened db in 28.883552ms
> I0729 17:10:10.499487 1176 leveldb.cpp:183] Compacted db in 13.674118ms
> I0729 17:10:10.500200 1176 leveldb.cpp:198] Created db iterator in 7394ns
> I0729 17:10:10.500692 1176 leveldb.cpp:204] Seeked to beginning of db in
> 2317ns
> I0729 17:10:10.501113 1176 leveldb.cpp:273] Iterated through 0 keys in the
> db in 1367ns
> I0729 17:10:10.501535 1176 replica.cpp:741] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0729 17:10:10.502233 1212 recover.cpp:425] Starting replica recovery
> I0729 17:10:10.502295 1212 recover.cpp:451] Replica is in EMPTY status
> I0729 17:10:10.502825 1212 replica.cpp:638] Replica in EMPTY status received
> a broadcasted recover request
> I0729 17:10:10.502877 1212 recover.cpp:188] Received a recover response from
> a replica in EMPTY status
> I0729 17:10:10.502980 1212 recover.cpp:542] Updating replica status to
> STARTING
> I0729 17:10:10.508482 1213 master.cpp:289] Master
> 20140729-171010-16842879-54701-1176 (trusty) started on 127.0.1.1:54701
> I0729 17:10:10.508607 1213 master.cpp:326] Master only allowing
> authenticated frameworks to register
> I0729 17:10:10.508632 1213 master.cpp:331] Master only allowing
> authenticated slaves to register
> I0729 17:10:10.508656 1213 credentials.hpp:36] Loading credentials for
> authentication from '/tmp/HealthCheckTest_GracePeriod_d7zCPr/credentials'
> I0729 17:10:10.509407 1213 master.cpp:360] Authorization enabled
> I0729 17:10:10.510030 1207 hierarchical_allocator_process.hpp:301]
> Initializing hierarchical allocator process with master :
> master@127.0.1.1:54701
> I0729 17:10:10.510113 1207 master.cpp:123] No whitelist given. Advertising
> offers for all slaves
> I0729 17:10:10.511699 1213 master.cpp:1129] The newly elected leader is
> master@127.0.1.1:54701 with id 20140729-171010-16842879-54701-1176
> I0729 17:10:10.512230 1213 master.cpp:1142] Elected as the leading master!
> I0729 17:10:10.512692 1213 master.cpp:960] Recovering from registrar
> I0729 17:10:10.513226 1210 registrar.cpp:313] Recovering registrar
> I0729 17:10:10.516006 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 12.946461ms
> I0729 17:10:10.516047 1212 replica.cpp:320] Persisted replica status to
> STARTING
> I0729 17:10:10.516129 1212 recover.cpp:451] Replica is in STARTING status
> I0729 17:10:10.516520 1212 replica.cpp:638] Replica in STARTING status
> received a broadcasted recover request
> I0729 17:10:10.516592 1212 recover.cpp:188] Received a recover response from
> a replica in STARTING status
> I0729 17:10:10.516767 1212 recover.cpp:542] Updating replica status to VOTING
> I0729 17:10:10.528376 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 11.537102ms
> I0729 17:10:10.528430 1212 replica.cpp:320] Persisted replica status to
> VOTING
> I0729 17:10:10.528501 1212 recover.cpp:556] Successfully joined the Paxos
> group
> I0729 17:10:10.528565 1212 recover.cpp:440] Recover process terminated
> I0729 17:10:10.528700 1212 log.cpp:656] Attempting to start the writer
> I0729 17:10:10.528960 1212 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0729 17:10:10.537821 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 8.830863ms
> I0729 17:10:10.537869 1212 replica.cpp:342] Persisted promised to 1
> I0729 17:10:10.540550 1209 coordinator.cpp:230] Coordinator attemping to
> fill missing position
> I0729 17:10:10.540856 1209 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0729 17:10:10.547430 1209 leveldb.cpp:343] Persisting action (8 bytes) to
> leveldb took 6.548344ms
> I0729 17:10:10.547471 1209 replica.cpp:676] Persisted action at 0
> I0729 17:10:10.547732 1209 replica.cpp:508] Replica received write request
> for position 0
> I0729 17:10:10.547765 1209 leveldb.cpp:438] Reading position from leveldb
> took 15676ns
> I0729 17:10:10.557169 1209 leveldb.cpp:343] Persisting action (14 bytes) to
> leveldb took 9.373798ms
> I0729 17:10:10.557241 1209 replica.c
[jira] [Updated] (MESOS-1653) HealthCheckTest.GracePeriod is flaky.
[
https://issues.apache.org/jira/browse/MESOS-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
haosdent updated MESOS-1653:
Labels: flaky health-check mesosphere (was: flaky mesosphere)
> HealthCheckTest.GracePeriod is flaky.
> -
>
> Key: MESOS-1653
> URL: https://issues.apache.org/jira/browse/MESOS-1653
> Project: Mesos
> Issue Type: Bug
> Components: test
>Reporter: Benjamin Mahler
>Assignee: Timothy Chen
> Labels: flaky, health-check, mesosphere
>
> {noformat}
> [--] 3 tests from HealthCheckTest
> [ RUN ] HealthCheckTest.GracePeriod
> Using temporary directory '/tmp/HealthCheckTest_GracePeriod_d7zCPr'
> I0729 17:10:10.484951 1176 leveldb.cpp:176] Opened db in 28.883552ms
> I0729 17:10:10.499487 1176 leveldb.cpp:183] Compacted db in 13.674118ms
> I0729 17:10:10.500200 1176 leveldb.cpp:198] Created db iterator in 7394ns
> I0729 17:10:10.500692 1176 leveldb.cpp:204] Seeked to beginning of db in
> 2317ns
> I0729 17:10:10.501113 1176 leveldb.cpp:273] Iterated through 0 keys in the
> db in 1367ns
> I0729 17:10:10.501535 1176 replica.cpp:741] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0729 17:10:10.502233 1212 recover.cpp:425] Starting replica recovery
> I0729 17:10:10.502295 1212 recover.cpp:451] Replica is in EMPTY status
> I0729 17:10:10.502825 1212 replica.cpp:638] Replica in EMPTY status received
> a broadcasted recover request
> I0729 17:10:10.502877 1212 recover.cpp:188] Received a recover response from
> a replica in EMPTY status
> I0729 17:10:10.502980 1212 recover.cpp:542] Updating replica status to
> STARTING
> I0729 17:10:10.508482 1213 master.cpp:289] Master
> 20140729-171010-16842879-54701-1176 (trusty) started on 127.0.1.1:54701
> I0729 17:10:10.508607 1213 master.cpp:326] Master only allowing
> authenticated frameworks to register
> I0729 17:10:10.508632 1213 master.cpp:331] Master only allowing
> authenticated slaves to register
> I0729 17:10:10.508656 1213 credentials.hpp:36] Loading credentials for
> authentication from '/tmp/HealthCheckTest_GracePeriod_d7zCPr/credentials'
> I0729 17:10:10.509407 1213 master.cpp:360] Authorization enabled
> I0729 17:10:10.510030 1207 hierarchical_allocator_process.hpp:301]
> Initializing hierarchical allocator process with master :
> master@127.0.1.1:54701
> I0729 17:10:10.510113 1207 master.cpp:123] No whitelist given. Advertising
> offers for all slaves
> I0729 17:10:10.511699 1213 master.cpp:1129] The newly elected leader is
> master@127.0.1.1:54701 with id 20140729-171010-16842879-54701-1176
> I0729 17:10:10.512230 1213 master.cpp:1142] Elected as the leading master!
> I0729 17:10:10.512692 1213 master.cpp:960] Recovering from registrar
> I0729 17:10:10.513226 1210 registrar.cpp:313] Recovering registrar
> I0729 17:10:10.516006 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 12.946461ms
> I0729 17:10:10.516047 1212 replica.cpp:320] Persisted replica status to
> STARTING
> I0729 17:10:10.516129 1212 recover.cpp:451] Replica is in STARTING status
> I0729 17:10:10.516520 1212 replica.cpp:638] Replica in STARTING status
> received a broadcasted recover request
> I0729 17:10:10.516592 1212 recover.cpp:188] Received a recover response from
> a replica in STARTING status
> I0729 17:10:10.516767 1212 recover.cpp:542] Updating replica status to VOTING
> I0729 17:10:10.528376 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 11.537102ms
> I0729 17:10:10.528430 1212 replica.cpp:320] Persisted replica status to
> VOTING
> I0729 17:10:10.528501 1212 recover.cpp:556] Successfully joined the Paxos
> group
> I0729 17:10:10.528565 1212 recover.cpp:440] Recover process terminated
> I0729 17:10:10.528700 1212 log.cpp:656] Attempting to start the writer
> I0729 17:10:10.528960 1212 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0729 17:10:10.537821 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 8.830863ms
> I0729 17:10:10.537869 1212 replica.cpp:342] Persisted promised to 1
> I0729 17:10:10.540550 1209 coordinator.cpp:230] Coordinator attemping to
> fill missing position
> I0729 17:10:10.540856 1209 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0729 17:10:10.547430 1209 leveldb.cpp:343] Persisting action (8 bytes) to
> leveldb took 6.548344ms
> I0729 17:10:10.547471 1209 replica.cpp:676] Persisted action at 0
> I0729 17:10:10.547732 1209 replica.cpp:508] Replica received write request
> for position 0
> I0729 17:10:10.547765 1209 leveldb.cpp:438] Reading position from leveldb
> took 15676ns
> I0729 17:10:10.557169 1209 leveldb.cpp:343] Persisting action (14 bytes) to
> leveldb took 9.373798ms
> I0729 17:10:10.557
[jira] [Updated] (MESOS-1653) HealthCheckTest.GracePeriod is flaky.
[
https://issues.apache.org/jira/browse/MESOS-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Cody Maloney updated MESOS-1653:
Labels: flaky (was: )
> HealthCheckTest.GracePeriod is flaky.
> -
>
> Key: MESOS-1653
> URL: https://issues.apache.org/jira/browse/MESOS-1653
> Project: Mesos
> Issue Type: Bug
> Components: test
>Reporter: Benjamin Mahler
>Assignee: Timothy Chen
> Labels: flaky
>
> {noformat}
> [--] 3 tests from HealthCheckTest
> [ RUN ] HealthCheckTest.GracePeriod
> Using temporary directory '/tmp/HealthCheckTest_GracePeriod_d7zCPr'
> I0729 17:10:10.484951 1176 leveldb.cpp:176] Opened db in 28.883552ms
> I0729 17:10:10.499487 1176 leveldb.cpp:183] Compacted db in 13.674118ms
> I0729 17:10:10.500200 1176 leveldb.cpp:198] Created db iterator in 7394ns
> I0729 17:10:10.500692 1176 leveldb.cpp:204] Seeked to beginning of db in
> 2317ns
> I0729 17:10:10.501113 1176 leveldb.cpp:273] Iterated through 0 keys in the
> db in 1367ns
> I0729 17:10:10.501535 1176 replica.cpp:741] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0729 17:10:10.502233 1212 recover.cpp:425] Starting replica recovery
> I0729 17:10:10.502295 1212 recover.cpp:451] Replica is in EMPTY status
> I0729 17:10:10.502825 1212 replica.cpp:638] Replica in EMPTY status received
> a broadcasted recover request
> I0729 17:10:10.502877 1212 recover.cpp:188] Received a recover response from
> a replica in EMPTY status
> I0729 17:10:10.502980 1212 recover.cpp:542] Updating replica status to
> STARTING
> I0729 17:10:10.508482 1213 master.cpp:289] Master
> 20140729-171010-16842879-54701-1176 (trusty) started on 127.0.1.1:54701
> I0729 17:10:10.508607 1213 master.cpp:326] Master only allowing
> authenticated frameworks to register
> I0729 17:10:10.508632 1213 master.cpp:331] Master only allowing
> authenticated slaves to register
> I0729 17:10:10.508656 1213 credentials.hpp:36] Loading credentials for
> authentication from '/tmp/HealthCheckTest_GracePeriod_d7zCPr/credentials'
> I0729 17:10:10.509407 1213 master.cpp:360] Authorization enabled
> I0729 17:10:10.510030 1207 hierarchical_allocator_process.hpp:301]
> Initializing hierarchical allocator process with master :
> master@127.0.1.1:54701
> I0729 17:10:10.510113 1207 master.cpp:123] No whitelist given. Advertising
> offers for all slaves
> I0729 17:10:10.511699 1213 master.cpp:1129] The newly elected leader is
> master@127.0.1.1:54701 with id 20140729-171010-16842879-54701-1176
> I0729 17:10:10.512230 1213 master.cpp:1142] Elected as the leading master!
> I0729 17:10:10.512692 1213 master.cpp:960] Recovering from registrar
> I0729 17:10:10.513226 1210 registrar.cpp:313] Recovering registrar
> I0729 17:10:10.516006 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 12.946461ms
> I0729 17:10:10.516047 1212 replica.cpp:320] Persisted replica status to
> STARTING
> I0729 17:10:10.516129 1212 recover.cpp:451] Replica is in STARTING status
> I0729 17:10:10.516520 1212 replica.cpp:638] Replica in STARTING status
> received a broadcasted recover request
> I0729 17:10:10.516592 1212 recover.cpp:188] Received a recover response from
> a replica in STARTING status
> I0729 17:10:10.516767 1212 recover.cpp:542] Updating replica status to VOTING
> I0729 17:10:10.528376 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 11.537102ms
> I0729 17:10:10.528430 1212 replica.cpp:320] Persisted replica status to
> VOTING
> I0729 17:10:10.528501 1212 recover.cpp:556] Successfully joined the Paxos
> group
> I0729 17:10:10.528565 1212 recover.cpp:440] Recover process terminated
> I0729 17:10:10.528700 1212 log.cpp:656] Attempting to start the writer
> I0729 17:10:10.528960 1212 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0729 17:10:10.537821 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 8.830863ms
> I0729 17:10:10.537869 1212 replica.cpp:342] Persisted promised to 1
> I0729 17:10:10.540550 1209 coordinator.cpp:230] Coordinator attemping to
> fill missing position
> I0729 17:10:10.540856 1209 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0729 17:10:10.547430 1209 leveldb.cpp:343] Persisting action (8 bytes) to
> leveldb took 6.548344ms
> I0729 17:10:10.547471 1209 replica.cpp:676] Persisted action at 0
> I0729 17:10:10.547732 1209 replica.cpp:508] Replica received write request
> for position 0
> I0729 17:10:10.547765 1209 leveldb.cpp:438] Reading position from leveldb
> took 15676ns
> I0729 17:10:10.557169 1209 leveldb.cpp:343] Persisting action (14 bytes) to
> leveldb took 9.373798ms
> I0729 17:10:10.557241 1209 replica.cpp:676] Persisted action at 0
> I0729 1
[jira] [Updated] (MESOS-1653) HealthCheckTest.GracePeriod is flaky.
[
https://issues.apache.org/jira/browse/MESOS-1653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benjamin Mahler updated MESOS-1653:
---
Fix Version/s: (was: 0.20.0)
> HealthCheckTest.GracePeriod is flaky.
> -
>
> Key: MESOS-1653
> URL: https://issues.apache.org/jira/browse/MESOS-1653
> Project: Mesos
> Issue Type: Bug
> Components: test
>Reporter: Benjamin Mahler
>Assignee: Timothy Chen
>
> {noformat}
> [--] 3 tests from HealthCheckTest
> [ RUN ] HealthCheckTest.GracePeriod
> Using temporary directory '/tmp/HealthCheckTest_GracePeriod_d7zCPr'
> I0729 17:10:10.484951 1176 leveldb.cpp:176] Opened db in 28.883552ms
> I0729 17:10:10.499487 1176 leveldb.cpp:183] Compacted db in 13.674118ms
> I0729 17:10:10.500200 1176 leveldb.cpp:198] Created db iterator in 7394ns
> I0729 17:10:10.500692 1176 leveldb.cpp:204] Seeked to beginning of db in
> 2317ns
> I0729 17:10:10.501113 1176 leveldb.cpp:273] Iterated through 0 keys in the
> db in 1367ns
> I0729 17:10:10.501535 1176 replica.cpp:741] Replica recovered with log
> positions 0 -> 0 with 1 holes and 0 unlearned
> I0729 17:10:10.502233 1212 recover.cpp:425] Starting replica recovery
> I0729 17:10:10.502295 1212 recover.cpp:451] Replica is in EMPTY status
> I0729 17:10:10.502825 1212 replica.cpp:638] Replica in EMPTY status received
> a broadcasted recover request
> I0729 17:10:10.502877 1212 recover.cpp:188] Received a recover response from
> a replica in EMPTY status
> I0729 17:10:10.502980 1212 recover.cpp:542] Updating replica status to
> STARTING
> I0729 17:10:10.508482 1213 master.cpp:289] Master
> 20140729-171010-16842879-54701-1176 (trusty) started on 127.0.1.1:54701
> I0729 17:10:10.508607 1213 master.cpp:326] Master only allowing
> authenticated frameworks to register
> I0729 17:10:10.508632 1213 master.cpp:331] Master only allowing
> authenticated slaves to register
> I0729 17:10:10.508656 1213 credentials.hpp:36] Loading credentials for
> authentication from '/tmp/HealthCheckTest_GracePeriod_d7zCPr/credentials'
> I0729 17:10:10.509407 1213 master.cpp:360] Authorization enabled
> I0729 17:10:10.510030 1207 hierarchical_allocator_process.hpp:301]
> Initializing hierarchical allocator process with master :
> master@127.0.1.1:54701
> I0729 17:10:10.510113 1207 master.cpp:123] No whitelist given. Advertising
> offers for all slaves
> I0729 17:10:10.511699 1213 master.cpp:1129] The newly elected leader is
> master@127.0.1.1:54701 with id 20140729-171010-16842879-54701-1176
> I0729 17:10:10.512230 1213 master.cpp:1142] Elected as the leading master!
> I0729 17:10:10.512692 1213 master.cpp:960] Recovering from registrar
> I0729 17:10:10.513226 1210 registrar.cpp:313] Recovering registrar
> I0729 17:10:10.516006 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 12.946461ms
> I0729 17:10:10.516047 1212 replica.cpp:320] Persisted replica status to
> STARTING
> I0729 17:10:10.516129 1212 recover.cpp:451] Replica is in STARTING status
> I0729 17:10:10.516520 1212 replica.cpp:638] Replica in STARTING status
> received a broadcasted recover request
> I0729 17:10:10.516592 1212 recover.cpp:188] Received a recover response from
> a replica in STARTING status
> I0729 17:10:10.516767 1212 recover.cpp:542] Updating replica status to VOTING
> I0729 17:10:10.528376 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 11.537102ms
> I0729 17:10:10.528430 1212 replica.cpp:320] Persisted replica status to
> VOTING
> I0729 17:10:10.528501 1212 recover.cpp:556] Successfully joined the Paxos
> group
> I0729 17:10:10.528565 1212 recover.cpp:440] Recover process terminated
> I0729 17:10:10.528700 1212 log.cpp:656] Attempting to start the writer
> I0729 17:10:10.528960 1212 replica.cpp:474] Replica received implicit
> promise request with proposal 1
> I0729 17:10:10.537821 1212 leveldb.cpp:306] Persisting metadata (8 bytes) to
> leveldb took 8.830863ms
> I0729 17:10:10.537869 1212 replica.cpp:342] Persisted promised to 1
> I0729 17:10:10.540550 1209 coordinator.cpp:230] Coordinator attemping to
> fill missing position
> I0729 17:10:10.540856 1209 replica.cpp:375] Replica received explicit
> promise request for position 0 with proposal 2
> I0729 17:10:10.547430 1209 leveldb.cpp:343] Persisting action (8 bytes) to
> leveldb took 6.548344ms
> I0729 17:10:10.547471 1209 replica.cpp:676] Persisted action at 0
> I0729 17:10:10.547732 1209 replica.cpp:508] Replica received write request
> for position 0
> I0729 17:10:10.547765 1209 leveldb.cpp:438] Reading position from leveldb
> took 15676ns
> I0729 17:10:10.557169 1209 leveldb.cpp:343] Persisting action (14 bytes) to
> leveldb took 9.373798ms
> I0729 17:10:10.557241 1209 replica.cpp:676] Persisted action at 0
> I0729 17:10:10.56064
