GitHub user uce opened a pull request:
https://github.com/apache/flink/pull/407
[builds] Allow to upload build artifacts to S3
This PR includes #389.
Currently, logging of Travis builds is disabled as the log files are
discarded after a build and stdout logging is limited in size. With this PR, we
can log to a file and have them uploaded automatically.
1. **Console logging**: For console logging, we can use
`MavenForkNumberPrefixLayout` to prefix the log output with the fork number,
which makes it easier to group log output together.
We might consider moving that layout to a different package, though.
```
7 > 17:12:23,220 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test
vertex (5/7) (attempt #0) to host3
7 > 17:12:23,220 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (6/7) - execution #0, sharingUnit=null,
locationConstraint=null}
7 > 17:12:23,220 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained
assignment: test vertex (6/7) --> dbc442af2610e30d3309d96e14fa9d8a @
akka://TestingActorSystem/user/$$a - 5 slots - 1067599825
7 > 17:12:23,221 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test
vertex (6/7) (attempt #0) to host1
7 > 17:12:23,221 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (7/7) - execution #0, sharingUnit=null,
locationConstraint=null}
7 > 17:12:23,222 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained
assignment: test vertex (7/7) --> 8ce93c214951df5a8ad32c912ba0e456 @
akka://TestingActorSystem/user/$$a - 5 slots - 749927456
7 > 17:12:23,222 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test
vertex (7/7) (attempt #0) to host2
7 > 17:12:23,223 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1
topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
7 > 17:12:23,223 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting
ExecutionJobVertex 126e16c348d4e5aa129104702700bf9e (test vertex) to 0
predecessors.
10 > 17:12:23,225 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying v1
(2/2) (attempt #0) to host3
6 > 17:12:23,236 DEBUG akka.event.EventStream
- shutting down: StandardOutLogger started
1 > 17:12:23,239 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Sending out
cancel request, to remove task execution from TaskManager.
1 > 17:12:23,244 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Concurrent
canceling/failing of TestVertex (1/1) - execution #0 while deployment was in
progress.
10 > 17:12:23,266 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1
topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.61 sec -
in org.apache.flink.runtime.executiongraph.ExecutionVertexCancelTest
Running org.apache.flink.runtime.io.network.api.reader.UnionBufferReaderTest
10 > 17:12:23,269 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting
ExecutionJobVertex e8869426e7765e498ad219e3af731729 (test vertex) to 0
predecessors.
10 > 17:12:23,269 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (1/2) - execution #0, sharingUnit=null,
locationConstraint=null}
10 > 17:12:23,270 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local
assignment: test vertex (1/2) --> ee9357ffc16d51c4f7522bc479a757ac @
akka://TestingActorSystem/user/$$a - 1 slots - 2073640037
10 > 17:12:23,270 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test
vertex (1/2) (attempt #0) to host1
10 > 17:12:23,270 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (2/2) - execution #0, sharingUnit=null,
locationConstraint=null}
10 > 17:12:23,271 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local
assignment: test vertex (2/2) --> 6620d9f45b7567e2b147f5761d8060e8 @
akka://TestingActorSystem/user/$$a - 1 slots - 1864116663
10 > 17:12:23,271 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test
vertex (2/2) (attempt #0) to host3
1 > 17:12:23,277 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying
TestVertex (1/1) (attempt #0) to localhost
10 > 17:12:23,277 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1
topologically sorted vertices to existing job graph with 0 vertices and 0
intermediate results.
10 > 17:12:23,278 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting
ExecutionJobVertex 533fcd1e05bcce9ec185ccea7085a5aa (test vertex) to 0
predecessors.
10 > 17:12:23,278 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (1/2) - execution #0, sharingUnit=null,
locationConstraint=null}
1 > 17:12:23,280 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Sending out
cancel request, to remove task execution from TaskManager.
1 > 17:12:23,280 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Cancel task
call did not find task. Probably akka message call race.
1 > 17:12:23,281 DEBUG
org.apache.flink.runtime.executiongraph.ExecutionGraph - Concurrent
canceling/failing of TestVertex (1/1) - execution #0 while deployment was in
progress.
10 > 17:12:23,281 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local
assignment: test vertex (1/2) --> b923679116189038c5dbd280d43cfe48 @
akka://TestingActorSystem/user/$$a - 1 slots - 44559647
10 > 17:12:23,282 INFO
org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test
vertex (1/2) (attempt #0) to host3
10 > 17:12:23,282 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task
{task=test vertex (2/2) - execution #0, sharingUnit=null,
locationConstraint=null}
10 > 17:12:23,283 DEBUG
org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local
assignment: test vertex (2/2) --> 1fb78d024e2664602ee31c47dd987f6c @
akka://TestingActorSystem/user/$$a - 1 slots - 1081769770
```
I've added an example log4j properties file to `flink-runtime/tests`.
2. **File logging**: Each forked build sets its own log file (e.g. `1.log`
and `2.log`) if it is enabled (example in `flink-runtime/tests` log4j
properties file). If the `UPLOAD_ARTIFACTS` variable is set in Travis and the
S3 credentials have been set for the respective repository, the files are
uploaded.
As an example, see [this
build](https://travis-ci.org/uce/incubator-flink/builds/50960309), which I've
stalled on purpose. It upload the jps traces:
https://s3.amazonaws.com/flink.a.o.uce.east/travis-artifacts/uce/incubator-flink/210/210.5/jps-traces.out
---
The used S3 bucket is my private bucket and the upload currently only works
for my forked repository. It is very easy to add more committers for their own
repositories/own buckets.
We can finalize the setup for interested committers after review. I still
want to add a minor change to compress the output.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/uce/incubator-flink travis-upload_artifacts
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/flink/pull/407.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #407
----
commit 7887599e27949f72373232205d15010c18108bc2
Author: Ufuk Celebi <[email protected]>
Date: 2015-02-12T11:19:40Z
[builds] Print Java process stack traces of stalled builds
commit cb17e87b3d47336f50999c2e239d8ec6467e0a59
Author: Ufuk Celebi <[email protected]>
Date: 2015-02-16T15:40:49Z
[builds] Add logger layout to prefix log output with Maven fork number
commit 488946ee273de9c374ec0a87652077c355718cae
Author: Ufuk Celebi <[email protected]>
Date: 2015-02-16T15:42:13Z
[builds] Move .gitignores and set JVM properties when running build
commit 2a942e0980e80201acc954abc20401a3f9c8764c
Author: Ufuk Celebi <[email protected]>
Date: 2015-02-16T15:48:07Z
[builds] Allow to upload build artifacts to S3
commit 5cd3e395cea2e64b76edebb26ec6a517b111fc85
Author: Ufuk Celebi <[email protected]>
Date: 2015-02-16T15:56:48Z
[builds] Fix rat excludes
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
