GitHub user uce opened a pull request: https://github.com/apache/flink/pull/407
[builds] Allow to upload build artifacts to S3 This PR includes #389. Currently, logging of Travis builds is disabled as the log files are discarded after a build and stdout logging is limited in size. With this PR, we can log to a file and have them uploaded automatically. 1. **Console logging**: For console logging, we can use `MavenForkNumberPrefixLayout` to prefix the log output with the fork number, which makes it easier to group log output together. We might consider moving that layout to a different package, though. ``` 7 > 17:12:23,220 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (5/7) (attempt #0) to host3 7 > 17:12:23,220 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (6/7) - execution #0, sharingUnit=null, locationConstraint=null} 7 > 17:12:23,220 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained assignment: test vertex (6/7) --> dbc442af2610e30d3309d96e14fa9d8a @ akka://TestingActorSystem/user/$$a - 5 slots - 1067599825 7 > 17:12:23,221 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (6/7) (attempt #0) to host1 7 > 17:12:23,221 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (7/7) - execution #0, sharingUnit=null, locationConstraint=null} 7 > 17:12:23,222 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Unconstrained assignment: test vertex (7/7) --> 8ce93c214951df5a8ad32c912ba0e456 @ akka://TestingActorSystem/user/$$a - 5 slots - 749927456 7 > 17:12:23,222 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (7/7) (attempt #0) to host2 7 > 17:12:23,223 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results. 7 > 17:12:23,223 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting ExecutionJobVertex 126e16c348d4e5aa129104702700bf9e (test vertex) to 0 predecessors. 10 > 17:12:23,225 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying v1 (2/2) (attempt #0) to host3 6 > 17:12:23,236 DEBUG akka.event.EventStream - shutting down: StandardOutLogger started 1 > 17:12:23,239 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Sending out cancel request, to remove task execution from TaskManager. 1 > 17:12:23,244 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Concurrent canceling/failing of TestVertex (1/1) - execution #0 while deployment was in progress. 10 > 17:12:23,266 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results. Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.61 sec - in org.apache.flink.runtime.executiongraph.ExecutionVertexCancelTest Running org.apache.flink.runtime.io.network.api.reader.UnionBufferReaderTest 10 > 17:12:23,269 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting ExecutionJobVertex e8869426e7765e498ad219e3af731729 (test vertex) to 0 predecessors. 10 > 17:12:23,269 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (1/2) - execution #0, sharingUnit=null, locationConstraint=null} 10 > 17:12:23,270 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (1/2) --> ee9357ffc16d51c4f7522bc479a757ac @ akka://TestingActorSystem/user/$$a - 1 slots - 2073640037 10 > 17:12:23,270 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (1/2) (attempt #0) to host1 10 > 17:12:23,270 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (2/2) - execution #0, sharingUnit=null, locationConstraint=null} 10 > 17:12:23,271 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (2/2) --> 6620d9f45b7567e2b147f5761d8060e8 @ akka://TestingActorSystem/user/$$a - 1 slots - 1864116663 10 > 17:12:23,271 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (2/2) (attempt #0) to host3 1 > 17:12:23,277 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying TestVertex (1/1) (attempt #0) to localhost 10 > 17:12:23,277 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Attaching 1 topologically sorted vertices to existing job graph with 0 vertices and 0 intermediate results. 10 > 17:12:23,278 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Connecting ExecutionJobVertex 533fcd1e05bcce9ec185ccea7085a5aa (test vertex) to 0 predecessors. 10 > 17:12:23,278 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (1/2) - execution #0, sharingUnit=null, locationConstraint=null} 1 > 17:12:23,280 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Sending out cancel request, to remove task execution from TaskManager. 1 > 17:12:23,280 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Cancel task call did not find task. Probably akka message call race. 1 > 17:12:23,281 DEBUG org.apache.flink.runtime.executiongraph.ExecutionGraph - Concurrent canceling/failing of TestVertex (1/1) - execution #0 while deployment was in progress. 10 > 17:12:23,281 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (1/2) --> b923679116189038c5dbd280d43cfe48 @ akka://TestingActorSystem/user/$$a - 1 slots - 44559647 10 > 17:12:23,282 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph - Deploying test vertex (1/2) (attempt #0) to host3 10 > 17:12:23,282 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Scheduling task {task=test vertex (2/2) - execution #0, sharingUnit=null, locationConstraint=null} 10 > 17:12:23,283 DEBUG org.apache.flink.runtime.jobmanager.scheduler.Scheduler - Local assignment: test vertex (2/2) --> 1fb78d024e2664602ee31c47dd987f6c @ akka://TestingActorSystem/user/$$a - 1 slots - 1081769770 ``` I've added an example log4j properties file to `flink-runtime/tests`. 2. **File logging**: Each forked build sets its own log file (e.g. `1.log` and `2.log`) if it is enabled (example in `flink-runtime/tests` log4j properties file). If the `UPLOAD_ARTIFACTS` variable is set in Travis and the S3 credentials have been set for the respective repository, the files are uploaded. As an example, see [this build](https://travis-ci.org/uce/incubator-flink/builds/50960309), which I've stalled on purpose. It upload the jps traces: https://s3.amazonaws.com/flink.a.o.uce.east/travis-artifacts/uce/incubator-flink/210/210.5/jps-traces.out --- The used S3 bucket is my private bucket and the upload currently only works for my forked repository. It is very easy to add more committers for their own repositories/own buckets. We can finalize the setup for interested committers after review. I still want to add a minor change to compress the output. You can merge this pull request into a Git repository by running: $ git pull https://github.com/uce/incubator-flink travis-upload_artifacts Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/407.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #407 ---- commit 7887599e27949f72373232205d15010c18108bc2 Author: Ufuk Celebi <u...@apache.org> Date: 2015-02-12T11:19:40Z [builds] Print Java process stack traces of stalled builds commit cb17e87b3d47336f50999c2e239d8ec6467e0a59 Author: Ufuk Celebi <u...@apache.org> Date: 2015-02-16T15:40:49Z [builds] Add logger layout to prefix log output with Maven fork number commit 488946ee273de9c374ec0a87652077c355718cae Author: Ufuk Celebi <u...@apache.org> Date: 2015-02-16T15:42:13Z [builds] Move .gitignores and set JVM properties when running build commit 2a942e0980e80201acc954abc20401a3f9c8764c Author: Ufuk Celebi <u...@apache.org> Date: 2015-02-16T15:48:07Z [builds] Allow to upload build artifacts to S3 commit 5cd3e395cea2e64b76edebb26ec6a517b111fc85 Author: Ufuk Celebi <u...@apache.org> Date: 2015-02-16T15:56:48Z [builds] Fix rat excludes ---- --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---