[GitHub] zeppelin pull request #1648: [ZEPPELIN-1673] Reduce CI log size and make it ...

[1ambda]

GitHub user 1ambda reopened a pull request:

    https://github.com/apache/zeppelin/pull/1648

    [ZEPPELIN-1673] Reduce CI log size and make it faster

    ### What is this PR for?
    
    #### 1. APT is now cached (saving almost **15 secs** per build)
    
    (before)
    <img width="657" alt="screen shot 2016-11-17 at 11 39 39 am" 
src="https://cloud.githubusercontent.com/assets/4968473/20375553/2b306884-acc3-11e6-995b-045dec268e0a.png";>
    
    (after)
    <img width="663" alt="screen shot 2016-11-17 at 11 39 51 am" 
src="https://cloud.githubusercontent.com/assets/4968473/20375557/2ec0c7aa-acc3-11e6-8468-f672be52826e.png";>
    
    #### 2. R packages are now cached (almost **250 secs per** build)
    
    (before)
    <img width="1131" alt="screen shot 2016-11-17 at 12 04 20 pm" 
src="https://cloud.githubusercontent.com/assets/4968473/20375573/4c9cbc84-acc3-11e6-9717-ce6a6d9ad360.png";>
    
    (after)
    <img width="1134" alt="screen shot 2016-11-17 at 12 37 19 pm" 
src="https://cloud.githubusercontent.com/assets/4968473/20375585/526ccc58-acc3-11e6-95a9-8446a298597a.png";>
    <img width="340" alt="screen shot 2016-11-17 at 12 37 34 pm" 
src="https://cloud.githubusercontent.com/assets/4968473/20375588/57227f68-acc3-11e6-8ef7-9a86b125c00d.png";>
    
    #### 3. Log size is reduced (at least 30%, removing useless maven info logs)
    
    actually, we reduced even more because we cached apt and R :)
    
    ```bash
    // (before result) 
https://api.travis-ci.org/jobs/176574582/log.txt?deansi=true
    // (before build) https://travis-ci.org/apache/zeppelin/builds/176574580
    
    // (after result) 
https://api.travis-ci.org/jobs/176576354/log.txt?deansi=true
    // (after build) https://travis-ci.org/apache/zeppelin/builds/176576351
    
    $ ls
    build8414_profile2.log build8416_profile2.log
    
    $ ls -alh
    total 10232
    drwxr-xr-x   4 lambda  staff   136B Nov 17 12:47 .
    drwxr-xr-x  13 lambda  staff   442B Nov 17 02:49 ..
    -rw-r--r--   1 lambda  staff   3.0M Nov 17 12:46 build8414_profile2.log
    -rw-r--r--   1 lambda  staff   2.0M Nov 17 12:47 build8416_profile2.log
    
    $ cat build8414_profile2.log | grep Download | wc -l
        7186
    $ cat build8416_profile2.log | grep Download | wc -l
           6
    ```
    
    #### Long Motivations
    
    Sometimes build fails due to log size for example, 
    
    ```
    Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec - 
in org.apache.zeppelin.ticket.TicketContainerTest
    Running org.apache.zeppelin.security.SecurityUtilsTest
    
    
    The log length has exceeded the limit of 4 MB (this usually means that the 
test suite is raising the same exception over and over).
    
    The job has been terminated
    ```
    
    we can prevent this kind of failure by removing useless maven log such as 
    
    ```
    [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.pom
 (13 KB at 1233.1 KB/sec)
    [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.jar
    [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.13/maven-checkstyle-plugin-2.13.jar
 (112 KB at 4858.2 KB/sec)
    [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.pom
    [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.pom
 (8 KB at 627.2 KB/sec)
    [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.jar
    [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-resources-plugin/2.7/maven-resources-plugin-2.7.jar
 (31 KB at 2048.3 KB/sec)
    [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom
    [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.pom
 (12 KB at 860.4 KB/sec)
    [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar
    [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-dependency-plugin/2.8/maven-dependency-plugin-2.8.jar
 (150 KB at 5342.7 KB/sec)
    ```
    
    ```sh
    $  CI cat 176445046.log | grep Download | wc -l
        7044
    $  CI cat 176445046.log | wc -l
       38046
    $  CI cat 176445046.log| grep -v Download > filtered.log
    $  CI ls -alh
    total 7.7M
    drwxr-xr-x  4 lambda staff  136 Nov 17 02:50 .
    drwxr-xr-x 13 lambda staff  442 Nov 17 02:49 ..
    -rw-r--r--  1 lambda staff 4.3M Nov 17 02:49 176445046.log
    -rw-r--r--  1 lambda staff 3.4M Nov 17 02:50 filtered.log
    ```
    
    ### What type of PR is it?
    [Improvement]
    
    ### What is the Jira issue?
    
    [ZEPPELIN-1673](https://issues.apache.org/jira/browse/ZEPPELIN-1673)
    
    ### How should this be tested?
    
    Checkout CI log and see if there are `Downloading` `Downloaded` log
    
    (see also https://api.travis-ci.org/jobs/176445045/log.txt?deansi=true)
    
    ### Screenshots (if appropriate)
    
    ### Questions:
    * Does the licenses files need update? - NO
    * Is there breaking changes for older versions? - NO
    * Does this needs documentation? - NO
    


You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/1ambda/zeppelin feat-improve-ci

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/zeppelin/pull/1648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #1648
    
----
commit 06c006eff45a3525cd82f261ad1c8b0537d7d277
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-16T17:59:02Z

    fix: Disable download INFO log

commit a9bb56f89b03c90169e4e79453aadb3350fa7883
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-16T18:17:56Z

    fix: Add quote to global env

commit f6e0ca0af2aa59d26488de978f94f2f87be9c626
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T01:08:49Z

    fix: Add mvn option

commit 1b9ca09ccfd1e2df7c6df6550b7f9923d8c5d392
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T01:11:44Z

    fix: travis cache dir

commit 98452a7ea78ad055b63b4e58a5d530531c03fcbc
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T01:19:44Z

    fix: set quite flag to mvn command

commit 7d7ae0bd54d59182c73e5590ae30969094a2771d
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T01:28:24Z

    chore: Test mvn version

commit f404bb317fac36c74a20920052ab2988560da9b8
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T01:53:30Z

    chore: Set maven log level to warn

commit 756f546209af521809956b8dae9cf2ae28bf7b7c
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T02:09:36Z

    feat: Add apt cache

commit 6e20282830e5dd12695f910bd457b77c4331af0b
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T02:37:35Z

    chore: Cache R packages

commit 58e14a94c3711cb09e6c711bd646c831cae7a61b
Author: 1ambda <1am...@gmail.com>
Date:   2016-11-17T03:03:26Z

    test: R cache

----


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---