Re: workspace cleanups needed on jenkins master

2019-01-03 Thread Udi Meiri 
Alan, Mikhail, feel free to merge https://github.com/apache/beam/pull/7410
when ready.

On Thu, Jan 3, 2019 at 5:39 PM Mikhail Gryzykhin  wrote:

> It is not required for https://s.apache.org/beam-community-metrics .
>
> I believe that's main dash we have atm.
>
> @Alan Myrvold  Can you confirm?
>
> --Mikhail
>
> Have feedback ?
>
>
> On Thu, Jan 3, 2019 at 1:59 PM Udi Meiri  wrote:
>
>>
>>
>> On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía  wrote:
>>
>>> Bringing this subject for awareness to dev@
>>> We are sadly part of this top.
>>> Does somebody know what this data is? And if we can clean it
>>> periodically?
>>> Can somebody with more sysadmin super powers take a look and act on this.
>>>
>>> -- Forwarded message -
>>> From: Chris Lambertus 
>>> Date: Thu, Dec 27, 2018 at 1:36 AM
>>> Subject: workspace cleanups needed on jenkins master
>>> To: 
>>>
>>>
>>> All,
>>>
>>> The Jenkins master needs to be cleaned up. Could the following
>>> projects please reduce your usage significantly by 5 January. After 5
>>> Jan Infra will be purging more aggressively and updating job
>>> configurations as needed. As a rule of thumb, we’d like to see
>>> projects retain no more than 1 week or 7 builds worth of historical
>>> data at the absolute maximum. Larger projects should retain less to
>>> avoid using up a disproportionate amount of space on the master.
>>>
>>> Some workspaces without any identifiable associated Project will be
>>> removed.
>>>
>>>
>>>
>>> 3911 GB .
>>> 275 GB ./incubator-netbeans-linux
>>> 270 GB ./pulsar-website-build
>>> 249 GB ./pulsar-master
>>> 199 GB ./Packaging
>>> 127 GB ./HBase
>>> 121 GB ./PreCommit-ZOOKEEPER-github-pr-build
>>> 107 GB ./Any23-trunk
>>> 102 GB ./incubator-netbeans-release
>>> 79 GB ./incubator-netbeans-linux-experiment
>>> 77 GB ./beam_PostCommit_Java_PVR_Flink_Batch
>>>
>>
>> Wow, this job has huge logs (400MB+).
>> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Java_PVR_Flink_Batch/330/console
>> A few weeks back I suggested removing the --info flag passed to Gradle.
>> I haven't done that yet, but it might help reduce the size of logs.
>> (which I assume are all stored on master?)
>>
>> Short-term, we could reduce retention back down to 14 instead of the current
>> 30
>> 
>> .
>> +Alan Myrvold  +Mikhail Gryzykhin
>>   we don't need 30 days retention any longer for test
>> dashboards, right?
>>
>>
>>> 76 GB ./HBase-1.3-JDK8
>>> 70 GB ./Jackrabbit-Oak-Windows
>>> 70 GB ./stanbol-0.12
>>> 59 GB ./HBase-Find-Flaky-Tests
>>> 54 GB ./CouchDB
>>> 51 GB ./beam_PostCommit_Java_PVR_Flink_Streaming
>>> 48 GB ./incubator-netbeans-windows
>>> 47 GB ./FlexJS
>>> 42 GB ./HBase
>>> 41 GB ./pulsar-pull-request
>>> 37 GB ./ZooKeeper_branch35_jdk8
>>> 32 GB ./HBase-Flaky-Tests
>>> 32 GB ./Atlas-master-NoTests
>>> 31 GB ./Atlas-1.0-NoTests
>>> 31 GB ./beam_PreCommit_Java_Commit
>>> 30 GB ./Zookeeper_UT_Java18
>>> 29 GB ./Phoenix-4.x-HBase-1.4
>>> 28 GB ./HBase-2.0-hadoop3-tests
>>> 27 GB ./flink-github-ci
>>> 27 GB ./ZooKeeper_branch35_jdk7
>>> 27 GB ./oodt-trunk
>>> 25 GB ./opennlp
>>> 25 GB ./Trinidad
>>> 22 GB ./Phoenix-4.x-HBase-1.3
>>> 21 GB ./ZooKeeper_Flaky_StressTest
>>> 21 GB ./Atlas-master-AllTests
>>> 21 GB ./beam_PostCommit_Java_ValidatesRunner_Flink
>>> 20 GB ./HBase-1.3-JDK7
>>> 20 GB ./PreCommit-HBASE-Build
>>> 18 GB ./hadoop-trunk-win
>>> 18 GB ./HBase-1.2-JDK7
>>> 18 GB ./HBASE-14070.HLC
>>> 18 GB ./maven-box
>>> 17 GB ./Atlas-1.0-AllTests
>>> 17 GB ./Archiva-TLP-Gitbox
>>> 17 GB ./Apache
>>> 17 GB ./Phoenix-5.x-HBase-2.0
>>> 17 GB ./Phoenix-omid2
>>> 16 GB ./Lucene-Solr-BadApples-NightlyTests-7.x
>>> 15 GB ./HBase-2.0
>>> 14 GB ./flume-trunk
>>> 14 GB ./beam_PostCommit_Java_ValidatesRunner_Samza
>>> 14 GB ./HBase-Trunk_matrix
>>> 13 GB ./commons-csv
>>> 13 GB ./HBase-Flaky-Tests-old-just-master
>>> 13 GB ./oodt-coverage
>>> 12 GB ./incubator-rya-master-with-optionals
>>> 12 GB ./Syncope-master-deploy
>>> 11 GB ./PreCommit-PHOENIX-Build
>>> 11 GB ./Stratos-Master-Nightly-Build
>>> 11 GB ./Phoenix-master
>>> 11 GB ./Hadoop-trunk-JACC
>>> 10 GB ./ctakes-trunk-package
>>> 10 GB ./FlexJS
>>> 10 GB ./Atlas-1.0-IntegrationTests
>>> 9 GB ./incubator-rya-master
>>> 9 GB ./Atlas-master-IntegrationTests
>>> 9 GB ./beam_PostCommit_Java_ValidatesRunner_Spark
>>> 9 GB ./ZooKeeper_UT_Java7
>>> 9 GB ./Qpid-Broker-J-7.0.x-TestMatrix
>>> 9 GB ./oodt-dependency-update
>>> 9 GB ./Apache
>>> 8 GB ./Struts-examples-JDK8-master
>>> 8 GB ./Phoenix-4.x-HBase-1.2
>>> 8 GB ./flume-github-pull-request
>>> 8 GB ./HBase-HBASE-14614
>>> 8 GB ./tika-trunk-jdk1.7
>>> 8 GB ./HBase-1.2-JDK8
>>> 8 GB ./HBase-1.5
>>> 7 GB ./Atlas-master-UnitTests
>>> 7 GB ./tika-2.x-windows
>>> 7 GB ./incubator-rya-master-with-optionals-pull-requests
>>> 7 GB ./Hive-trunk
>>> 7 GB ./beam_PreCommit_Java_Cron
>>> 7 GB

Re: workspace cleanups needed on jenkins master

2019-01-03 Thread Mikhail Gryzykhin 
It is not required for https://s.apache.org/beam-community-metrics .

I believe that's main dash we have atm.

@Alan Myrvold  Can you confirm?

--Mikhail

Have feedback ?


On Thu, Jan 3, 2019 at 1:59 PM Udi Meiri  wrote:

>
>
> On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía  wrote:
>
>> Bringing this subject for awareness to dev@
>> We are sadly part of this top.
>> Does somebody know what this data is? And if we can clean it periodically?
>> Can somebody with more sysadmin super powers take a look and act on this.
>>
>> -- Forwarded message -
>> From: Chris Lambertus 
>> Date: Thu, Dec 27, 2018 at 1:36 AM
>> Subject: workspace cleanups needed on jenkins master
>> To: 
>>
>>
>> All,
>>
>> The Jenkins master needs to be cleaned up. Could the following
>> projects please reduce your usage significantly by 5 January. After 5
>> Jan Infra will be purging more aggressively and updating job
>> configurations as needed. As a rule of thumb, we’d like to see
>> projects retain no more than 1 week or 7 builds worth of historical
>> data at the absolute maximum. Larger projects should retain less to
>> avoid using up a disproportionate amount of space on the master.
>>
>> Some workspaces without any identifiable associated Project will be
>> removed.
>>
>>
>>
>> 3911 GB .
>> 275 GB ./incubator-netbeans-linux
>> 270 GB ./pulsar-website-build
>> 249 GB ./pulsar-master
>> 199 GB ./Packaging
>> 127 GB ./HBase
>> 121 GB ./PreCommit-ZOOKEEPER-github-pr-build
>> 107 GB ./Any23-trunk
>> 102 GB ./incubator-netbeans-release
>> 79 GB ./incubator-netbeans-linux-experiment
>> 77 GB ./beam_PostCommit_Java_PVR_Flink_Batch
>>
>
> Wow, this job has huge logs (400MB+).
> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Java_PVR_Flink_Batch/330/console
> A few weeks back I suggested removing the --info flag passed to Gradle.
> I haven't done that yet, but it might help reduce the size of logs. (which
> I assume are all stored on master?)
>
> Short-term, we could reduce retention back down to 14 instead of the current
> 30
> 
> .
> +Alan Myrvold  +Mikhail Gryzykhin   we
> don't need 30 days retention any longer for test dashboards, right?
>
>
>> 76 GB ./HBase-1.3-JDK8
>> 70 GB ./Jackrabbit-Oak-Windows
>> 70 GB ./stanbol-0.12
>> 59 GB ./HBase-Find-Flaky-Tests
>> 54 GB ./CouchDB
>> 51 GB ./beam_PostCommit_Java_PVR_Flink_Streaming
>> 48 GB ./incubator-netbeans-windows
>> 47 GB ./FlexJS
>> 42 GB ./HBase
>> 41 GB ./pulsar-pull-request
>> 37 GB ./ZooKeeper_branch35_jdk8
>> 32 GB ./HBase-Flaky-Tests
>> 32 GB ./Atlas-master-NoTests
>> 31 GB ./Atlas-1.0-NoTests
>> 31 GB ./beam_PreCommit_Java_Commit
>> 30 GB ./Zookeeper_UT_Java18
>> 29 GB ./Phoenix-4.x-HBase-1.4
>> 28 GB ./HBase-2.0-hadoop3-tests
>> 27 GB ./flink-github-ci
>> 27 GB ./ZooKeeper_branch35_jdk7
>> 27 GB ./oodt-trunk
>> 25 GB ./opennlp
>> 25 GB ./Trinidad
>> 22 GB ./Phoenix-4.x-HBase-1.3
>> 21 GB ./ZooKeeper_Flaky_StressTest
>> 21 GB ./Atlas-master-AllTests
>> 21 GB ./beam_PostCommit_Java_ValidatesRunner_Flink
>> 20 GB ./HBase-1.3-JDK7
>> 20 GB ./PreCommit-HBASE-Build
>> 18 GB ./hadoop-trunk-win
>> 18 GB ./HBase-1.2-JDK7
>> 18 GB ./HBASE-14070.HLC
>> 18 GB ./maven-box
>> 17 GB ./Atlas-1.0-AllTests
>> 17 GB ./Archiva-TLP-Gitbox
>> 17 GB ./Apache
>> 17 GB ./Phoenix-5.x-HBase-2.0
>> 17 GB ./Phoenix-omid2
>> 16 GB ./Lucene-Solr-BadApples-NightlyTests-7.x
>> 15 GB ./HBase-2.0
>> 14 GB ./flume-trunk
>> 14 GB ./beam_PostCommit_Java_ValidatesRunner_Samza
>> 14 GB ./HBase-Trunk_matrix
>> 13 GB ./commons-csv
>> 13 GB ./HBase-Flaky-Tests-old-just-master
>> 13 GB ./oodt-coverage
>> 12 GB ./incubator-rya-master-with-optionals
>> 12 GB ./Syncope-master-deploy
>> 11 GB ./PreCommit-PHOENIX-Build
>> 11 GB ./Stratos-Master-Nightly-Build
>> 11 GB ./Phoenix-master
>> 11 GB ./Hadoop-trunk-JACC
>> 10 GB ./ctakes-trunk-package
>> 10 GB ./FlexJS
>> 10 GB ./Atlas-1.0-IntegrationTests
>> 9 GB ./incubator-rya-master
>> 9 GB ./Atlas-master-IntegrationTests
>> 9 GB ./beam_PostCommit_Java_ValidatesRunner_Spark
>> 9 GB ./ZooKeeper_UT_Java7
>> 9 GB ./Qpid-Broker-J-7.0.x-TestMatrix
>> 9 GB ./oodt-dependency-update
>> 9 GB ./Apache
>> 8 GB ./Struts-examples-JDK8-master
>> 8 GB ./Phoenix-4.x-HBase-1.2
>> 8 GB ./flume-github-pull-request
>> 8 GB ./HBase-HBASE-14614
>> 8 GB ./tika-trunk-jdk1.7
>> 8 GB ./HBase-1.2-JDK8
>> 8 GB ./HBase-1.5
>> 7 GB ./Atlas-master-UnitTests
>> 7 GB ./tika-2.x-windows
>> 7 GB ./incubator-rya-master-with-optionals-pull-requests
>> 7 GB ./Hive-trunk
>> 7 GB ./beam_PreCommit_Java_Cron
>> 7 GB ./Atlas-1.0-UnitTests
>> 6 GB ./Jackrabbit
>> 6 GB ./beam_PostCommit_Java_PVR_Flink_PR
>> 6 GB ./Lucene-Solr-Clover-master
>> 6 GB ./Syncope-2_0_X-deploy
>> 6 GB ./beam_PostCommit_Java_ValidatesRunner_Apex
>> 6 GB ./Tika-trunk
>> 6 GB ./pirk
>> 6 GB ./Syncope-2_1_X-deploy
>> 6 GB ./PLC4X
>> 6 GB

Re: workspace cleanups needed on jenkins master

2019-01-03 Thread Udi Meiri 
On Thu, Dec 27, 2018 at 11:02 AM Ismaël Mejía  wrote:

> Bringing this subject for awareness to dev@
> We are sadly part of this top.
> Does somebody know what this data is? And if we can clean it periodically?
> Can somebody with more sysadmin super powers take a look and act on this.
>
> -- Forwarded message -
> From: Chris Lambertus 
> Date: Thu, Dec 27, 2018 at 1:36 AM
> Subject: workspace cleanups needed on jenkins master
> To: 
>
>
> All,
>
> The Jenkins master needs to be cleaned up. Could the following
> projects please reduce your usage significantly by 5 January. After 5
> Jan Infra will be purging more aggressively and updating job
> configurations as needed. As a rule of thumb, we’d like to see
> projects retain no more than 1 week or 7 builds worth of historical
> data at the absolute maximum. Larger projects should retain less to
> avoid using up a disproportionate amount of space on the master.
>
> Some workspaces without any identifiable associated Project will be
> removed.
>
>
>
> 3911 GB .
> 275 GB ./incubator-netbeans-linux
> 270 GB ./pulsar-website-build
> 249 GB ./pulsar-master
> 199 GB ./Packaging
> 127 GB ./HBase
> 121 GB ./PreCommit-ZOOKEEPER-github-pr-build
> 107 GB ./Any23-trunk
> 102 GB ./incubator-netbeans-release
> 79 GB ./incubator-netbeans-linux-experiment
> 77 GB ./beam_PostCommit_Java_PVR_Flink_Batch
>

Wow, this job has huge logs (400MB+).
https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Java_PVR_Flink_Batch/330/console
A few weeks back I suggested removing the --info flag passed to Gradle.
I haven't done that yet, but it might help reduce the size of logs. (which
I assume are all stored on master?)

Short-term, we could reduce retention back down to 14 instead of the current
30

.
+Alan Myrvold  +Mikhail Gryzykhin   we
don't need 30 days retention any longer for test dashboards, right?


> 76 GB ./HBase-1.3-JDK8
> 70 GB ./Jackrabbit-Oak-Windows
> 70 GB ./stanbol-0.12
> 59 GB ./HBase-Find-Flaky-Tests
> 54 GB ./CouchDB
> 51 GB ./beam_PostCommit_Java_PVR_Flink_Streaming
> 48 GB ./incubator-netbeans-windows
> 47 GB ./FlexJS
> 42 GB ./HBase
> 41 GB ./pulsar-pull-request
> 37 GB ./ZooKeeper_branch35_jdk8
> 32 GB ./HBase-Flaky-Tests
> 32 GB ./Atlas-master-NoTests
> 31 GB ./Atlas-1.0-NoTests
> 31 GB ./beam_PreCommit_Java_Commit
> 30 GB ./Zookeeper_UT_Java18
> 29 GB ./Phoenix-4.x-HBase-1.4
> 28 GB ./HBase-2.0-hadoop3-tests
> 27 GB ./flink-github-ci
> 27 GB ./ZooKeeper_branch35_jdk7
> 27 GB ./oodt-trunk
> 25 GB ./opennlp
> 25 GB ./Trinidad
> 22 GB ./Phoenix-4.x-HBase-1.3
> 21 GB ./ZooKeeper_Flaky_StressTest
> 21 GB ./Atlas-master-AllTests
> 21 GB ./beam_PostCommit_Java_ValidatesRunner_Flink
> 20 GB ./HBase-1.3-JDK7
> 20 GB ./PreCommit-HBASE-Build
> 18 GB ./hadoop-trunk-win
> 18 GB ./HBase-1.2-JDK7
> 18 GB ./HBASE-14070.HLC
> 18 GB ./maven-box
> 17 GB ./Atlas-1.0-AllTests
> 17 GB ./Archiva-TLP-Gitbox
> 17 GB ./Apache
> 17 GB ./Phoenix-5.x-HBase-2.0
> 17 GB ./Phoenix-omid2
> 16 GB ./Lucene-Solr-BadApples-NightlyTests-7.x
> 15 GB ./HBase-2.0
> 14 GB ./flume-trunk
> 14 GB ./beam_PostCommit_Java_ValidatesRunner_Samza
> 14 GB ./HBase-Trunk_matrix
> 13 GB ./commons-csv
> 13 GB ./HBase-Flaky-Tests-old-just-master
> 13 GB ./oodt-coverage
> 12 GB ./incubator-rya-master-with-optionals
> 12 GB ./Syncope-master-deploy
> 11 GB ./PreCommit-PHOENIX-Build
> 11 GB ./Stratos-Master-Nightly-Build
> 11 GB ./Phoenix-master
> 11 GB ./Hadoop-trunk-JACC
> 10 GB ./ctakes-trunk-package
> 10 GB ./FlexJS
> 10 GB ./Atlas-1.0-IntegrationTests
> 9 GB ./incubator-rya-master
> 9 GB ./Atlas-master-IntegrationTests
> 9 GB ./beam_PostCommit_Java_ValidatesRunner_Spark
> 9 GB ./ZooKeeper_UT_Java7
> 9 GB ./Qpid-Broker-J-7.0.x-TestMatrix
> 9 GB ./oodt-dependency-update
> 9 GB ./Apache
> 8 GB ./Struts-examples-JDK8-master
> 8 GB ./Phoenix-4.x-HBase-1.2
> 8 GB ./flume-github-pull-request
> 8 GB ./HBase-HBASE-14614
> 8 GB ./tika-trunk-jdk1.7
> 8 GB ./HBase-1.2-JDK8
> 8 GB ./HBase-1.5
> 7 GB ./Atlas-master-UnitTests
> 7 GB ./tika-2.x-windows
> 7 GB ./incubator-rya-master-with-optionals-pull-requests
> 7 GB ./Hive-trunk
> 7 GB ./beam_PreCommit_Java_Cron
> 7 GB ./Atlas-1.0-UnitTests
> 6 GB ./Jackrabbit
> 6 GB ./beam_PostCommit_Java_PVR_Flink_PR
> 6 GB ./Lucene-Solr-Clover-master
> 6 GB ./Syncope-2_0_X-deploy
> 6 GB ./beam_PostCommit_Java_ValidatesRunner_Apex
> 6 GB ./Tika-trunk
> 6 GB ./pirk
> 6 GB ./Syncope-2_1_X-deploy
> 6 GB ./PLC4X
> 6 GB ./myfaces-current-2.0-integration-tests
> 5 GB ./commons-lang
> 5 GB ./Nemo
> 5 GB ./Mesos-Buildbot
> 5 GB ./Qpid-Broker-J-7.1.x-TestMatrix
> 5 GB ./beam_PostCommit_Java_Nexmark_Flink
> 5 GB ./Qpid-Broker-J-TestMatrix
> 5 GB ./ZooKeeper-Hammer
> 5 GB ./Camel
> 5 GB ./Royale
> 5 GB ./tika-branch-1x
> 5 GB ./ManifoldCF-ant
> 5 GB ./PreCommit-SQOOP-Build
> 5 GB ./HBase-1.4
> 5 GB ./ZooKeeper_UT_Stress
> 4 GB