[jira] [Updated] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing

2022-08-12 Thread Chesnay Schepler (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chesnay Schepler updated FLINK-28263:
-
Issue Type: Technical Debt  (was: Bug)

> TPC-DS Bash e2e tests don't clean-up after completing
> -
>
> Key: FLINK-28263
> URL: https://issues.apache.org/jira/browse/FLINK-28263
> Project: Flink
>  Issue Type: Technical Debt
>  Components: Tests
>Affects Versions: 1.16.0
>Reporter: Martijn Visser
>Assignee: Martijn Visser
>Priority: Critical
>  Labels: pull-request-available, test-stability
> Fix For: 1.16.0, 1.15.2, 1.14.6
>
>
> When debugging the disk space usage for the e2e tests, the top 20 folders 
> with the largest file size are:
> {code:java}
> 2022-06-27T09:32:59.8000587Z Jun 27 09:32:59 List top 20 directories with 
> largest file size
> 2022-06-27T09:33:00.9811803Z Jun 27 09:33:00 4088524  .
> 2022-06-27T09:33:00.9813428Z Jun 27 09:33:00 1277080  ./flink-end-to-end-tests
> 2022-06-27T09:33:00.9814324Z Jun 27 09:33:00 624512   ./flink-dist
> 2022-06-27T09:33:00.9815152Z Jun 27 09:33:00 624124   ./flink-dist/target
> 2022-06-27T09:33:00.9816093Z Jun 27 09:33:00 500032   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:33:00.9817429Z Jun 27 09:33:00 500028   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:33:00.9818167Z Jun 27 09:33:00 486412   ./.git
> 2022-06-27T09:33:00.9819096Z Jun 27 09:33:00 479416   ./.git/objects
> 2022-06-27T09:33:00.9819512Z Jun 27 09:33:00 479408   ./.git/objects/pack
> 2022-06-27T09:33:00.9820584Z Jun 27 09:33:00 461456   ./flink-connectors
> 2022-06-27T09:33:00.9821403Z Jun 27 09:33:00 449832   
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:33:00.9821992Z Jun 27 09:33:00 349236   ./flink-table
> 2022-06-27T09:33:00.9822631Z Jun 27 09:33:00 293008   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:33:00.9823233Z Jun 27 09:33:00 251272   ./flink-filesystems
> 2022-06-27T09:33:00.9823818Z Jun 27 09:33:00 246588   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test
> 2022-06-27T09:33:00.9824502Z Jun 27 09:33:00 246464   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target
> 2022-06-27T09:33:00.9825210Z Jun 27 09:33:00 196656   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib
> 2022-06-27T09:33:00.9825966Z Jun 27 09:33:00 184364   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target/KinesisExample.jar
> 2022-06-27T09:33:00.9826652Z Jun 27 09:33:00 156136   
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:33:00.9827284Z Jun 27 09:33:00 151180   
> ./flink-end-to-end-tests/flink-tpcds-test/target
> {code}
> See 
> https://dev.azure.com/martijn0323/Flink/_build/results?buildId=2732=logs=0e31ee24-31a6-528c-a4bf-45cde9b2a14e=ff03a8fa-e84e-5199-efb2-5433077ce8e2=5093
> After running {{TPC-DS end-to-end test}} and after the clean-up, the 
> following directories are listed in the top 20:
> {code:java}
> 2022-06-27T09:49:51.7694429Z Jun 27 09:49:51 List top 20 directories with 
> largest file size AFTER cleaning temorary folders and files
> 2022-06-27T09:49:52.9617221Z Jun 27 09:49:52 5315996  .
> 2022-06-27T09:49:52.9618830Z Jun 27 09:49:52 2504556  ./flink-end-to-end-tests
> 2022-06-27T09:49:52.9619848Z Jun 27 09:49:52 1383612  
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:49:52.9620796Z Jun 27 09:49:52 1378656  
> ./flink-end-to-end-tests/flink-tpcds-test/target
> 2022-06-27T09:49:52.9621730Z Jun 27 09:49:52 1223944  
> ./flink-end-to-end-tests/flink-tpcds-test/target/table
> 2022-06-27T09:49:52.9622844Z Jun 27 09:49:52 624508   ./flink-dist
> 2022-06-27T09:49:52.9623585Z Jun 27 09:49:52 624120   ./flink-dist/target
> 2022-06-27T09:49:52.9624398Z Jun 27 09:49:52 500028   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:49:52.9625366Z Jun 27 09:49:52 500024   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:49:52.9625994Z Jun 27 09:49:52 486412   ./.git
> 2022-06-27T09:49:52.9626514Z Jun 27 09:49:52 479416   ./.git/objects
> 2022-06-27T09:49:52.9631740Z Jun 27 09:49:52 479408   ./.git/objects/pack
> 2022-06-27T09:49:52.9632755Z Jun 27 09:49:52 461456   ./flink-connectors
> 2022-06-27T09:49:52.9633717Z Jun 27 09:49:52 449832   
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:49:52.9634769Z Jun 27 09:49:52 379348   
> ./flink-end-to-end-tests/flink-tpcds-test/target/table/store_sales.dat
> 2022-06-27T09:49:52.9635596Z Jun 27 09:49:52 349236   ./flink-table
> 2022-06-27T09:49:52.9636489Z Jun 27 09:49:52 293008   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:49:52.9637526Z

[jira] [Updated] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing

2022-06-28 Thread ASF GitHub Bot (Jira) 
Title: Message Title


 
 
 
 

 
 
 

 
   
 ASF GitHub Bot updated an issue  
 

  
 
 
 
 

 
 
  
 
 
 
 

 
 Flink /  FLINK-28263  
 
 
  TPC-DS Bash e2e tests don't clean-up after completing   
 

  
 
 
 
 

 
Change By: 
 ASF GitHub Bot  
 
 
Labels: 
 pull-request-available test-stability  
 

  
 
 
 
 

 
 
 

 
 
 Add Comment  
 

  
 

  
 
 
 
  
 

  
 
 
 
 

 
 This message was sent by Atlassian Jira (v8.20.10#820010-sha1:ace47f9)

[jira] [Updated] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing

2022-06-27 Thread Martijn Visser (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Martijn Visser updated FLINK-28263:
---
Labels: test-stability  (was: pull-request-available)

> TPC-DS Bash e2e tests don't clean-up after completing
> -
>
> Key: FLINK-28263
> URL: https://issues.apache.org/jira/browse/FLINK-28263
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.16.0
>Reporter: Martijn Visser
>Priority: Critical
>  Labels: test-stability
>
> When debugging the disk space usage for the e2e tests, the top 20 folders 
> with the largest file size are:
> {code:java}
> 2022-06-27T09:32:59.8000587Z Jun 27 09:32:59 List top 20 directories with 
> largest file size
> 2022-06-27T09:33:00.9811803Z Jun 27 09:33:00 4088524  .
> 2022-06-27T09:33:00.9813428Z Jun 27 09:33:00 1277080  ./flink-end-to-end-tests
> 2022-06-27T09:33:00.9814324Z Jun 27 09:33:00 624512   ./flink-dist
> 2022-06-27T09:33:00.9815152Z Jun 27 09:33:00 624124   ./flink-dist/target
> 2022-06-27T09:33:00.9816093Z Jun 27 09:33:00 500032   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:33:00.9817429Z Jun 27 09:33:00 500028   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:33:00.9818167Z Jun 27 09:33:00 486412   ./.git
> 2022-06-27T09:33:00.9819096Z Jun 27 09:33:00 479416   ./.git/objects
> 2022-06-27T09:33:00.9819512Z Jun 27 09:33:00 479408   ./.git/objects/pack
> 2022-06-27T09:33:00.9820584Z Jun 27 09:33:00 461456   ./flink-connectors
> 2022-06-27T09:33:00.9821403Z Jun 27 09:33:00 449832   
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:33:00.9821992Z Jun 27 09:33:00 349236   ./flink-table
> 2022-06-27T09:33:00.9822631Z Jun 27 09:33:00 293008   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:33:00.9823233Z Jun 27 09:33:00 251272   ./flink-filesystems
> 2022-06-27T09:33:00.9823818Z Jun 27 09:33:00 246588   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test
> 2022-06-27T09:33:00.9824502Z Jun 27 09:33:00 246464   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target
> 2022-06-27T09:33:00.9825210Z Jun 27 09:33:00 196656   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib
> 2022-06-27T09:33:00.9825966Z Jun 27 09:33:00 184364   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target/KinesisExample.jar
> 2022-06-27T09:33:00.9826652Z Jun 27 09:33:00 156136   
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:33:00.9827284Z Jun 27 09:33:00 151180   
> ./flink-end-to-end-tests/flink-tpcds-test/target
> {code}
> See 
> https://dev.azure.com/martijn0323/Flink/_build/results?buildId=2732=logs=0e31ee24-31a6-528c-a4bf-45cde9b2a14e=ff03a8fa-e84e-5199-efb2-5433077ce8e2=5093
> After running {{TPC-DS end-to-end test}} and after the clean-up, the 
> following directories are listed in the top 20:
> {code:java}
> 2022-06-27T09:49:51.7694429Z Jun 27 09:49:51 List top 20 directories with 
> largest file size AFTER cleaning temorary folders and files
> 2022-06-27T09:49:52.9617221Z Jun 27 09:49:52 5315996  .
> 2022-06-27T09:49:52.9618830Z Jun 27 09:49:52 2504556  ./flink-end-to-end-tests
> 2022-06-27T09:49:52.9619848Z Jun 27 09:49:52 1383612  
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:49:52.9620796Z Jun 27 09:49:52 1378656  
> ./flink-end-to-end-tests/flink-tpcds-test/target
> 2022-06-27T09:49:52.9621730Z Jun 27 09:49:52 1223944  
> ./flink-end-to-end-tests/flink-tpcds-test/target/table
> 2022-06-27T09:49:52.9622844Z Jun 27 09:49:52 624508   ./flink-dist
> 2022-06-27T09:49:52.9623585Z Jun 27 09:49:52 624120   ./flink-dist/target
> 2022-06-27T09:49:52.9624398Z Jun 27 09:49:52 500028   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:49:52.9625366Z Jun 27 09:49:52 500024   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:49:52.9625994Z Jun 27 09:49:52 486412   ./.git
> 2022-06-27T09:49:52.9626514Z Jun 27 09:49:52 479416   ./.git/objects
> 2022-06-27T09:49:52.9631740Z Jun 27 09:49:52 479408   ./.git/objects/pack
> 2022-06-27T09:49:52.9632755Z Jun 27 09:49:52 461456   ./flink-connectors
> 2022-06-27T09:49:52.9633717Z Jun 27 09:49:52 449832   
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:49:52.9634769Z Jun 27 09:49:52 379348   
> ./flink-end-to-end-tests/flink-tpcds-test/target/table/store_sales.dat
> 2022-06-27T09:49:52.9635596Z Jun 27 09:49:52 349236   ./flink-table
> 2022-06-27T09:49:52.9636489Z Jun 27 09:49:52 293008   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:49:52.9637526Z Jun 27 09:49:52 288980   
> ./flink-end-to-end-tests/flink-tpcds-test/target/table/catalog_sales.dat
>

[jira] [Updated] (FLINK-28263) TPC-DS Bash e2e tests don't clean-up after completing

2022-06-27 Thread ASF GitHub Bot (Jira) 


 [ 
https://issues.apache.org/jira/browse/FLINK-28263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

ASF GitHub Bot updated FLINK-28263:
---
Labels: pull-request-available  (was: )

> TPC-DS Bash e2e tests don't clean-up after completing
> -
>
> Key: FLINK-28263
> URL: https://issues.apache.org/jira/browse/FLINK-28263
> Project: Flink
>  Issue Type: Bug
>  Components: Tests
>Affects Versions: 1.16.0
>Reporter: Martijn Visser
>Priority: Critical
>  Labels: pull-request-available
>
> When debugging the disk space usage for the e2e tests, the top 20 folders 
> with the largest file size are:
> {code:java}
> 2022-06-27T09:32:59.8000587Z Jun 27 09:32:59 List top 20 directories with 
> largest file size
> 2022-06-27T09:33:00.9811803Z Jun 27 09:33:00 4088524  .
> 2022-06-27T09:33:00.9813428Z Jun 27 09:33:00 1277080  ./flink-end-to-end-tests
> 2022-06-27T09:33:00.9814324Z Jun 27 09:33:00 624512   ./flink-dist
> 2022-06-27T09:33:00.9815152Z Jun 27 09:33:00 624124   ./flink-dist/target
> 2022-06-27T09:33:00.9816093Z Jun 27 09:33:00 500032   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:33:00.9817429Z Jun 27 09:33:00 500028   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:33:00.9818167Z Jun 27 09:33:00 486412   ./.git
> 2022-06-27T09:33:00.9819096Z Jun 27 09:33:00 479416   ./.git/objects
> 2022-06-27T09:33:00.9819512Z Jun 27 09:33:00 479408   ./.git/objects/pack
> 2022-06-27T09:33:00.9820584Z Jun 27 09:33:00 461456   ./flink-connectors
> 2022-06-27T09:33:00.9821403Z Jun 27 09:33:00 449832   
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:33:00.9821992Z Jun 27 09:33:00 349236   ./flink-table
> 2022-06-27T09:33:00.9822631Z Jun 27 09:33:00 293008   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:33:00.9823233Z Jun 27 09:33:00 251272   ./flink-filesystems
> 2022-06-27T09:33:00.9823818Z Jun 27 09:33:00 246588   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test
> 2022-06-27T09:33:00.9824502Z Jun 27 09:33:00 246464   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target
> 2022-06-27T09:33:00.9825210Z Jun 27 09:33:00 196656   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/lib
> 2022-06-27T09:33:00.9825966Z Jun 27 09:33:00 184364   
> ./flink-end-to-end-tests/flink-streaming-kinesis-test/target/KinesisExample.jar
> 2022-06-27T09:33:00.9826652Z Jun 27 09:33:00 156136   
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:33:00.9827284Z Jun 27 09:33:00 151180   
> ./flink-end-to-end-tests/flink-tpcds-test/target
> {code}
> See 
> https://dev.azure.com/martijn0323/Flink/_build/results?buildId=2732=logs=0e31ee24-31a6-528c-a4bf-45cde9b2a14e=ff03a8fa-e84e-5199-efb2-5433077ce8e2=5093
> After running {{TPC-DS end-to-end test}} and after the clean-up, the 
> following directories are listed in the top 20:
> {code:java}
> 2022-06-27T09:49:51.7694429Z Jun 27 09:49:51 List top 20 directories with 
> largest file size AFTER cleaning temorary folders and files
> 2022-06-27T09:49:52.9617221Z Jun 27 09:49:52 5315996  .
> 2022-06-27T09:49:52.9618830Z Jun 27 09:49:52 2504556  ./flink-end-to-end-tests
> 2022-06-27T09:49:52.9619848Z Jun 27 09:49:52 1383612  
> ./flink-end-to-end-tests/flink-tpcds-test
> 2022-06-27T09:49:52.9620796Z Jun 27 09:49:52 1378656  
> ./flink-end-to-end-tests/flink-tpcds-test/target
> 2022-06-27T09:49:52.9621730Z Jun 27 09:49:52 1223944  
> ./flink-end-to-end-tests/flink-tpcds-test/target/table
> 2022-06-27T09:49:52.9622844Z Jun 27 09:49:52 624508   ./flink-dist
> 2022-06-27T09:49:52.9623585Z Jun 27 09:49:52 624120   ./flink-dist/target
> 2022-06-27T09:49:52.9624398Z Jun 27 09:49:52 500028   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin
> 2022-06-27T09:49:52.9625366Z Jun 27 09:49:52 500024   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT
> 2022-06-27T09:49:52.9625994Z Jun 27 09:49:52 486412   ./.git
> 2022-06-27T09:49:52.9626514Z Jun 27 09:49:52 479416   ./.git/objects
> 2022-06-27T09:49:52.9631740Z Jun 27 09:49:52 479408   ./.git/objects/pack
> 2022-06-27T09:49:52.9632755Z Jun 27 09:49:52 461456   ./flink-connectors
> 2022-06-27T09:49:52.9633717Z Jun 27 09:49:52 449832   
> ./.git/objects/pack/pack-0bdd9e3186d0cb404910c5843d19b5cb80b84fe0.pack
> 2022-06-27T09:49:52.9634769Z Jun 27 09:49:52 379348   
> ./flink-end-to-end-tests/flink-tpcds-test/target/table/store_sales.dat
> 2022-06-27T09:49:52.9635596Z Jun 27 09:49:52 349236   ./flink-table
> 2022-06-27T09:49:52.9636489Z Jun 27 09:49:52 293008   
> ./flink-dist/target/flink-1.16-SNAPSHOT-bin/flink-1.16-SNAPSHOT/opt
> 2022-06-27T09:49:52.9637526Z Jun 27 09:49:52 288980   
> ./flink-end-to-end-tests/flink-tpcds-test/target/table/catalog_sales.dat
>