[ 
https://issues.apache.org/jira/browse/KAFKA-9603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bruno Cadonna closed KAFKA-9603.
--------------------------------

> Number of open files keeps increasing in Streams application
> ------------------------------------------------------------
>
>                 Key: KAFKA-9603
>                 URL: https://issues.apache.org/jira/browse/KAFKA-9603
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 2.1.0, 2.2.0, 2.4.0, 2.3.1
>         Environment: Spring Boot 2.2.4, OpenJDK 13, Centos image
>            Reporter: Bruno Iljazovic
>            Assignee: Bruno Cadonna
>            Priority: Major
>             Fix For: 2.6.0
>
>
> Problem appeared when upgrading from *2.0.1* to *2.3.1*. 
> Relevant Kafka Streams code:
> {code:java}
> KStream<String, Event1> events1 =
>     builder.stream(FIRST_TOPIC_NAME, Consumed.with(stringSerde, event1Serde, 
> event1TimestampExtractor(), null))
>            .mapValues(...);        
> KStream<String, Event2> events2 =
>     builder.stream(SECOND_TOPIC_NAME, Consumed.with(stringSerde, event2Serde, 
> event2TimestampExtractor(), null))
>            .mapValues(...);        
> var joinWindows = JoinWindows.of(Duration.of(1, MINUTES).toMillis())
>                              .until(Duration.of(1, HOURS).toMillis());
> events2.join(events1, this::join, joinWindows, Joined.with(stringSerde, 
> event2Serde, event1Serde))
>                .foreach(...);
> {code}
> Number of open *.sst files keeps increasing until eventually it hits the os 
> limit (65536) and causes this exception:
> {code:java}
> Caused by: org.rocksdb.RocksDBException: While open a file for appending: 
> /.../0_8/KSTREAM-JOINOTHER-0000000010-store/KSTREAM-JOINOTHER-0000000010-store.1579435200000/001354.sst:
>  Too many open files
>       at org.rocksdb.RocksDB.flush(Native Method)
>       at org.rocksdb.RocksDB.flush(RocksDB.java:2394)
> {code}
> Here are example files that are opened and never closed:
> {code:java}
> /.../0_27/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1582459200000/000114.sst
> /.../0_27/KSTREAM-JOINOTHER-0000000010-store/KSTREAM-JOINOTHER-0000000010-store.1582459200000/000065.sst
> /.../0_29/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1582156800000/000115.sst
> /.../0_29/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1582459200000/000112.sst
> /.../0_31/KSTREAM-JOINTHIS-0000000009-store/KSTREAM-JOINTHIS-0000000009-store.1581854400000/000051.sst
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Reply via email to