[ https://issues.apache.org/jira/browse/HDDS-1486?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aravindan Vijayan resolved HDDS-1486. ------------------------------------- Resolution: Cannot Reproduce > Ozone write fails in allocateBlock while writing >10MB files in multiple > threads. > --------------------------------------------------------------------------------- > > Key: HDDS-1486 > URL: https://issues.apache.org/jira/browse/HDDS-1486 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Reporter: Aravindan Vijayan > Priority: Major > Labels: intermittent > Attachments: Datanode Logs.zip > > > 15 node physical cluster. All Datanodes are up and running. > Client using 16 threads attempting to write 16000 x 10MB+ files using the > FsStress utility > (https://github.com/arp7/FsPerfTest) fails with the following error. > This is an intermittent issue. > *Server side exceptions* > {code} > 19/04/22 10:13:32 ERROR io.KeyOutputStream: Try to allocate more blocks for > write failed, already allocated 0 blocks for this write. > 19/04/18 14:33:23 WARN io.KeyOutputStream: Encountered exception > java.io.IOException: Unexpected Storage Container Exception: > java.util.concurrent.CompletionException: > java.util.concurrent.CompletionException: > org.apache.ratis.protocol.AlreadyClosedException: SlidingWindow$Client > client-ADE7F801D3AD->RAFT is closed.. The last committed block length is 0, > uncommitted data length is 10485760 retry count 0 > {code} > *Client side exceptions* > {code} > FAILED org.apache.ratis.protocol.NotLeaderException: Server > c6e64cc4-91e9-4b36-83e4-6d84a4e71b7f is not the leader > (f44c1413-0847-45e3-982d-ac3aec15dffc:10.17.200.23:9858). Request must be > sent to leader., logIndex=0, > commits[c6e64cc4-91e9-4b36-83e4-6d84a4e71b7f:c131161, > 287eccfb-8461-419a-8732-529d042380b3:c131161, > f44c1413-0847-45e3-982d-ac3aec15dffc:c131161] > {code} > In the case of small key sizes (<1MB) and big key sizes with single thread, > the above client side exceptions are infrequent. However, in the case of > multithreaded 10MB+ size keys, the exceptions occur about 50% of the time and > eventually cause write failures. I have attached one such failed pipeline > logs. > [^Datanode Logs.zip] -- This message was sent by Atlassian JIRA (v7.6.14#76016) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org