Re: Re[2]: Failed to write class name to file
Hello! In newer versions Ignite will never use /tmp by default. Regards, -- Ilya Kasnacheev ср, 31 мар. 2021 г. в 13:26, Matej Kováč : > Thank you Zhenya, > it seems that the problem was caused by the fact that centos deleted the > whole ignite work directory as it was unaccessed for 10 days. > Ignite probably creates the workdir only when starting, but the > application did not restart during 2 weeks. It seems that Ignite failed to > serialize class because it assumed that the marshaller folder still exists. > > The solution is to never use /tmp for ignite workdir (or disable > tmpfiles.d). > > On Wed, 2021-03-31 at 09:01 +0300, Zhenya Stanilovsky wrote: > > > what output > of: stat /tmp/ignite-workspace/marshaller/-961185899.classname0 ? > If it`s all ok here, possibly some privileges issues ? > > > Hello, > > we are using CentOS Linux release 7.8.2003 (Core), which deletes files > from /tmp no sooner than 10 days after accessed, so this should not be an > issue. > > > On Tue, 2021-03-30 at 18:40 +0300, Zhenya Stanilovsky wrote: > > hi, > https://askubuntu.com/questions/20783/how-is-the-tmp-directory-cleaned-up > ? > > > > Hi, > > We use a cluster consisting of 2 Ignite nodes. Deployment is done without > downtime or cluster reset. > > When attempted to deploy new version of an application contaning new > caches on node 1, Ignite fails to write the class file on node 2. Peer > class loading is enabled. > The directory does exist and already contains other serialized classes and > has proper permissions. > > Somehow Ignite fails to open a file it has just created, with > FileNotFoundException: *No such file or directory* > Do you have any suggestions what could be the cause? > > Thank you, > Matej > > > [ERROR] [29.03.2021 14:34:09.686] [] [68.166.6:47500]-#2] > [i.i.MarshallerMappingFileStore]: Failed to write class name to file > [platformId=0id=-961185899, clsName=org.profile.AbTest, > file=/tmp/ignite-workspace/marshaller/-961185899.classname0] > java.io.FileNotFoundException: > /tmp/ignite-workspace/marshaller/-961185899.classname0 (No such file or > directory) > at java.base/java.io.FileOutputStream.open0(Native Method) > at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) > at java.base/java.io.FileOutputStream.(FileOutputStream.java:237) > at java.base/java.io.FileOutputStream.(FileOutputStream.java:187) > at > org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) > at > org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) > at > org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) > at > org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) > at > org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336) > at > org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:906) > at > org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:2090) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4816) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3089) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2795) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7766) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) > at > org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) > at > org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) > at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) > [ERROR] [29.03.2021 14:34:09.687] [] [68.166.6:47500]-#2] > [i.i.MarshallerMappingFileStore]: Failed to write class name to file > [platformId=0id=-1710898632, clsName=org.profile.GridCard, > file=/tmp/ignite-customer2/marshaller/-1710898632.classname0] > java.io.FileNotFoundException: > /tmp/ignite-customer2/marshaller/-1710898632.classname0 (No such file or > directory) > at java.base/java.io.FileOutputStream.open0(Native Method) > at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) > at java.base/java.io.FileOutputStream.(FileOutputStream.java:237) > at java.base/java.io.FileOutputStream.(FileOutputStream.java:187) > at > org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) > at > org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) > at > org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.
Re: Re[2]: Failed to write class name to file
Thank you Zhenya, it seems that the problem was caused by the fact that centos deleted the whole ignite work directory as it was unaccessed for 10 days. Ignite probably creates the workdir only when starting, but the application did not restart during 2 weeks. It seems that Ignite failed to serialize class because it assumed that the marshaller folder still exists. The solution is to never use /tmp for ignite workdir (or disable tmpfiles.d). On Wed, 2021-03-31 at 09:01 +0300, Zhenya Stanilovsky wrote: what output of: stat /tmp/ignite-workspace/marshaller/-961185899.classname0 ? If it`s all ok here, possibly some privileges issues ? Hello, we are using CentOS Linux release 7.8.2003 (Core), which deletes files from /tmp no sooner than 10 days after accessed, so this should not be an issue. On Tue, 2021-03-30 at 18:40 +0300, Zhenya Stanilovsky wrote: hi, https://askubuntu.com/questions/20783/how-is-the-tmp-directory-cleaned-up ? Hi, We use a cluster consisting of 2 Ignite nodes. Deployment is done without downtime or cluster reset. When attempted to deploy new version of an application contaning new caches on node 1, Ignite fails to write the class file on node 2. Peer class loading is enabled. The directory does exist and already contains other serialized classes and has proper permissions. Somehow Ignite fails to open a file it has just created, with FileNotFoundException: No such file or directory Do you have any suggestions what could be the cause? Thank you, Matej [ERROR] [29.03.2021 14:34:09.686] [] [68.166.6:47500]-#2] [i.i.MarshallerMappingFileStore]: Failed to write class name to file [platformId=0id=-961185899, clsName=org.profile.AbTest, file=/tmp/ignite-workspace/marshaller/-961185899.classname0] java.io.FileNotFoundException: /tmp/ignite-workspace/marshaller/-961185899.classname0 (No such file or directory) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.(FileOutputStream.java:187) at org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) at org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) at org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336) at org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:906) at org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:2090) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4816) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3089) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2795) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7766) at org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) at org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) [ERROR] [29.03.2021 14:34:09.687] [] [68.166.6:47500]-#2] [i.i.MarshallerMappingFileStore]: Failed to write class name to file [platformId=0id=-1710898632, clsName=org.profile.GridCard, file=/tmp/ignite-customer2/marshaller/-1710898632.classname0] java.io.FileNotFoundException: /tmp/ignite-customer2/marshaller/-1710898632.classname0 (No such file or directory) at java.base/java.io.FileOutputStream.open0(Native Method) at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) at java.base/java.io.FileOutputStream.(FileOutputStream.java:237) at java.base/java.io.FileOutputStream.(FileOutputStream.java:187) at org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) at org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) at org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) at org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336)
Re[2]: Failed to write class name to file
what output of: stat /tmp/ignite-workspace/marshaller/-961185899.classname0 ? If it`s all ok here, possibly some privileges issues ? >Hello, > >we are using CentOS Linux release 7.8.2003 (Core), which deletes files from >/tmp no sooner than 10 days after accessed, so this should not be an issue. > > >On Tue, 2021-03-30 at 18:40 +0300, Zhenya Stanilovsky wrote: >>hi, >>https://askubuntu.com/questions/20783/how-is-the-tmp-directory-cleaned-up ? >> >> >>>Hi, >>> >>>We use a cluster consisting of 2 Ignite nodes. Deployment is done without >>>downtime or cluster reset. >>> >>>When attempted to deploy new version of an application contaning new caches >>>on node 1, Ignite fails to write the class file on node 2. Peer class >>>loading is enabled. >>>The directory does exist and already contains other serialized classes and >>>has proper permissions. >>> >>>Somehow Ignite fails to open a file it has just created, with >>>FileNotFoundException: No such file or directory >>>Do you have any suggestions what could be the cause? >>> >>>Thank you, >>>Matej >>> >>> >>>[ERROR] [29.03.2021 14:34:09.686] [] [68.166.6:47500]-#2] >>>[i.i.MarshallerMappingFileStore]: Failed to write class name to file >>>[platformId=0id=-961185899, clsName=org.profile.AbTest, >>>file=/tmp/ignite-workspace/marshaller/-961185899.classname0] >>>java.io.FileNotFoundException: >>>/tmp/ignite-workspace/marshaller/-961185899.classname0 (No such file or >>>directory) >>>at java.base/java.io.FileOutputStream.open0(Native Method) >>>at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) >>>at java.base/java.io.FileOutputStream.(FileOutputStream.java:237) >>>at java.base/java.io.FileOutputStream.(FileOutputStream.java:187) >>>at >>>org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) >>>at >>>org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) >>>at >>>org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) >>>at >>>org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) >>>at >>>org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336) >>>at >>>org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:906) >>>at >>>org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:2090) >>>at >>>org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processNodeAddedMessage(ServerImpl.java:4816) >>>at >>>org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:3089) >>>at >>>org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.processMessage(ServerImpl.java:2795) >>>at >>>org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorker.body(ServerImpl.java:7766) >>>at >>>org.apache.ignite.spi.discovery.tcp.ServerImpl$RingMessageWorker.body(ServerImpl.java:2946) >>>at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:120) >>>at >>>org.apache.ignite.spi.discovery.tcp.ServerImpl$MessageWorkerThread.body(ServerImpl.java:7697) >>>at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:61) >>>[ERROR] [29.03.2021 14:34:09.687] [] [68.166.6:47500]-#2] >>>[i.i.MarshallerMappingFileStore]: Failed to write class name to file >>>[platformId=0id=-1710898632, clsName=org.profile.GridCard, >>>file=/tmp/ignite-customer2/marshaller/-1710898632.classname0] >>>java.io.FileNotFoundException: >>>/tmp/ignite-customer2/marshaller/-1710898632.classname0 (No such file or >>>directory) >>>at java.base/java.io.FileOutputStream.open0(Native Method) >>>at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298) >>>at java.base/java.io.FileOutputStream.(FileOutputStream.java:237) >>>at java.base/java.io.FileOutputStream.(FileOutputStream.java:187) >>>at >>>org.apache.ignite.internal.MarshallerMappingFileStore.writeMapping(MarshallerMappingFileStore.java:97) >>>at >>>org.apache.ignite.internal.MarshallerMappingFileStore.mergeAndWriteMapping(MarshallerMappingFileStore.java:222) >>>at >>>org.apache.ignite.internal.MarshallerContextImpl.onMappingDataReceived(MarshallerContextImpl.java:191) >>>at >>>org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.processIncomingMappings(GridMarshallerMappingProcessor.java:356) >>>at >>>org.apache.ignite.internal.processors.marshaller.GridMarshallerMappingProcessor.onJoiningNodeDataReceived(GridMarshallerMappingProcessor.java:336) >>>at >>>org.apache.ignite.internal.managers.discovery.GridDiscoveryManager$5.onExchange(GridDiscoveryManager.java:906) >>>at >>>org.apache.ignite.spi.discovery.tcp.TcpDiscoverySpi.onExchange(TcpDiscoverySpi.java:2090) >>>at >>>org.apache.ig
