Re: commit log on NFS volume

2016-11-01 Thread Benjamin Roth 
The Second exception states there is a File of an sstable Missing. Is it
possible you didnt only delete commit logs or nfs Mount is stale or Not
mounted?

Am 01.11.2016 19:52 schrieb "John Sanda" :

> Using nfs for a distribited System like Cassandra is like putting a
>> Ferrari on a Truck and going for a Race with the Truck. It is simply
>> nonsense.
>
>
> As I mentioned in my original post, I am aware that using NFS is
> considered bad and even documented as an anti-pattern. Your analogy,
> interesting as it may be, is not helpful. It simply restating what has
> already been said. I don't even know that NFS is to blame for the
> CommitLogReplayException that I cited.
>
> On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth 
> wrote:
>
>> Using nfs for a distribited System like Cassandra is like putting a
>> Ferrari on a Truck and going for a Race with the Truck. It is simply
>> nonsense.
>>
>> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" :
>>
>>> Hi,
>>>
>>> it's not only performance issue. In case of network problem writer tread
>>> can be blocked, also in case of failure loss of data can occur.
>>>
>>> Best regards, Vladimir Yudovin,
>>>
>>> *Winguzone  - Hosted Cloud
>>> CassandraLaunch your cluster in minutes.*
>>>
>>>
>>>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda
>>> >* wrote 
>>>
>>> I know that using NFS is discouraged, particularly for the commit log.
>>> Can anyone shed some light into what kinds of problems I might encounter
>>> aside from performance? The reason for my inquiry is because I have some
>>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>>> problems like reoccurring corrupted commit log segments on start up:
>>>
>>> ERROR 19:38:42 Exiting due to error while processing commit log during 
>>> initialization.
>>> 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
>>>  Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
>>> [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
>>> [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>>  [apache-cassandra-2.2.1.jar]
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
>>> [apache-cassandra-2.2.1.jar]
>>>
>>>
>>> In one deployment after removing all of corrupted commit log segments I
>>> got a different error:
>>>
>>> Exception (java.lang.RuntimeException) encountered during startup: 
>>> java.nio.file.NoSuchFileException: 
>>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
>>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> at 
>>> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>>> at 
>>> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>>> at 
>>> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>>> at 
>>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>>> at 
>>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>>> at 
>>> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>> at 
>>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
>>> Caused by: java.nio.file.NoSuchFileException: 
>>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>> at 
>>>

Re: commit log on NFS volume

2016-11-01 Thread John Sanda 
>
> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.


As I mentioned in my original post, I am aware that using NFS is considered
bad and even documented as an anti-pattern. Your analogy, interesting as it
may be, is not helpful. It simply restating what has already been said. I
don't even know that NFS is to blame for the CommitLogReplayException that
I cited.

On Tue, Nov 1, 2016 at 2:43 PM, Benjamin Roth 
wrote:

> Using nfs for a distribited System like Cassandra is like putting a
> Ferrari on a Truck and going for a Race with the Truck. It is simply
> nonsense.
>
> Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" :
>
>> Hi,
>>
>> it's not only performance issue. In case of network problem writer tread
>> can be blocked, also in case of failure loss of data can occur.
>>
>> Best regards, Vladimir Yudovin,
>>
>> *Winguzone  - Hosted Cloud
>> CassandraLaunch your cluster in minutes.*
>>
>>
>>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda > >* wrote 
>>
>> I know that using NFS is discouraged, particularly for the commit log.
>> Can anyone shed some light into what kinds of problems I might encounter
>> aside from performance? The reason for my inquiry is because I have some
>> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
>> problems like reoccurring corrupted commit log segments on start up:
>>
>> ERROR 19:38:42 Exiting due to error while processing commit log during 
>> initialization.
>> 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException:
>>  Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
>> [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>  [apache-cassandra-2.2.1.jar]
>> at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
>> [apache-cassandra-2.2.1.jar]
>>
>>
>> In one deployment after removing all of corrupted commit log segments I
>> got a different error:
>>
>> Exception (java.lang.RuntimeException) encountered during startup: 
>> java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>  at 
>> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>>  at 
>> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>>  at 
>> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>>  at 
>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>>  at 
>> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>>  at 
>> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>>  at 
>> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
>> Caused by: java.nio.file.NoSuchFileException: 
>> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>>  at 
>> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>>  at 
>> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>>

Re: commit log on NFS volume

2016-11-01 Thread Benjamin Roth 
Using nfs for a distribited System like Cassandra is like putting a Ferrari
on a Truck and going for a Race with the Truck. It is simply nonsense.

Am 01.11.2016 19:39 schrieb "Vladimir Yudovin" :

> Hi,
>
> it's not only performance issue. In case of network problem writer tread
> can be blocked, also in case of failure loss of data can occur.
>
> Best regards, Vladimir Yudovin,
>
> *Winguzone  - Hosted Cloud
> CassandraLaunch your cluster in minutes.*
>
>
>  On Tue, 01 Nov 2016 14:10:10 -0400*John Sanda  >* wrote 
>
> I know that using NFS is discouraged, particularly for the commit log. Can
> anyone shed some light into what kinds of problems I might encounter aside
> from performance? The reason for my inquiry is because I have some
> deployments with Cassandra 2.2.1 that use NFS and are experiencing some
> problems like reoccurring corrupted commit log segments on start up:
>
> ERROR 19:38:42 Exiting due to error while processing commit log during 
> initialization.
> 
> org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
> Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
> [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
> [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
> [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>  [apache-cassandra-2.2.1.jar]
> at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
> [apache-cassandra-2.2.1.jar]
>
>
> In one deployment after removing all of corrupted commit log segments I
> got a different error:
>
> Exception (java.lang.RuntimeException) encountered during startup: 
> java.nio.file.NoSuchFileException: 
> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
> java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>   at 
> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55)
>   at 
> org.apache.cassandra.io.util.ChannelProxy.(ChannelProxy.java:66)
>   at 
> org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
>   at 
> org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
>   at 
> org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
>   at 
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216)
>   at 
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488)
>   at 
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595)
> Caused by: java.nio.file.NoSuchFileException: 
> /cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
>   at 
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>   at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>   at 
> sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
>   at java.nio.channels.FileChannel.open(FileChannel.java:287)
>   at java.nio.channels.FileChannel.open(FileChannel.java:335)
>   at 
> org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51)
>   ... 8 more
>
>
> The latter error looks like it involves compaction and might be unrelated.
> I don't know if it matters, but I have commit log compression enabled in
> these environments.
>
> --
>
> - John
>
>
>

Re: commit log on NFS volume

2016-11-01 Thread Vladimir Yudovin 
Hi,



it's not only performance issue. In case of network problem writer tread can be 
blocked, also in case of failure loss of data can occur.



Best regards, Vladimir Yudovin, 

Winguzone - Hosted Cloud Cassandra
Launch your cluster in minutes.





 On Tue, 01 Nov 2016 14:10:10 -0400John Sanda john.sa...@gmail.com 
wrote 




I know that using NFS is discouraged, particularly for the commit log. Can 
anyone shed some light into what kinds of problems I might encounter aside from 
performance? The reason for my inquiry is because I have some deployments with 
Cassandra 2.2.1 that use NFS and are experiencing some problems like 
reoccurring corrupted commit log segments on start up:



 ERROR 19:38:42 Exiting due to error while processing commit log during 
initialization. 
org.apache.cassandra.db.commitlog.CommitLogReplayer$CommitLogReplayException: 
Mutation checksum failure at 33296351 in CommitLog-5-1474325237114.log at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.handleReplayError(CommitLogReplayer.java:622)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.replaySyncSection(CommitLogReplayer.java:492)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:388)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:147)
 [apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:189) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:169) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:266) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) 
[apache-cassandra-2.2.1.jar] at 
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
[apache-cassandra-2.2.1.jar]



In one deployment after removing all of corrupted commit log segments I got a 
different error:



Exception (java.lang.RuntimeException) encountered during startup: 
java.nio.file.NoSuchFileException: 
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
 java.lang.RuntimeException: java.nio.file.NoSuchFileException: 
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
 at org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:55) 
at org.apache.cassandra.io.util.ChannelProxy.init(ChannelProxy.java:66) 
at 
org.apache.cassandra.io.util.RandomAccessReader.open(RandomAccessReader.java:78)
 at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:91)
 at 
org.apache.cassandra.io.sstable.metadata.MetadataSerializer.deserialize(MetadataSerializer.java:101)
 at 
org.apache.cassandra.db.ColumnFamilyStore.removeUnfinishedCompactionLeftovers(ColumnFamilyStore.java:672)
 at 
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:216) at 
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:488) 
at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:595) 
Caused by: java.nio.file.NoSuchFileException: 
/cassandra_data/data/hawkular_metrics/metrics_idx-1ea881506ee311e6890b131c1bc89929/la-86-big-Statistics.db
 at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102) at 
sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107) at 
sun.nio.fs.UnixFileSystemProvider.newFileChannel(UnixFileSystemProvider.java:177)
 at java.nio.channels.FileChannel.open(FileChannel.java:287) at 
java.nio.channels.FileChannel.open(FileChannel.java:335) at 
org.apache.cassandra.io.util.ChannelProxy.openChannel(ChannelProxy.java:51) ... 
8 more



The latter error looks like it involves compaction and might be unrelated. I 
don't know if it matters, but I have commit log compression enabled in these 
environments.



-- 



- John