Re: [EXTERNAL] Re: cold vs hot data

[Alaa Zubaidi (PDF)]

This is one of the options that we are thinking of, but this will require
more storage, which is something that we are trying to avoid.
We will test the performance for the batch inserts.
Thanks

On Tue, Sep 18, 2018 at 6:35 AM, Durity, Sean R  wrote:

> Wouldn’t you have the same problem with two similar tables with different
> primary keys (eg., UserByID and UserByName)? This is a very common pattern
> in Cassandra – inserting into multiple tables… That’s what batches are for
> – atomicity.
>
> I don’t understand the additional concern here.
>
>
>
>
>
>
>
> Sean Durity
>
>
>
> *From:* DuyHai Doan 
> *Sent:* Monday, September 17, 2018 4:23 PM
> *To:* user 
> *Subject:* Re: [EXTERNAL] Re: cold vs hot data
>
>
>
> Sean
>
>
>
> Without transactions à la SQL, how can you guarantee atomicity between
> both tables for upserts ? I mean, one write could succeed with hot table
> and fail for cold table
>
>
>
> The only solution I see is using logged batch, with a huge overhead and
> perf hit on for the writes
>
>
>
> On Mon, Sep 17, 2018 at 8:28 PM, Durity, Sean R <
> sean_r_dur...@homedepot.com> wrote:
>
> An idea:
>
> On initial insert, insert into 2 tables:
> Hot with short TTL
> Cold/archive with a longer (or no) TTL
> Then your hot data is always in the same table, but being expired. And you
> can access the archive table only for the more rare circumstances. Then you
> could have the HOT table on a different volume of faster storage. If the
> hot/cold tables are in different keyspaces, then you could also have
> different replication (a HOT DC and an archive DC, for example)
>
>
> Sean Durity
>
>
>
> -Original Message-
> From: Mateusz 
> Sent: Friday, September 14, 2018 2:40 AM
> To: user@cassandra.apache.org
> Subject: [EXTERNAL] Re: cold vs hot data
>
> On piątek, 14 września 2018 02:46:43 CEST Alaa Zubaidi (PDF) wrote:
> > The data can grow to +100TB however the hot data will be in most cases
> > less than 10TB but we still need to keep the rest of data accessible.
> > Anyone has this problem?
> > What is the best way to make the cluster more efficient?
> > Is there a way to somehow automatically move the old data to different
> > storage (rack, dc, etc)?
> > Any ideas?
>
> We solved it using lvmcache.
>
> --
> Mateusz
> (...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
> krótko mówiąc - podpora społeczeństwa."
> Nikos Kazantzakis - "Grek Zorba"
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
> 
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and liability for the accuracy and content of
> this attachment and for any damages or losses arising from any
> inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other
> items of a destructive nature, which may be contained in this attachment
> and shall not be liable for direct, indirect, consequential or special
> damages in connection with this e-mail message or its attachment.
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>
>
> --
>
> The information in this Internet Email is confidential and may be legally
> privileged. It is intended solely for the addressee. Access to this Email
> by anyone else is unauthorized. If you are not the intended recipient, any
> disclosure, copying, distribution or any action taken or omitted to be
> taken in reliance on it, is prohibited and may be unlawful. When addressed
> to our clients any opinions or advice contained in this Email are subject
> to the terms and conditions expressed in any applicable governing The Home
> Depot terms of business or client engagement letter. The Home Depot
> disclaims all responsibility and li

Re: cold vs hot data

[Alaa Zubaidi (PDF)]

Let me check lvmcache..Thanks

On Thu, Sep 13, 2018 at 11:39 PM, Mateusz 
wrote:

> On piątek, 14 września 2018 02:46:43 CEST Alaa Zubaidi (PDF) wrote:
> > The data can grow to +100TB however the hot data will be in most cases
> less
> > than 10TB but we still need to keep the rest of data accessible.
> > Anyone has this problem?
> > What is the best way to make the cluster more efficient?
> > Is there a way to somehow automatically move the old data to different
> > storage (rack, dc, etc)?
> > Any ideas?
>
> We solved it using lvmcache.
>
> --
> Mateusz
> (...) mam brata - poważny, domator, liczykrupa, hipokryta, pobożniś,
> krótko mówiąc - podpora społeczeństwa."
> Nikos Kazantzakis - "Grek Zorba"
>
>
>
>
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
This message may contain confidential and privileged information. If it has 
been sent to you in error, please reply to advise the sender of the error 
and then immediately permanently delete it and all attachments to it from 
your systems. If you are not the intended recipient, do not read, copy, 
disclose or otherwise use this message or any attachments to it. The sender 
disclaims any liability for such unauthorized use.  PLEASE NOTE that all 
incoming e-mails sent to PDF e-mail accounts will be archived and may be 
scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”).  If you have any 
concerns about this process, please contact us at legal.departm...@pdf.com 
<mailto:legal.departm...@pdf.com>.

Re: cold vs hot data

[Alaa Zubaidi (PDF)]

Thanks Ben, I will try this on 4.0 when its available.


On Thu, Sep 13, 2018 at 7:06 PM, Ben Slater 
wrote:

> Not quite a solution but you will probably be interested in the discussion
> on this ticket: https://issues.apache.org/jira/browse/CASSANDRA-8460
>
> On Fri, 14 Sep 2018 at 10:46 Alaa Zubaidi (PDF) 
> wrote:
>
>> Hi,
>>
>> We are using Apache Cassandra 3.11.2 on RedHat 7
>> The data can grow to +100TB however the hot data will be in most cases
>> less than 10TB but we still need to keep the rest of data accessible.
>> Anyone has this problem?
>> What is the best way to make the cluster more efficient?
>> Is there a way to somehow automatically move the old data to different
>> storage (rack, dc, etc)?
>> Any ideas?
>>
>> Regards,
>>
>> --
>>
>> Alaa
>>
>>
>> *This message may contain confidential and privileged information. If it
>> has been sent to you in error, please reply to advise the sender of the
>> error and then immediately permanently delete it and all attachments to it
>> from your systems. If you are not the intended recipient, do not read,
>> copy, disclose or otherwise use this message or any attachments to it. The
>> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
>> all incoming e-mails sent to PDF e-mail accounts will be archived and may
>> be scanned by us and/or by external service providers to detect and prevent
>> threats to our systems, investigate illegal or inappropriate behavior,
>> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
>> concerns about this process, please contact us at *
>> *legal.departm...@pdf.com* *.*
>
> --
>
>
> *Ben Slater*
>
> *Chief Product Officer <https://www.instaclustr.com/>*
>
> <https://www.facebook.com/instaclustr>   <https://twitter.com/instaclustr>
><https://www.linkedin.com/company/instaclustr>
>
> Read our latest technical blog posts here
> <https://www.instaclustr.com/blog/>.
>
> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
> and Instaclustr Inc (USA).
>
> This email and any attachments may contain confidential and legally
> privileged information.  If you are not the intended recipient, do not copy
> or disclose its content, but please reply to this email immediately and
> highlight the error to the sender and then immediately delete the message.
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
This message may contain confidential and privileged information. If it has 
been sent to you in error, please reply to advise the sender of the error 
and then immediately permanently delete it and all attachments to it from 
your systems. If you are not the intended recipient, do not read, copy, 
disclose or otherwise use this message or any attachments to it. The sender 
disclaims any liability for such unauthorized use.  PLEASE NOTE that all 
incoming e-mails sent to PDF e-mail accounts will be archived and may be 
scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”).  If you have any 
concerns about this process, please contact us at legal.departm...@pdf.com 
<mailto:legal.departm...@pdf.com>.

cold vs hot data

[Alaa Zubaidi (PDF)]

Hi,

We are using Apache Cassandra 3.11.2 on RedHat 7
The data can grow to +100TB however the hot data will be in most cases less
than 10TB but we still need to keep the rest of data accessible.
Anyone has this problem?
What is the best way to make the cluster more efficient?
Is there a way to somehow automatically move the old data to different
storage (rack, dc, etc)?
Any ideas?

Regards,

-- 

Alaa

-- 
This message may contain confidential and privileged information. If it has 
been sent to you in error, please reply to advise the sender of the error 
and then immediately permanently delete it and all attachments to it from 
your systems. If you are not the intended recipient, do not read, copy, 
disclose or otherwise use this message or any attachments to it. The sender 
disclaims any liability for such unauthorized use.  PLEASE NOTE that all 
incoming e-mails sent to PDF e-mail accounts will be archived and may be 
scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”).  If you have any 
concerns about this process, please contact us at legal.departm...@pdf.com 
.

Re: overhead of empty tables

[Alaa Zubaidi (PDF)]

Thanks Dinesh,
We have 36332 files under the data folder

On Thu, Feb 15, 2018 at 3:45 PM, Dinesh Joshi <
dinesh.jo...@yahoo.com.invalid> wrote:

> Each table in a keyspace is stored as a separate directory in the data
> directory. If you many tables you'll have a lot of files. Some file systems
> have issues dealing with a lot of files in a single directory. Other than
> that, there will likely be some book keeping overhead within the Cassandra
> process. How many tables are we talking about here?
>
> Here's more information about it: https://docs.datastax.com/
> en/cassandra/3.0/cassandra/dml/dmlHowDataWritten.html
>
> Dinesh
>
>
> On Thursday, February 15, 2018, 3:34:49 PM PST, Alaa Zubaidi (PDF) <
> alaa.zuba...@pdf.com> wrote:
>
>
> Is there any overhead if my keyspace contains many empty tables?
> Thanks
> Alaa
>
> *This message may contain confidential and privileged information. If it
> has been sent to you in error, please reply to advise the sender of the
> error and then immediately permanently delete it and all attachments to it
> from your systems. If you are not the intended recipient, do not read,
> copy, disclose or otherwise use this message or any attachments to it. The
> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
> all incoming e-mails sent to PDF e-mail accounts will be archived and may
> be scanned by us and/or by external service providers to detect and prevent
> threats to our systems, investigate illegal or inappropriate behavior,
> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
> concerns about this process, please contact us at *
> *legal.departm...@pdf.com* *.*
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

overhead of empty tables

[Alaa Zubaidi (PDF)]

Is there any overhead if my keyspace contains many empty tables?
Thanks
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Migrate from Windows to Linux

[Alaa Zubaidi (PDF)]

Thanks Alain, yes, I saw some of these, but I was hoping that there are new
experiments.
This is useful, now I need to find new HW to test these options.
Thanks for your help..
Alaa

On Sat, Jan 27, 2018 at 9:04 AM, Alain RODRIGUEZ  wrote:

> Hello Alaa,
>
> Over time, people who tried seems to have failed to do a live migration
> from windows to linux (and probably the other way around). It appears to be
> something unsupported:
>
> http://grokbase.com/t/cassandra/user/13anrd7qv1/mixed-linux-
> windows-cluster-in-cassandra-1-2
> http://grokbase.com/t/cassandra/user/115vy2hy4w/mixing-
> different-os-in-a-cassandra-cluster
>
> Those discussions are a bit old. Yet if this incompatibility still true,
> you can probably:
>
> 1 - Stop the cluster and use sstable loader to load data in the new linux
> cluster
>
> or
>
> 2 - Fork writes to the new cluster, run the sstable loader then switch
> clients, apparently doable without downtime
>
> This 2 solutions were discussed there http://grokbase.com/t/cassandr
> a/user/125rhxydwb/migrating-from-a-windows-cluster-to-a-linux-cluster
>
> 3 - Try to run mixed cluster in a testing environment, check the Cassandra
> code, possibly make a ticket or offer a patch if it does not work. If doing
> so, add a new data center with linux, not mixed node within the same rack
> or data center. It's safer and more efficient. Even though in this specific
> case it might fail.
>
> C*heers,
> ---
> Alain Rodriguez - @arodream - al...@thelastpickle.com
> France / Spain
>
> The Last Pickle - Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> 2018-01-26 20:49 GMT+00:00 Alaa Zubaidi (PDF) :
>
>> Hi,
>> What is the best way to migrate my Cassandra 2.x cluster from Windows to
>> Linux?
>> - Can I mix Windows and linux nodes in the same cluster? bring linux node
>> one by one, and shutdown windows nodes one by one?
>> - Can I use multiple racks feature? how would this work?
>> - Other ideas?
>>
>> Regards.
>> Alaa
>>
>> *This message may contain confidential and privileged information. If it
>> has been sent to you in error, please reply to advise the sender of the
>> error and then immediately permanently delete it and all attachments to it
>> from your systems. If you are not the intended recipient, do not read,
>> copy, disclose or otherwise use this message or any attachments to it. The
>> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
>> all incoming e-mails sent to PDF e-mail accounts will be archived and may
>> be scanned by us and/or by external service providers to detect and prevent
>> threats to our systems, investigate illegal or inappropriate behavior,
>> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
>> concerns about this process, please contact us at *
>> *legal.departm...@pdf.com* *.*
>
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Migrate from Windows to Linux

[Alaa Zubaidi (PDF)]

Hi,
What is the best way to migrate my Cassandra 2.x cluster from Windows to
Linux?
- Can I mix Windows and linux nodes in the same cluster? bring linux node
one by one, and shutdown windows nodes one by one?
- Can I use multiple racks feature? how would this work?
- Other ideas?

Regards.
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Bring 2 nodes down

[Alaa Zubaidi (PDF)]

Hi,
I have a cluster of 8 Nodes, 4 physical machines 2 VMs each physical
machine.
RF=3, and we have read/write with QUORUM consistency requirement.

One of the machines needs to be down for an hour or two to fix local disk.
What is the best way to do that with out losing data?

Regards
-- Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Corrupt SSTABLE over and over

[Alaa Zubaidi (PDF)]

Hi Bryan,

Changing disk_failure_policy to best_effort, and running nodetool scrub,
did not work, it generated another error:
java.nio.file.AccessDeniedException

Also tried to remove all files (data, commitlog, savedcaches) and restart
the node fresh, and still I am getting corruption.

and Still nothing that indicate there is a HW issue?
All other nodes are fine

Regards,
Alaa


On Fri, Aug 12, 2016 at 12:00 PM, Bryan Cheng  wrote:

> Should also add that if the scope of corruption is _very_ large, and you
> have a good, aggressive repair policy (read: you are confident in the
> consistency of the data elsewhere in the cluster), you may just want to
> decommission and rebuild that node.
>
> On Fri, Aug 12, 2016 at 11:55 AM, Bryan Cheng 
> wrote:
>
>> Looks like you're doing the offline scrub- have you tried online?
>>
>> Here's my typical process for corrupt SSTables.
>>
>> With disk_failure_policy set to stop, examine the failing sstables. If
>> they are very small (in the range of kbs), it is unlikely that there is any
>> salvageable data there. Just delete them, start the machine, and schedule a
>> repair ASAP.
>>
>> If they are large, then it may be worth salvaging. If the scope of
>> corruption is reasonable (limited to a few sstables scattered among
>> different keyspaces), set disk_failure_policy to best_effort, start the
>> machine up, and run the nodetool scrub. This is online scrub, faster than
>> offline scrub (at least of 2.1.12, the last time I had to do this).
>>
>> Only if all else fails, attempt the very painful offline sstablescrub.
>>
>> Is the VMWare client Windows? (Trying to make sure its not just the
>> host). YMMV but in the past Windows was somewhat of a neglected platform
>> wrt Cassandra. I think you'd have a lot easier time getting help if running
>> Linux is an option here.
>>
>>
>>
>> On Fri, Aug 12, 2016 at 9:16 AM, Alaa Zubaidi (PDF) > > wrote:
>>
>>> Hi Jason,
>>>
>>> Thanks for your input...
>>> Thats what I am afraid of?
>>> Did you find any HW error in the VMware and HW logs? any indication that
>>> the HW is the reason? I need to make sure that this is the reason before
>>> asking the customer to spend more money?
>>>
>>> Thanks,
>>> Alaa
>>>
>>> On Thu, Aug 11, 2016 at 11:02 PM, Jason Wee  wrote:
>>>
>>>> cassandra run on virtual server (vmware)?
>>>>
>>>> > I tried sstablescrub but it crashed with hs-err-pid-...
>>>> maybe try with larger heap allocated to sstablescrub
>>>>
>>>> this sstable corrupt i ran into it as well (on cassandra 1.2), first i
>>>> try nodetool scrub, still persist, then offline sstablescrub still
>>>> persist, wipe the node and it happen again, then i change the hardware
>>>> (disk and mem). things went good.
>>>>
>>>> hth
>>>>
>>>> jason
>>>>
>>>>
>>>> On Fri, Aug 12, 2016 at 9:20 AM, Alaa Zubaidi (PDF)
>>>>  wrote:
>>>> > Hi,
>>>> >
>>>> > I have a 16 Node cluster, Cassandra 2.2.1 on Windows, local
>>>> installation
>>>> > (NOT on the cloud)
>>>> >
>>>> > and I am getting
>>>> > Error [CompactionExecutor:2] 2016-08-12 06:51:52, 983 Cassandra
>>>> > Daemon.java:183 - Execption in thread Thread[CompactionExecutor:2,1m
>>>> ain]
>>>> > org.apache.cassandra.io.FSReaderError:
>>>> > org.apache.cassandra.io.sstable.CorruptSSTableExecption:
>>>> > org.apache.cassandra.io.compress.CurrptBlockException:
>>>> > (E:\\la-4886-big-Data.db): corruption detected, chunk at
>>>> 4969092 of
>>>> > length 10208.
>>>> > at
>>>> > org.apache.cassandra.io.util.RandomAccessReader.readBytes(Ra
>>>> ndomAccessReader.java:357)
>>>> > ~[apache-cassandra-2.2.1.jar:2.2.1]
>>>> > 
>>>> > 
>>>> > ERROR [CompactionExecutor:2] ... FileUtils.java:463 - Existing
>>>> > forcefully due to file system exception on startup, disk failure
>>>> policy
>>>> > "stop"
>>>> >
>>>> > I tried sstablescrub but it crashed with hs-err-pid-...
>>>> > I removed the corrupted file and started the Node again, after one
>>>> day the
>>>> > corruption came back again, I removed the files, and restarted
>

Re: Corrupt SSTABLE over and over

[Alaa Zubaidi (PDF)]

Hi Jason,

Thanks for your input...
Thats what I am afraid of?
Did you find any HW error in the VMware and HW logs? any indication that
the HW is the reason? I need to make sure that this is the reason before
asking the customer to spend more money?

Thanks,
Alaa

On Thu, Aug 11, 2016 at 11:02 PM, Jason Wee  wrote:

> cassandra run on virtual server (vmware)?
>
> > I tried sstablescrub but it crashed with hs-err-pid-...
> maybe try with larger heap allocated to sstablescrub
>
> this sstable corrupt i ran into it as well (on cassandra 1.2), first i
> try nodetool scrub, still persist, then offline sstablescrub still
> persist, wipe the node and it happen again, then i change the hardware
> (disk and mem). things went good.
>
> hth
>
> jason
>
>
> On Fri, Aug 12, 2016 at 9:20 AM, Alaa Zubaidi (PDF)
>  wrote:
> > Hi,
> >
> > I have a 16 Node cluster, Cassandra 2.2.1 on Windows, local installation
> > (NOT on the cloud)
> >
> > and I am getting
> > Error [CompactionExecutor:2] 2016-08-12 06:51:52, 983 Cassandra
> > Daemon.java:183 - Execption in thread Thread[CompactionExecutor:2,1main]
> > org.apache.cassandra.io.FSReaderError:
> > org.apache.cassandra.io.sstable.CorruptSSTableExecption:
> > org.apache.cassandra.io.compress.CurrptBlockException:
> > (E:\\la-4886-big-Data.db): corruption detected, chunk at
> 4969092 of
> > length 10208.
> > at
> > org.apache.cassandra.io.util.RandomAccessReader.readBytes(
> RandomAccessReader.java:357)
> > ~[apache-cassandra-2.2.1.jar:2.2.1]
> > 
> > 
> > ERROR [CompactionExecutor:2] ... FileUtils.java:463 - Existing
> > forcefully due to file system exception on startup, disk failure policy
> > "stop"
> >
> > I tried sstablescrub but it crashed with hs-err-pid-...
> > I removed the corrupted file and started the Node again, after one day
> the
> > corruption came back again, I removed the files, and restarted
> Cassandra, it
> > worked for few days, then I ran "nodetool repair" after it finished,
> > Cassandra failed again but with commitlog corruption, after removing the
> > commitlog files, it failed again with another sstable corruption.
> >
> > I was also checking the HW, file system, and memory, the VMware logs
> showed
> > no HW error, also the HW management logs showed NO problems or issues.
> > Also checked the Windows Logs (Application and System) the only thing I
> > found is on the system logs "Cassandra Service terminated with
> > service-specific error Cannot create another system semaphore.
> >
> > I could not find any thing regarding that error, all comments point to
> > application log.
> >
> > Any help is appreciated..
> >
> > --
> >
> > Alaa Zubaidi
> >
> >
> > This message may contain confidential and privileged information. If it
> has
> > been sent to you in error, please reply to advise the sender of the error
> > and then immediately permanently delete it and all attachments to it from
> > your systems. If you are not the intended recipient, do not read, copy,
> > disclose or otherwise use this message or any attachments to it. The
> sender
> > disclaims any liability for such unauthorized use. PLEASE NOTE that all
> > incoming e-mails sent to PDF e-mail accounts will be archived and may be
> > scanned by us and/or by external service providers to detect and prevent
> > threats to our systems, investigate illegal or inappropriate behavior,
> > and/or eliminate unsolicited promotional e-mails (“spam”). If you have
> any
> > concerns about this process, please contact us at
> legal.departm...@pdf.com.
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Corrupt SSTABLE over and over

[Alaa Zubaidi (PDF)]

One more thing I noticed..
The corrupted SSTable is mentioned twice in the log file
[CompactionExecutor:10253] 2016-08-11 08:59:01,952 - Compacting (.)
[...la-1104-big-Data.db, ]
[CompactionExecutor:10253] 2016-08-11 09:32:04,814 - Compacting (.)
[...la-1104-big-Data.db]

Is it possible Cassandra is trying to compact the same file again while its
being compacted by another process?

Regards,
Alaa

On Thu, Aug 11, 2016 at 6:20 PM, Alaa Zubaidi (PDF) 
wrote:

> Hi,
>
> I have a 16 Node cluster, Cassandra 2.2.1 on Windows, local installation
> (NOT on the cloud)
>
> and I am getting
> Error [CompactionExecutor:2] 2016-08-12 06:51:52, 983 Cassandra
> Daemon.java:183 - Execption in thread Thread[CompactionExecutor:2,1main]
> org.apache.cassandra.io.FSReaderError: 
> org.apache.cassandra.io.sstable.CorruptSSTableExecption:
> org.apache.cassandra.io.compress.CurrptBlockException:
> (E:\\la-4886-big-Data.db): corruption detected, chunk at 4969092
> of length 10208.
> at 
> org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:357)
> ~[apache-cassandra-2.2.1.jar:2.2.1]
> 
> 
> ERROR [CompactionExecutor:2] ... FileUtils.java:463 - Existing
> forcefully due to file system exception on startup, disk failure policy
> "stop"
>
> I tried sstablescrub but it crashed with hs-err-pid-...
> I removed the corrupted file and started the Node again, after one day the
> corruption came back again, I removed the files, and restarted Cassandra,
> it worked for few days, then I ran "nodetool repair" after it finished,
> Cassandra failed again but with commitlog corruption, after removing the
> commitlog files, it failed again with another sstable corruption.
>
> I was also checking the HW, file system, and memory, the VMware logs
> showed no HW error, also the HW management logs showed NO problems or
> issues.
> Also checked the Windows Logs (Application and System) the only thing I
> found is on the system logs "Cassandra Service terminated with
> service-specific error Cannot create another system semaphore.
>
> I could not find any thing regarding that error, all comments point to
> application log.
>
> Any help is appreciated..
>
> --
>
> Alaa Zubaidi
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Corrupt SSTABLE over and over

[Alaa Zubaidi (PDF)]

Hi,

I have a 16 Node cluster, Cassandra 2.2.1 on Windows, local installation
(NOT on the cloud)

and I am getting
Error [CompactionExecutor:2] 2016-08-12 06:51:52, 983 Cassandra
Daemon.java:183 - Execption in thread Thread[CompactionExecutor:2,1main]
org.apache.cassandra.io.FSReaderError:
org.apache.cassandra.io.sstable.CorruptSSTableExecption:
org.apache.cassandra.io.compress.CurrptBlockException:
(E:\\la-4886-big-Data.db): corruption detected, chunk at 4969092 of
length 10208.
at
org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:357)
~[apache-cassandra-2.2.1.jar:2.2.1]


ERROR [CompactionExecutor:2] ... FileUtils.java:463 - Existing
forcefully due to file system exception on startup, disk failure policy
"stop"

I tried sstablescrub but it crashed with hs-err-pid-...
I removed the corrupted file and started the Node again, after one day the
corruption came back again, I removed the files, and restarted Cassandra,
it worked for few days, then I ran "nodetool repair" after it finished,
Cassandra failed again but with commitlog corruption, after removing the
commitlog files, it failed again with another sstable corruption.

I was also checking the HW, file system, and memory, the VMware logs showed
no HW error, also the HW management logs showed NO problems or issues.
Also checked the Windows Logs (Application and System) the only thing I
found is on the system logs "Cassandra Service terminated with
service-specific error Cannot create another system semaphore.

I could not find any thing regarding that error, all comments point to
application log.

Any help is appreciated..

-- 

Alaa Zubaidi

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: nodetool -h fails Connection refused

[Alaa Zubaidi (PDF)]

Thanks Nate...
It works now..

On Wed, Apr 20, 2016 at 8:05 AM, Nate McCall  wrote:

> You need to set LOCAL_JMX=false
>
> It will then read the rest of this stanza:
>
> https://github.com/apache/cassandra/blob/cassandra-2.2/conf/cassandra-env.sh#L284-L288
>
> Using the defaults above as-is, you will need to add JMX authentication.
> Details are here:
>
> https://docs.datastax.com/en/cassandra/2.2/cassandra/configuration/secureNodetoolSSL.html
>
> A lot of this can be controlled with system properties as well:
> http://docs.oracle.com/javase/8/docs/technotes/guides/management/agent.html
>
> The default config files for JMX authentication and access included in the
> JVM also have extensive details in the comments:
> $JAVA_HOME/jre/lib/management/jmxremote.access
> $JAVA_HOME/jre/lib/management/jmxremote.password.template
>
>
>
> On Tue, Apr 19, 2016 at 8:40 PM, Alaa Zubaidi (PDF) 
> wrote:
>
>> Hi,
>>
>> I am trying to run nodetool remotely. but its not working:
>> I am running Cassandra 2.2.5 on CentOS 6.
>> listen_address: is set to 
>> rpc_address: is set to 0.0.0.0
>> broadcast_rpc_address: is set to 
>>
>> I changed the following in cassadnra-env.sh
>> JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname="
>> -Dcom.sun.management.jmxremote.port=7199
>> -Dcom.sun.management.jmxremote.ssl=false
>> -Dcom.sun.management.jmxremote.authenticate=false
>>
>> "nodetool -h  -p  status" results in:
>> failed to connect to 'hostname' - Connection Exception: 'Connection
>> refused"
>>
>> netstat -nl | grep 7199
>> tcp00127.0.0.1:71990.0.0.0:*LISTEN
>>
>> ONLY "nodetool -h localhost" works
>>
>> Any idea how to fix it?
>>
>> Thanks,
>> Alaa
>>
>>
>> *This message may contain confidential and privileged information. If it
>> has been sent to you in error, please reply to advise the sender of the
>> error and then immediately permanently delete it and all attachments to it
>> from your systems. If you are not the intended recipient, do not read,
>> copy, disclose or otherwise use this message or any attachments to it. The
>> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
>> all incoming e-mails sent to PDF e-mail accounts will be archived and may
>> be scanned by us and/or by external service providers to detect and prevent
>> threats to our systems, investigate illegal or inappropriate behavior,
>> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
>> concerns about this process, please contact us at *
>> *legal.departm...@pdf.com* *.*
>
>
>
>
> --
> -
> Nate McCall
> Austin, TX
> @zznate
>
> Co-Founder & Sr. Technical Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

nodetool -h fails Connection refused

[Alaa Zubaidi (PDF)]

Hi,

I am trying to run nodetool remotely. but its not working:
I am running Cassandra 2.2.5 on CentOS 6.
listen_address: is set to 
rpc_address: is set to 0.0.0.0
broadcast_rpc_address: is set to 

I changed the following in cassadnra-env.sh
JVM_OPTS="$JVM_OPTS -Djava.rmi.server.hostname="
-Dcom.sun.management.jmxremote.port=7199
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

"nodetool -h  -p  status" results in:
failed to connect to 'hostname' - Connection Exception: 'Connection refused"

netstat -nl | grep 7199
tcp00127.0.0.1:71990.0.0.0:*LISTEN

ONLY "nodetool -h localhost" works

Any idea how to fix it?

Thanks,
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Cassandra 2.2.1 on Windows (disk_access_mode: mmap_index_only)

[Alaa Zubaidi (PDF)]

Hi,
Just an update...
we changed the disk_access_mode to mmap_index_only and the memory problem
disappeared. However, we did NOT confirm yet, the exact affect on
performance, but in general acceptable so far, it could due to the fact
that its not optimized anyways yet, since we just started using 2.2.1
recently.

We also had to add this to cassandra.yaml file
index_summary_resize_interval_in_minutes=-1
after issues with automatic compaction CASSANDRA-10270

If any one has any feedback on this or on windows configuration or
cassandra configuration on windows, it would be mush appreciated.

Thanks,

On Wed, Oct 28, 2015 at 1:57 PM, Alaa Zubaidi (PDF) 
wrote:

> Thanks Rob,
> We are using 7GB JVM on a 32GB RAM VM
> Alaa
>
> On Wed, Oct 28, 2015 at 11:18 AM, Robert Coli 
> wrote:
>
>> On Tue, Oct 27, 2015 at 8:47 PM, Alaa Zubaidi (PDF) > > wrote:
>>
>>> We have Cassandra 2.2.1 on Windows 2008R2-64-bit.
>>>
>>> We are noticing that during compaction Cassandra consumes all the
>>> available memory for the VM and bring the VM to a crawl. compaction
>>> settings are default, and we are using sized tiered compaction.
>>>
>>> When investigating this, we find through VMMAP that the memory is
>>> consumed by mapped files.
>>> These are almost all the sstables (index and data) for all keyspaces,
>>> mapped into memory (system and user defined keyspaces) .
>>>
>>> Is this normal, or should these files not be mapped into memroy after
>>> compaction?
>>>
>>
>> https://wiki.apache.org/cassandra/FAQ#mmap
>>
>> However that is probably not really your problem, you probably just have
>> too small of a heap if GC is failing?
>>
>> =Rob
>>
>>
>
>
>
> --
>
> Alaa Zubaidi
> PDF Solutions, Inc.
> 333 West San Carlos Street, Suite 1000
> San Jose, CA 95110  USA
> Tel: 408-283-5639
> fax: 408-938-6479
> email: alaa.zuba...@pdf.com
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Cassandra 2.2.1 on Windows

[Alaa Zubaidi (PDF)]

Thanks Rob,
We are using 7GB JVM on a 32GB RAM VM
Alaa

On Wed, Oct 28, 2015 at 11:18 AM, Robert Coli  wrote:

> On Tue, Oct 27, 2015 at 8:47 PM, Alaa Zubaidi (PDF) 
> wrote:
>
>> We have Cassandra 2.2.1 on Windows 2008R2-64-bit.
>>
>> We are noticing that during compaction Cassandra consumes all the
>> available memory for the VM and bring the VM to a crawl. compaction
>> settings are default, and we are using sized tiered compaction.
>>
>> When investigating this, we find through VMMAP that the memory is
>> consumed by mapped files.
>> These are almost all the sstables (index and data) for all keyspaces,
>> mapped into memory (system and user defined keyspaces) .
>>
>> Is this normal, or should these files not be mapped into memroy after
>> compaction?
>>
>
> https://wiki.apache.org/cassandra/FAQ#mmap
>
> However that is probably not really your problem, you probably just have
> too small of a heap if GC is failing?
>
> =Rob
>
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Cassandra 2.2.1 on Windows

[Alaa Zubaidi (PDF)]

Hi,
We have Cassandra 2.2.1 on Windows 2008R2-64-bit.

We are noticing that during compaction Cassandra consumes all the available
memory for the VM and bring the VM to a crawl. compaction settings are
default, and we are using sized tiered compaction.

When investigating this, we find through VMMAP that the memory is consumed
by mapped files.
These are almost all the sstables (index and data) for all keyspaces,
mapped into memory (system and user defined keyspaces) .

Is this normal, or should these files not be mapped into memroy after
compaction?

Regards,
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Cassandra 2.2.1 stuck at 100% on Windows

[Alaa Zubaidi (PDF)]

It was a process installed by IT that triggered this, when disabling the
process everything went to normal.

Thanks.
Alaa

On Fri, Oct 16, 2015 at 11:32 AM, Alaa Zubaidi (PDF) 
wrote:

> Thanks guys,
> I will look into this more, and put an update here, if I find anything
>
> On Fri, Oct 16, 2015 at 10:40 AM, Josh McKenzie 
> wrote:
>
>> One option: use process explorer to find out the TID's of the java
>> process (instructions
>> <https://superuser.com/questions/462969/how-can-i-view-the-active-threads-of-a-running-program>),
>> screen cap that, then also run jstack against the running cassandra process
>> out to a file a few times (instructions
>> <https://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html>
>> ).
>>
>> We should be able to at least link up the TID to the hex thread # in the
>> jstack output to figure out who/what is spinning on there.
>>
>> On Fri, Oct 16, 2015 at 1:28 PM, Michael Shuler 
>> wrote:
>>
>>> On 10/16/2015 12:02 PM, Alaa Zubaidi (PDF) wrote:
>>>
>>>> No OOM in any of the log files, and NO long GC at that time.
>>>> I attached the last 2 minutes before it hangs until we restart cassandra
>>>> after hour an half.
>>>>
>>>
>>> Your logs show gossip issues with some seed nodes. `nodetool gossipinfo`
>>> on all nodes might be an interesting place to start.
>>>
>>> --
>>> Michael
>>>
>>
>>
>
>
> --
>
> Alaa Zubaidi
> PDF Solutions, Inc.
> 333 West San Carlos Street, Suite 1000
> San Jose, CA 95110  USA
> Tel: 408-283-5639
> fax: 408-938-6479
> email: alaa.zuba...@pdf.com
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Cassandra 2.2.1 stuck at 100% on Windows

[Alaa Zubaidi (PDF)]

Thanks guys,
I will look into this more, and put an update here, if I find anything

On Fri, Oct 16, 2015 at 10:40 AM, Josh McKenzie 
wrote:

> One option: use process explorer to find out the TID's of the java process
> (instructions
> <https://superuser.com/questions/462969/how-can-i-view-the-active-threads-of-a-running-program>),
> screen cap that, then also run jstack against the running cassandra process
> out to a file a few times (instructions
> <https://docs.oracle.com/javase/7/docs/technotes/tools/share/jstack.html>
> ).
>
> We should be able to at least link up the TID to the hex thread # in the
> jstack output to figure out who/what is spinning on there.
>
> On Fri, Oct 16, 2015 at 1:28 PM, Michael Shuler 
> wrote:
>
>> On 10/16/2015 12:02 PM, Alaa Zubaidi (PDF) wrote:
>>
>>> No OOM in any of the log files, and NO long GC at that time.
>>> I attached the last 2 minutes before it hangs until we restart cassandra
>>> after hour an half.
>>>
>>
>> Your logs show gossip issues with some seed nodes. `nodetool gossipinfo`
>> on all nodes might be an interesting place to start.
>>
>> --
>> Michael
>>
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Cassandra 2.2.1 stuck at 100% on Windows

[Alaa Zubaidi (PDF)]

Thanks Rob,

No OOM in any of the log files, and NO long GC at that time.
I attached the last 2 minutes before it hangs until we restart cassandra
after hour an half.
Regards,
Alaa

On Thu, Oct 15, 2015 at 6:29 PM, Robert Coli  wrote:

> On Thu, Oct 15, 2015 at 6:04 PM, Alaa Zubaidi (PDF) 
> wrote:
>
>> We are running Cassandra 2.2.1 on Windows 2008R2, and we see that
>> multiple Nodes are stuck at 99% CPU bringing the whole VM to a halt.
>> We suspect that there is another process that IT/Windows is causing the
>> CPU issue, but the problem is Cassandra does NOT recover, the CPU
>> utilization start climbing until the VM is not usable. If we restart
>> Cassandra, things go back to normal.
>>
>
> Most cases where a JVM does not recover and churns at maxed CPU are the
> result of GC failure and/or OOM.
>
> Check your logs for OOM and long GCs.
>
> Also FWIW you are among a relatively small group of Windows operators.
> Other than with the people working at datastax to support Windows, there is
> not a whole lot of well understood operational best practice for Cassandra
> on Windows.
>
> =Rob
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*


07.log
Description: Binary data

Cassandra 2.2.1 stuck at 100% on Windows

[Alaa Zubaidi (PDF)]

Hi,
We are running Cassandra 2.2.1 on Windows 2008R2, and we see that multiple
Nodes are stuck at 100% CPU bringing the whole VM to a halt.
We suspect that there is another process that IT/Windows is causing the CPU
issue, but the problem is Cassandra does NOT recover, the CPU utilization
start climbing until the VM is not usable.If we restart Cassandra it
Anyone have seen this before?

Thanks
-- Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Cassandra 2.2.1 stuck at 100% on Windows

[Alaa Zubaidi (PDF)]

Hi,
We are running Cassandra 2.2.1 on Windows 2008R2, and we see that multiple
Nodes are stuck at 99% CPU bringing the whole VM to a halt.
We suspect that there is another process that IT/Windows is causing the CPU
issue, but the problem is Cassandra does NOT recover, the CPU utilization
start climbing until the VM is not usable. If we restart Cassandra, things
go back to normal.
Anyone have seen this before?

Thanks
-- Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Spark partitions from CassandraRDD

[Alaa Zubaidi (PDF)]

Hi,

I testing Spark and Cassandra, Spark 1.4, Cassandra 2.1.7 cassandra spark
connector 1.4, running in standalone mode.

I am getting 4000 rows from Cassandra (4mb row), where the row keys are
random.
.. sc.cassandraTable[RES](keyspace,res_name).where(res_where).cache

I am expecting that it will generate few partitions.
However, I can ONLY see 1 partition.
I cached the CassandraRDD and in the UI storage tab it shows ONLY 1
partition.

Any idea, why I am getting 1 partition?

Thanks,
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: CQL Data Model question

[Alaa Zubaidi (PDF)]

Thanks Ngoc, Jack

On Tue, May 12, 2015 at 4:56 AM, Ngoc Minh VO 
wrote:

>  Hello,
>
>
>
> The problem with your approach is: you will need to specify all the 30
> filters (in the pre-defined order in PK) when querying.
>
>
>
> I would go for this data model:
>
> CREATE TABLE t (
>
> name text,
>
> filter_name1 text, filter_value1 text,
>
> filter_name2 text, filter_value2 text,
>
> filter_name3 text, filter_value3 text, -- since you only have up to 3
> filters in one query
>
> PRIMARY KEY (name, f_n1, f_v1, f_n2, f_v2, f_n3, f_v3)
>
> );
>
>
>
> And denormalize the data when you import to the table :
>
> One line in Oracle table with K filters become C(3, K) lines in C* table.
>
>
>
> Best regards,
>
> Minh
>
>
>
> *From:* Alaa Zubaidi (PDF) [mailto:alaa.zuba...@pdf.com]
> *Sent:* lundi 11 mai 2015 20:32
> *To:* user@cassandra.apache.org
> *Subject:* CQL Data Model question
>
>
>
> Hi,
>
>
>
> I am trying to port an Oracle Table to Cassandra.
>
> the table is a wide table (931 columns) and could have millions of rows.
>
>  name, filter1, filter2filter30, data1, data2...data900
>
>
>
> The user would retrieve multiple rows from this table and filter (30
> filter columns) by one or more (up to 3) of the filter columns, it could be
> any of the filter columns.
>
> (select * from table1 where name = .. and filter1 = .. and filter5= .. ;)
>
>
>
> What is the best design for this in Cassandra/CQL?
>
>
>
> I tried the following:
>
> Create table tab1 (
>
> name text,
>
> flt1 text,
>
> flt2 text,
>
> flt3 text,
>
> ..
>
> flt30 text,
>
> data text,
>
> PRIMARY KEY (name, flt1, flt2, flt3, . flt30) );
>
>
>
> Is there any side effects of having 30 composite keys?
>
>
>
> Thanks
>
>
> *This message may contain confidential and privileged information. If it
> has been sent to you in error, please reply to advise the sender of the
> error and then immediately permanently delete it and all attachments to it
> from your systems. If you are not the intended recipient, do not read,
> copy, disclose or otherwise use this message or any attachments to it. The
> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
> all incoming e-mails sent to PDF e-mail accounts will be archived and may
> be scanned by us and/or by external service providers to detect and prevent
> threats to our systems, investigate illegal or inappropriate behavior,
> and/or eliminate unsolicited promotional e-mails (“spam”). If you have any
> concerns about this process, please contact us at *
> *legal.departm...@pdf.com* *.*
>
> This message and any attachments (the "message") is
> intended solely for the intended addressees and is confidential.
> If you receive this message in error,or are not the intended recipient(s),
> please delete it and any copies from your systems and immediately notify
> the sender. Any unauthorized view, use that does not comply with its
> purpose,
> dissemination or disclosure, either whole or partial, is prohibited. Since
> the internet
> cannot guarantee the integrity of this message which may not be reliable,
> BNP PARIBAS
> (and its subsidiaries) shall not be liable for the message if modified,
> changed or falsified.
> Do not print this message unless it is necessary,consider the environment.
>
>
> --
>
> Ce message et toutes les pieces jointes (ci-apres le "message")
> sont etablis a l'intention exclusive de ses destinataires et sont
> confidentiels.
> Si vous recevez ce message par erreur ou s'il ne vous est pas destine,
> merci de le detruire ainsi que toute copie de votre systeme et d'en avertir
> immediatement l'expediteur. Toute lecture non autorisee, toute utilisation
> de
> ce message qui n'est pas conforme a sa destination, toute diffusion ou
> toute
> publication, totale ou partielle, est interdite. L'Internet ne permettant
> pas d'assurer
> l'integrite de ce message electronique susceptible d'alteration, BNP
> Paribas
> (et ses filiales) decline(nt) toute responsabilite au titre de ce message
> dans l'hypothese
> ou il aurait ete modifie, deforme ou falsifie.
> N'imprimez ce message que si necessaire, pensez a l'environnement.
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may 

CQL Data Model question

[Alaa Zubaidi (PDF)]

Hi,

I am trying to port an Oracle Table to Cassandra.
the table is a wide table (931 columns) and could have millions of rows.
 name, filter1, filter2filter30, data1, data2...data900

The user would retrieve multiple rows from this table and filter (30 filter
columns) by one or more (up to 3) of the filter columns, it could be any of
the filter columns.
(select * from table1 where name = .. and filter1 = .. and filter5= .. ;)

What is the best design for this in Cassandra/CQL?

I tried the following:
Create table tab1 (
name text,
flt1 text,
flt2 text,
flt3 text,
..
flt30 text,
data text,
PRIMARY KEY (name, flt1, flt2, flt3, . flt30) );

Is there any side effects of having 30 composite keys?

Thanks

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Cassandra on Window with SCOM running

[Alaa Zubaidi (PDF)]

Hi,
We are running an Old version of Cassandra 0.7.4 on Windows
At one point, the average read times is 2.1 seconds when SCOM (System
Center Operation Manager) is running, and 0.04 seconds when its disabled.

The problem is we CANNOT disable SCOM on these Windows VMs.
Any have seen this before? experience with SCOM in this situations?
any comments of feedback that could help us improve the performance while
this process is running?
Other than upgrading Cassandra to the latest (we know this is an option but
its NOT doable at this point).

Thanks and Regards,
Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails (“spam”). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Cassandra Caused by: UnavailableException()

[Alaa Zubaidi (PDF)]

Hi,
We have an 8 Node 1.0.10 Cassandra cluster with Replication of 3, running
on customer location, they all should be in the same DC, but we are NOT
100% sure.
We are having problem with the our own Cassandra client, the log file of
the client shows many "socket timeout" and "socket closed" errors, and many
delays, in the Reads.

while debugging the issue we found the following error in the our Cassandra
client:
Caused by: UnavailableException()

Cassandra log file does not show errors..

What does that mean? any idea?
 =
[com.pdf.common.COMN_Log4jLogger] Failed to write Context Flow row to
column 20140317210716767_a3ccda262dbdd287a5e538ec95483a74_40:
UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$batch_mutate_result.read(Cassandra.java:19487)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_batch_mutate(Cassandra.java:1035)
at
org.apache.cassandra.thrift.Cassandra$Client.batch_mutate(Cassandra.java:1009)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.putLTStoreContextFlowRow(DSTO_CassandraClient.java:2673)
at
com.pdf.rtapi.RAPI_DataStore.putLTStoreContextFlowRow(RAPI_DataStore.java:1254)
at
com.pdf.rtapi.RAPI_InternalInterface.putLTStoreContextFlowRow(RAPI_InternalInterface.java:756)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.putLTStoreContextFlowRow(RtApiFacadeImpl.java:340)
at
com.pdf.online.service.impl.OnlineServiceImpl.notifyDCContextFlow(OnlineServiceImpl.java:1499)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:602)


Different error..

2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Recipe name is:
P13_01_28NM_M5_002AS3_NOVA
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470)
[com.pdf.online.service.impl.OnlineServiceImpl] Reading framed data (trace
+ context) for 'dcId' = 44140
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- Creating key base:
equipmentId_moduleId: POL1702_CleanerBrush1
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- Hashed key:
12169de1f273c1d38c893c0dfcbf10ff
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
RAPI_DataStore::readFramedData() -- DSTO_ContextKey =
12169de1f273c1d38c893c0dfcbf10ff:20131115070726207, dcId = 44140
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
Call checkConnection().
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
Call checkConnection().
2014-Mar-17 21:51:20,531 - INFO  TID=Thread-6
(HornetQ-client-global-threads-4002470) [com.pdf.common.COMN_Log4jLogger]
UnavailableException : null: UnavailableException()
at
org.apache.cassandra.thrift.Cassandra$get_slice_result.read(Cassandra.java:7764)
at
org.apache.cassandra.thrift.Cassandra$Client.recv_get_slice(Cassandra.java:570)
at
org.apache.cassandra.thrift.Cassandra$Client.get_slice(Cassandra.java:542)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readContextData(DSTO_CassandraClient.java:1894)
at
com.pdf.datastore.cassandra.DSTO_CassandraClient.readFramedData(DSTO_CassandraClient.java:1848)
at com.pdf.rtapi.RAPI_DataStore.readFramedData(RAPI_DataStore.java:720)
at
com.pdf.rtapi.RAPI_InternalInterface.readFramedData(RAPI_InternalInterface.java:741)
at
com.pdf.common.rtapi.impl.RtApiFacadeImpl.readFramedData(RtApiFacadeImpl.java:261)
at
com.pdf.online.service.impl.OnlineServiceImpl.readCassandraData(OnlineServiceImpl.java:2189)
at
com.pdf.online.service.impl.OnlineServiceImpl.process(OnlineServiceImpl.java:328)
at
com.pdf.online.mdb.OnlineEventDrivenBean.process(OnlineEventDrivenBean.java:302)
at
com.pdf.online.mdb.OnlineEventDrivenBean.onMessage(OnlineEventDrivenBean.java:211)
at sun.reflect.GeneratedMethodAccessor83.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at org.jboss.invocation.Invocation.performCall(Invocation.java:359)
at
org.jboss.ejb.MessageDrivenContainer$ContainerInterceptor.invoke(MessageDrivenContainer.java:495)
at
org.jboss.resource.connectionmanager.CachedConnectionInterceptor.invoke(CachedConnectionInterceptor.java:158)
at
org.jboss.ejb.plugins.CallValidationInterceptor.invoke(CallValidationInterceptor.java:63)
at
org.jboss.ejb.plugins.AbstractTxInterceptor.invokeNext(AbstractTxInterceptor.java:121)
at
org.jboss.ejb.plugins.AbstractTxInterceptorBMT.invokeNext(AbstractTxInterceptorBMT.java:173)
at
org.jboss.ejb.plugins.MessageDrivenTxInterceptorBMT.invoke(MessageDrivenTxInterceptorBM

Re: Cassandra Internal error processing get_slice java.lang.AssertionError

[Alaa Zubaidi (PDF)]

Hi Jacob,
Unfortunately, we need to stick with 1.0.10 for while..
Any idea what does this error means?

Thanks,
Alaa


On Wed, Mar 5, 2014 at 9:05 PM, Jacob Rhoden  wrote:

> If this is a _new_ 8 node cluster, is there a specific reason you starting
> with 1.0.x, not 1.2.x? (Upgrading later will be more work)
>
> __
> Sent from iPhone
>
> On 6 Mar 2014, at 9:22 am, "Alaa Zubaidi (PDF)" 
> wrote:
>
> Cassandra 1.0.10 with default settings..
>
>
> On Wed, Mar 5, 2014 at 4:57 PM, Robert Coli  wrote:
>
>> On Wed, Mar 5, 2014 at 4:35 PM, Alaa Zubaidi (PDF) 
>> wrote:
>>
>>> I have a new 8 Node cluster, and when started loading data I see the
>>> error below showing up a lot in 3 of the 8 Nodes:
>>>
>>
>> What version of cassandra? What mode of thrift server operation, sync or
>> hsha?
>>
>> =Rob
>>
>>
>
>
>
> --
>
> Alaa Zubaidi
> PDF Solutions, Inc.
> 333 West San Carlos Street, Suite 1000
> San Jose, CA 95110  USA
> Tel: 408-283-5639
> fax: 408-938-6479
>
> email: alaa.zuba...@pdf.com
>
>
> *This message may contain confidential and privileged information. If it
> has been sent to you in error, please reply to advise the sender of the
> error and then immediately permanently delete it and all attachments to it
> from your systems. If you are not the intended recipient, do not read,
> copy, disclose or otherwise use this message or any attachments to it. The
> sender disclaims any liability for such unauthorized use. PLEASE NOTE that
> all incoming e-mails sent to PDF e-mail accounts will be archived and may
> be scanned by us and/or by external service providers to detect and prevent
> threats to our systems, investigate illegal or inappropriate behavior,
> and/or eliminate unsolicited promotional e-mails ("spam"). If you have any
> concerns about this process, please contact us at *
> *legal.departm...@pdf.com* *.*
>
>


-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails ("spam"). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Re: Cassandra Internal error processing get_slice java.lang.AssertionError

[Alaa Zubaidi (PDF)]

Cassandra 1.0.10 with default settings..


On Wed, Mar 5, 2014 at 4:57 PM, Robert Coli  wrote:

> On Wed, Mar 5, 2014 at 4:35 PM, Alaa Zubaidi (PDF) 
> wrote:
>
>> I have a new 8 Node cluster, and when started loading data I see the
>> error below showing up a lot in 3 of the 8 Nodes:
>>
>
> What version of cassandra? What mode of thrift server operation, sync or
> hsha?
>
> =Rob
>
>



-- 

Alaa Zubaidi
PDF Solutions, Inc.
333 West San Carlos Street, Suite 1000
San Jose, CA 95110  USA
Tel: 408-283-5639
fax: 408-938-6479
email: alaa.zuba...@pdf.com

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails ("spam"). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*

Cassandra Internal error processing get_slice java.lang.AssertionError

[Alaa Zubaidi (PDF)]

Hi,

I have a new 8 Node cluster, and when started loading data I see the error
below showing up a lot in 3 of the 8 Nodes:
ERROR [Thrift:65] 2014-03-05 13:19:42,450 Cassandra.java (line 3041)
Internal error processing get_slice
java.lang.AssertionError
at
org.apache.cassandra.db.SliceFromReadCommand.maybeGenerateRetryCommand(SliceFromReadCommand.java:78)
at
org.apache.cassandra.service.StorageProxy.fetchRows(StorageProxy.java:724)
at org.apache.cassandra.service.StorageProxy.read(StorageProxy.java:564)
at
org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:128)
at
org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:283)
at
org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:365)
at
org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:326)
at
org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:3033)
at
org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:2889)
at
org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:187)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)

Any idea?

-- 

Alaa

-- 
*This message may contain confidential and privileged information. If it 
has been sent to you in error, please reply to advise the sender of the 
error and then immediately permanently delete it and all attachments to it 
from your systems. If you are not the intended recipient, do not read, 
copy, disclose or otherwise use this message or any attachments to it. The 
sender disclaims any liability for such unauthorized use. PLEASE NOTE that 
all incoming e-mails sent to PDF e-mail accounts will be archived and may 
be scanned by us and/or by external service providers to detect and prevent 
threats to our systems, investigate illegal or inappropriate behavior, 
and/or eliminate unsolicited promotional e-mails ("spam"). If you have any 
concerns about this process, please contact us at *
*legal.departm...@pdf.com* *.*