Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.

2015-05-11 Thread Carlos Rolo 
Thanks also!

I did it, JAVA-761 
created!

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Mon, May 11, 2015 at 6:48 PM, Sebastian Estevez <
sebastian.este...@datastax.com> wrote:

> I hit this issue today with the c# driver. I still think the drivers should
> handle peers inconsistencies better and maybe even output warnings about
> them.
>
> I opened CSHARP-296, @rolo, it's probably a good idea to open a similar one
> for java.
> On May 11, 2015 11:24 AM, "Carlos Rolo"  wrote:
>
> > Thanks!
> >
> > Regards,
> >
> > Carlos Juzarte Rolo
> > Cassandra Consultant
> >
> > Pythian - Love your data
> >
> > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > linkedin.com/in/carlosjuzarterolo
> > *
> > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > www.pythian.com
> >
> > On Mon, May 11, 2015 at 4:29 PM, Brandon Williams 
> > wrote:
> >
> > > https://issues.apache.org/jira/browse/CASSANDRA-9180
> > >
> > > On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo  wrote:
> > >
> > > > Hi all,
> > > >
> > > > I just wanted to know if this should be worth filling a bug or not
> > > > (Couldn't find any similar).
> > > >
> > > > I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> > > > because of hardware failure (virtualized environment).
> > > >
> > > > The process was automated, so what was supposed to happen was:
> > > >
> > > > - Node 4 joins
> > > > - wait until status is UN and then 2min more
> > > > - Node 5 joins
> > > > - wait until status is UN and then 2min more
> > > > - Node 6 joins
> > > > - wait until status is UN and then 2min more
> > > >
> > > > What happened:
> > > > - Node 4 joins
> > > > - Wait...
> > > > - Node 5 joins
> > > > - VM fails while node is starting.
> > > > - VM 6 starts, no node with UN, waits 2min
> > > > - Node 6 joins
> > > > - VM fails while node is starting.
> > > >
> > > > After this, nodetool reports 4 nodes all UN
> > > > While trying an application (Datastax Java Driver 2.1) the debug log
> > > > reports that it tries to connect to Node 5 and 6 and fails.
> > > >
> > > > Checking system.peers table, I see both nodes there. So I tried
> > "nodetool
> > > > removenode " with the IDs in the table.
> > > >
> > > > It blows up with the following exception:
> > > > Exception in thread "main" java.lang.UnsupportedOperationException:
> > Host
> > > ID
> > > > not found.
> > > >
> > > > Then I decided to do the following:
> > > > DELETE from peers where ID in (ID1, ID2);
> > > >
> > > > All good, cluster still happy and driver not complaining anymore.
> > > > Is this expected behavior?
> > > >
> > > >
> > > >
> > > > Regards,
> > > >
> > > > Carlos Juzarte Rolo
> > > > Cassandra Consultant
> > > >
> > > > Pythian - Love your data
> > > >
> > > > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > > > linkedin.com/in/carlosjuzarterolo
> > > > *
> > > > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > > > www.pythian.com
> > > >
> > > > --
> > > >
> > > >
> > > > --
> > > >
> > > >
> > > >
> > > >
> > >
> >
> > --
> >
> >
> > --
> >
> >
> >
> >
>

-- 


--

Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.

2015-05-11 Thread Sebastian Estevez 
I hit this issue today with the c# driver. I still think the drivers should
handle peers inconsistencies better and maybe even output warnings about
them.

I opened CSHARP-296, @rolo, it's probably a good idea to open a similar one
for java.
On May 11, 2015 11:24 AM, "Carlos Rolo"  wrote:

> Thanks!
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *
> linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> On Mon, May 11, 2015 at 4:29 PM, Brandon Williams 
> wrote:
>
> > https://issues.apache.org/jira/browse/CASSANDRA-9180
> >
> > On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo  wrote:
> >
> > > Hi all,
> > >
> > > I just wanted to know if this should be worth filling a bug or not
> > > (Couldn't find any similar).
> > >
> > > I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> > > because of hardware failure (virtualized environment).
> > >
> > > The process was automated, so what was supposed to happen was:
> > >
> > > - Node 4 joins
> > > - wait until status is UN and then 2min more
> > > - Node 5 joins
> > > - wait until status is UN and then 2min more
> > > - Node 6 joins
> > > - wait until status is UN and then 2min more
> > >
> > > What happened:
> > > - Node 4 joins
> > > - Wait...
> > > - Node 5 joins
> > > - VM fails while node is starting.
> > > - VM 6 starts, no node with UN, waits 2min
> > > - Node 6 joins
> > > - VM fails while node is starting.
> > >
> > > After this, nodetool reports 4 nodes all UN
> > > While trying an application (Datastax Java Driver 2.1) the debug log
> > > reports that it tries to connect to Node 5 and 6 and fails.
> > >
> > > Checking system.peers table, I see both nodes there. So I tried
> "nodetool
> > > removenode " with the IDs in the table.
> > >
> > > It blows up with the following exception:
> > > Exception in thread "main" java.lang.UnsupportedOperationException:
> Host
> > ID
> > > not found.
> > >
> > > Then I decided to do the following:
> > > DELETE from peers where ID in (ID1, ID2);
> > >
> > > All good, cluster still happy and driver not complaining anymore.
> > > Is this expected behavior?
> > >
> > >
> > >
> > > Regards,
> > >
> > > Carlos Juzarte Rolo
> > > Cassandra Consultant
> > >
> > > Pythian - Love your data
> > >
> > > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > > linkedin.com/in/carlosjuzarterolo
> > > *
> > > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > > www.pythian.com
> > >
> > > --
> > >
> > >
> > > --
> > >
> > >
> > >
> > >
> >
>
> --
>
>
> --
>
>
>
>

Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.

2015-05-11 Thread Carlos Rolo 
Thanks!

Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

On Mon, May 11, 2015 at 4:29 PM, Brandon Williams  wrote:

> https://issues.apache.org/jira/browse/CASSANDRA-9180
>
> On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo  wrote:
>
> > Hi all,
> >
> > I just wanted to know if this should be worth filling a bug or not
> > (Couldn't find any similar).
> >
> > I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> > because of hardware failure (virtualized environment).
> >
> > The process was automated, so what was supposed to happen was:
> >
> > - Node 4 joins
> > - wait until status is UN and then 2min more
> > - Node 5 joins
> > - wait until status is UN and then 2min more
> > - Node 6 joins
> > - wait until status is UN and then 2min more
> >
> > What happened:
> > - Node 4 joins
> > - Wait...
> > - Node 5 joins
> > - VM fails while node is starting.
> > - VM 6 starts, no node with UN, waits 2min
> > - Node 6 joins
> > - VM fails while node is starting.
> >
> > After this, nodetool reports 4 nodes all UN
> > While trying an application (Datastax Java Driver 2.1) the debug log
> > reports that it tries to connect to Node 5 and 6 and fails.
> >
> > Checking system.peers table, I see both nodes there. So I tried "nodetool
> > removenode " with the IDs in the table.
> >
> > It blows up with the following exception:
> > Exception in thread "main" java.lang.UnsupportedOperationException: Host
> ID
> > not found.
> >
> > Then I decided to do the following:
> > DELETE from peers where ID in (ID1, ID2);
> >
> > All good, cluster still happy and driver not complaining anymore.
> > Is this expected behavior?
> >
> >
> >
> > Regards,
> >
> > Carlos Juzarte Rolo
> > Cassandra Consultant
> >
> > Pythian - Love your data
> >
> > rolo@pythian | Twitter: cjrolo | Linkedin: *
> > linkedin.com/in/carlosjuzarterolo
> > *
> > Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> > www.pythian.com
> >
> > --
> >
> >
> > --
> >
> >
> >
> >
>

-- 


--

Re: Nodes failed to bootstrap, no nodetool info but system.peer populated.

2015-05-11 Thread Brandon Williams 
https://issues.apache.org/jira/browse/CASSANDRA-9180

On Mon, May 11, 2015 at 4:17 AM, Carlos Rolo  wrote:

> Hi all,
>
> I just wanted to know if this should be worth filling a bug or not
> (Couldn't find any similar).
>
> I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
> because of hardware failure (virtualized environment).
>
> The process was automated, so what was supposed to happen was:
>
> - Node 4 joins
> - wait until status is UN and then 2min more
> - Node 5 joins
> - wait until status is UN and then 2min more
> - Node 6 joins
> - wait until status is UN and then 2min more
>
> What happened:
> - Node 4 joins
> - Wait...
> - Node 5 joins
> - VM fails while node is starting.
> - VM 6 starts, no node with UN, waits 2min
> - Node 6 joins
> - VM fails while node is starting.
>
> After this, nodetool reports 4 nodes all UN
> While trying an application (Datastax Java Driver 2.1) the debug log
> reports that it tries to connect to Node 5 and 6 and fails.
>
> Checking system.peers table, I see both nodes there. So I tried "nodetool
> removenode " with the IDs in the table.
>
> It blows up with the following exception:
> Exception in thread "main" java.lang.UnsupportedOperationException: Host ID
> not found.
>
> Then I decided to do the following:
> DELETE from peers where ID in (ID1, ID2);
>
> All good, cluster still happy and driver not complaining anymore.
> Is this expected behavior?
>
>
>
> Regards,
>
> Carlos Juzarte Rolo
> Cassandra Consultant
>
> Pythian - Love your data
>
> rolo@pythian | Twitter: cjrolo | Linkedin: *
> linkedin.com/in/carlosjuzarterolo
> *
> Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
> www.pythian.com
>
> --
>
>
> --
>
>
>
>

Nodes failed to bootstrap, no nodetool info but system.peer populated.

2015-05-11 Thread Carlos Rolo 
Hi all,

I just wanted to know if this should be worth filling a bug or not
(Couldn't find any similar).

I have a 3 node cluster (2.0.14). Decided to add 3 new ones. 2 failed
because of hardware failure (virtualized environment).

The process was automated, so what was supposed to happen was:

- Node 4 joins
- wait until status is UN and then 2min more
- Node 5 joins
- wait until status is UN and then 2min more
- Node 6 joins
- wait until status is UN and then 2min more

What happened:
- Node 4 joins
- Wait...
- Node 5 joins
- VM fails while node is starting.
- VM 6 starts, no node with UN, waits 2min
- Node 6 joins
- VM fails while node is starting.

After this, nodetool reports 4 nodes all UN
While trying an application (Datastax Java Driver 2.1) the debug log
reports that it tries to connect to Node 5 and 6 and fails.

Checking system.peers table, I see both nodes there. So I tried "nodetool
removenode " with the IDs in the table.

It blows up with the following exception:
Exception in thread "main" java.lang.UnsupportedOperationException: Host ID
not found.

Then I decided to do the following:
DELETE from peers where ID in (ID1, ID2);

All good, cluster still happy and driver not complaining anymore.
Is this expected behavior?



Regards,

Carlos Juzarte Rolo
Cassandra Consultant

Pythian - Love your data

rolo@pythian | Twitter: cjrolo | Linkedin: *linkedin.com/in/carlosjuzarterolo
*
Mobile: +31 6 159 61 814 | Tel: +1 613 565 8696 x1649
www.pythian.com

-- 


--