Re: WAN replication issue in cloud native environments

[Jacob Barrett]

> On Jan 21, 2020, at 1:24 PM, Nabarun Nag  wrote:
> 
> Suggestion:
> - Instead, can we create a new class that contains the memberID and
> ServerLocation and that new class object is added as a key in the
> connectionMap.

I poked around a bit in this code and the ServerLocation is also in the 
LoadHolder class so we can simplify this even more by just using the member ID 
as the key in all these maps. When we need the ServerLocation we can get that 
from the LoadHolder. 

The addServer call comes from a caller that has the CacheServerProfile, which 
has the member ID. The updateLoad caller is a DistributedMessage which has a 
sender member that is the member ID. Lastly, removeServer caller has a 
CacheServerProfile as well we can again get the member ID.

-Jake

Request for Docker Hub permissions

[Dave Barnes]

Please grant me permission to upload Apache Geode artifats to Docker hub
using my ID: dbarnes97. Thanks!

Re: WAN replication issue in cloud native environments

[Nabarun Nag]

Hi Alberto,

Thank you for the contributions to the Apache Geode project

Here are a few feedback and pointers that we came up with:
1. Right now looking at your solution, we can see that you are modifying
the class "ServerLocation" which is used to stored as a key in the
connectionMap in LocatorLoadSnapshot.
2. ServerLocation was modified to include the memberID to differentiate
each server with the same hostname-for-sender and same pair of start and
end ports.
3. As ServerLocation is transmitted, there were a lot of changes in terms
of serialization etc. and also modification in ops code.

Suggestion:
- Instead, can we create a new class that contains the memberID and
ServerLocation and that new class object is added as a key in the
connectionMap.
- When member leaves, only that entry is removed from the connectionMap.
- When the remote locator is requesting the receiver information we
continue sending the ServerLocation, that we extract from the newly created
class.

Advantage:
- No changes required in terms of serialization as we are still sending the
ServerLocation like before.
- No changes to the ops.
- No extra bits sent over the wire


Please do let us know what do you feel about this solution.

Regards
Naba








On Tue, Jan 21, 2020 at 7:01 AM Alberto Bustamante Reyes
 wrote:

> Hi,
>
> I have been implementing a possible solution for this issue, and although
> I have not finished yet, I would like to kindly ask for comments.
>
> I created some Helm charts to explain and reproduce the problem, if you
> are interested they are here:
> https://github.com/alb3rtobr/geode-cloudnative-wan-replication
>
> The solution consists on adding to ServerLocation the id of the member
> hosting the server, to allow to differentiate two or more gateway receivers
> with the same ip but that are in different locations. I verified that this
> change fixes the problem.
>
> After that, I have been working on fixing issues with the existing tests.
> In the meanwhile, it will be useful to get some feedback about the
> solution, specially if there are impacts I have not considered yet (maybe
> they are the reason for the failing tests Im currently working on).
>
> The code can be found on this PR:
> https://github.com/apache/geode/pull/4489
>
> Thanks in advance!
>
> Alberto B.
>
>
> 
> De: Anilkumar Gingade 
> Enviado: viernes, 6 de diciembre de 2019 18:56
> Para: geode 
> Cc: Charlie Black 
> Asunto: Re: WAN replication issue in cloud native environments
>
> Alberto,
>
> Can you please file a JIRA ticket for this. This could come up often as
> more and more deployments move to K8s.
>
> -Anil.
>
>
> On Fri, Dec 6, 2019 at 8:33 AM Sai Boorlagadda 
> wrote:
>
> > > if one gw receiver stops, the locator will publish to any remote
> locator
> > that there are no receivers up.
> >
> > I am not sure if locators proactively update remote locators about change
> > in receivers list rather I think the senders figures this out on
> connection
> > issues.
> > But I see the problem that local-site locators have only one member in
> the
> > list of receivers that they maintain as all receivers register with a
> > single  address.
> >
> > One idea I had earlier is to statically set receivers list to locators
> > (just like remote-locators property) which are exchanged with gw-senders.
> > This way we can introduce a boolean flag to turn off wan discovery and
> use
> > the statically configured addresses. This can be also useful for
> > remote-locators if they are behind a service.
> >
> > Sai
> >
> > On Thu, Dec 5, 2019 at 2:33 AM Alberto Bustamante Reyes
> >  wrote:
> >
> > > Thanks Charlie, but the issue is not about connectivity. Summarizing
> the
> > > issue, the problem is that if you have two or more gw receivers that
> are
> > > started with the same value of "hostname-for-senders", "start-port" and
> > > "end-port" (being "start-port" and "end-port" equal) parameters, if one
> > gw
> > > receiver stops, the locator will publish to any remote locator that
> there
> > > are no receivers up.
> > >
> > > And this use case is likely to happen on cloud-native environments, as
> > > described.
> > >
> > > BR/
> > >
> > > Alberto B.
> > > 
> > > De: Charlie Black 
> > > Enviado: miércoles, 4 de diciembre de 2019 18:11
> > > Para: dev@geode.apache.org 
> > > Asunto: Re: WAN replication issue in cloud native environments
> > >
> > > Alberto,
> > >
> > > Something else to think about SNI based routing.   I believe Mario
> might
> > be
> > > working on adding SNI to Geode - he at least had a proposal that he
> > > e-mailed out.
> > >
> > > Basics are the destination host is in the SNI field and the proxy can
> > > inspect and route the request to the right service instance. Plus
> we
> > > have the option to not terminate the SSL at the proxy.
> > >
> > > Full disclosure - I haven't tried out SNI based routing myself and it
> is
> > > something that I thought 

Re: Request for Confluence edit permissions

[Dan Smith]

Hi Dave,

Done! You should have access now.

Thanks,
-Dan

On Tue, Jan 21, 2020 at 12:08 PM Dave Barnes  wrote:

> This is a request for permission to edit the Apache Geode Wiki on
> Confluence using my Apache credentials:
>   Username: dbarnes
>   Email: dbar...@apache.com
> I've been using another login to access confluence, but I'd like to be able
> to do it with my Apache identity.
> Thanks,
> Dave Barnes
> (committer & PMC member)
>

Request for Confluence edit permissions

[Dave Barnes]

This is a request for permission to edit the Apache Geode Wiki on
Confluence using my Apache credentials:
  Username: dbarnes
  Email: dbar...@apache.com
I've been using another login to access confluence, but I'd like to be able
to do it with my Apache identity.
Thanks,
Dave Barnes
(committer & PMC member)

Re: privacy protection

[Michael Oleske]

Something to consider is what scenarios do you want to protect against.
Full disk encryption protects against a drive pull attack, such as an
attack walks into a data storage room full disks, pulls a disk from the
room, and then makes a run for it.  Since the full disk is encrypted, the
attacker will have to break that.  Encrypting values protects against
insiders from seeing values, such as a rogue administrator that is trying
to get personally identifiable information (they would be able to decrypt
the disk, but wouldn't be able to decrypt the values).  I'm personally not
sure what Geode needs to do or could be doing versus what's best practice
when building applications that use Geode/when running Geode.

-michael

On Tue, Jan 21, 2020 at 10:33 AM Dan Smith  wrote:

> Hi Mario,
>
> It's something we've talked about before, I think it would be a good idea.
> The two workarounds I have seen done are
> - full disk encryption - storing the geode disk files on an encrypted disk
> - encrypting values  - At the client side, converting values into encrypted
> bytes before storing them in geode.
>
> But it would nice to just support encryption at the disk store level.
>
> -Dan
>
> On Tue, Jan 21, 2020 at 4:21 AM Udo Kohlmeyer  wrote:
>
> > Hi there Mario,
> >
> > The idea of encryption-at-rest, is something that has been on my radar
> > for at least 4yrs now.
> >
> > I would not mind having a chat about what your requirements are and how
> > we can get this into Geode.
> >
> > Looking forward to chatting to you about this.
> >
> > --Udo
> >
> > On 1/15/20 2:20 AM, Mario Kevo wrote:
> > > Hi geode-dev,
> > >
> > > Is it possible somehow to protect all files that containing user
> data(or
> > user data itself) being stored in disk for Geode.
> > > This includes all persistence data(OpLogs), backups and possible other
> > files containing user data.
> > > Also protection is needed for all of the files potentailly used for
> > replication and cluster high availability mechanism.
> > >
> > > If this feature is not available, do you have it in the plan already?
> > > Do you included it in Geode roadmap?
> > >
> > > BR,
> > > Mario
> > >
> > >
> >
>

Re: privacy protection

[Dan Smith]

Hi Mario,

It's something we've talked about before, I think it would be a good idea.
The two workarounds I have seen done are
- full disk encryption - storing the geode disk files on an encrypted disk
- encrypting values  - At the client side, converting values into encrypted
bytes before storing them in geode.

But it would nice to just support encryption at the disk store level.

-Dan

On Tue, Jan 21, 2020 at 4:21 AM Udo Kohlmeyer  wrote:

> Hi there Mario,
>
> The idea of encryption-at-rest, is something that has been on my radar
> for at least 4yrs now.
>
> I would not mind having a chat about what your requirements are and how
> we can get this into Geode.
>
> Looking forward to chatting to you about this.
>
> --Udo
>
> On 1/15/20 2:20 AM, Mario Kevo wrote:
> > Hi geode-dev,
> >
> > Is it possible somehow to protect all files that containing user data(or
> user data itself) being stored in disk for Geode.
> > This includes all persistence data(OpLogs), backups and possible other
> files containing user data.
> > Also protection is needed for all of the files potentailly used for
> replication and cluster high availability mechanism.
> >
> > If this feature is not available, do you have it in the plan already?
> > Do you included it in Geode roadmap?
> >
> > BR,
> > Mario
> >
> >
>

Call for presentations for ApacheCon North America 2020 now open

[Rich Bowen]

Dear Apache enthusiast,

(You’re receiving this message because you are subscribed to one or more 
project mailing lists at the Apache Software Foundation.)


The call for presentations for ApacheCon North America 2020 is now open 
at https://apachecon.com/acna2020/cfp


ApacheCon will be held at the Sheraton, New Orleans, September 28th 
through October 2nd, 2020.


As in past years, ApacheCon will feature tracks focusing on the various 
technologies within the Apache ecosystem, and so the call for 
presentations will ask you to select one of those tracks, or “General” 
if the content falls outside of one of our already-organized tracks. 
These tracks are:


Karaf
Internet of Things
Fineract
Community
Content Delivery
Solr/Lucene (Search)
Gobblin/Big Data Integration
Ignite
Observability
Cloudstack
Geospatial
Graph
Camel/Integration
Flagon
Tomcat
Cassandra
Groovy
Web/httpd
General/Other

The CFP will close Friday, May 1, 2020 8:00 AM (America/New_York time).

Submit early, submit often, at https://apachecon.com/acna2020/cfp

Rich, for the ApacheCon Planners

RE: WAN replication issue in cloud native environments

[Alberto Bustamante Reyes]

Hi,

I have been implementing a possible solution for this issue, and although I 
have not finished yet, I would like to kindly ask for comments.

I created some Helm charts to explain and reproduce the problem, if you are 
interested they are here: 
https://github.com/alb3rtobr/geode-cloudnative-wan-replication

The solution consists on adding to ServerLocation the id of the member hosting 
the server, to allow to differentiate two or more gateway receivers with the 
same ip but that are in different locations. I verified that this change fixes 
the problem.

After that, I have been working on fixing issues with the existing tests. In 
the meanwhile, it will be useful to get some feedback about the solution, 
specially if there are impacts I have not considered yet (maybe they are the 
reason for the failing tests Im currently working on).

The code can be found on this PR: https://github.com/apache/geode/pull/4489

Thanks in advance!

Alberto B.



De: Anilkumar Gingade 
Enviado: viernes, 6 de diciembre de 2019 18:56
Para: geode 
Cc: Charlie Black 
Asunto: Re: WAN replication issue in cloud native environments

Alberto,

Can you please file a JIRA ticket for this. This could come up often as
more and more deployments move to K8s.

-Anil.


On Fri, Dec 6, 2019 at 8:33 AM Sai Boorlagadda 
wrote:

> > if one gw receiver stops, the locator will publish to any remote locator
> that there are no receivers up.
>
> I am not sure if locators proactively update remote locators about change
> in receivers list rather I think the senders figures this out on connection
> issues.
> But I see the problem that local-site locators have only one member in the
> list of receivers that they maintain as all receivers register with a
> single  address.
>
> One idea I had earlier is to statically set receivers list to locators
> (just like remote-locators property) which are exchanged with gw-senders.
> This way we can introduce a boolean flag to turn off wan discovery and use
> the statically configured addresses. This can be also useful for
> remote-locators if they are behind a service.
>
> Sai
>
> On Thu, Dec 5, 2019 at 2:33 AM Alberto Bustamante Reyes
>  wrote:
>
> > Thanks Charlie, but the issue is not about connectivity. Summarizing the
> > issue, the problem is that if you have two or more gw receivers that are
> > started with the same value of "hostname-for-senders", "start-port" and
> > "end-port" (being "start-port" and "end-port" equal) parameters, if one
> gw
> > receiver stops, the locator will publish to any remote locator that there
> > are no receivers up.
> >
> > And this use case is likely to happen on cloud-native environments, as
> > described.
> >
> > BR/
> >
> > Alberto B.
> > 
> > De: Charlie Black 
> > Enviado: miércoles, 4 de diciembre de 2019 18:11
> > Para: dev@geode.apache.org 
> > Asunto: Re: WAN replication issue in cloud native environments
> >
> > Alberto,
> >
> > Something else to think about SNI based routing.   I believe Mario might
> be
> > working on adding SNI to Geode - he at least had a proposal that he
> > e-mailed out.
> >
> > Basics are the destination host is in the SNI field and the proxy can
> > inspect and route the request to the right service instance. Plus we
> > have the option to not terminate the SSL at the proxy.
> >
> > Full disclosure - I haven't tried out SNI based routing myself and it is
> > something that I thought could work as I was reading about it.   From the
> > whiteboard I have done I think this will do ingress and egress just fine.
> > Potentially easier then port mapping and `hostname for clients` playing
> > around.
> >
> > Just something to think about.
> >
> > Charlie
> >
> >
> > On Wed, Dec 4, 2019 at 3:19 AM Alberto Bustamante Reyes
> >  wrote:
> >
> > > Hi Jacob,
> > >
> > > Yes,we are using LoadBalancer service type. But note the problem is not
> > > the transport layer but on Geode as GW senders are complaining
> > > “sender-2-parallel : Could not connect due to: There are no active
> > > servers.” when one of the servers in the receiving cluster is killed.
> > >
> > > So, there is still one server alive in the receiving cluster but GW
> > sender
> > > does not know it and the locator is not able to inform about its
> > existence.
> > > Looking at the code it seems internal data structures (maps) holding
> the
> > > profiles use object whose equality check relies only on hostname and
> > port.
> > > This makes it impossible to differentiate servers when the same
> > > “hostname-for-senders” and port are used. When the killed server comes
> > back
> > > up, the locator profiles are updated (internal map back to size()=1
> > > although 2+ servers are there) and GW senders happily reconnect.
> > >
> > > The solution with the Geode as-is would be to expose each GW receiver
> on
> > a
> > > different port outside of k8s cluster, this includes creating N
> > Kubernetes
> > > services for 

Re: privacy protection

[Udo Kohlmeyer]

Hi there Mario,

The idea of encryption-at-rest, is something that has been on my radar 
for at least 4yrs now.


I would not mind having a chat about what your requirements are and how 
we can get this into Geode.


Looking forward to chatting to you about this.

--Udo

On 1/15/20 2:20 AM, Mario Kevo wrote:

Hi geode-dev,

Is it possible somehow to protect all files that containing user data(or user 
data itself) being stored in disk for Geode.
This includes all persistence data(OpLogs), backups and possible other files 
containing user data.
Also protection is needed for all of the files potentailly used for replication 
and cluster high availability mechanism.

If this feature is not available, do you have it in the plan already?
Do you included it in Geode roadmap?

BR,
Mario

Odg: privacy protection

[Mario Kevo]

Hi,

Just kindly reminder on this.

BR,
Mario

Šalje: Mario Kevo 
Poslano: 14. siječnja 2020. 16:20
Prima: dev@geode.apache.org 
Predmet: privacy protection

Hi geode-dev,

Is it possible somehow to protect all files that containing user data(or user 
data itself) being stored in disk for Geode.
This includes all persistence data(OpLogs), backups and possible other files 
containing user data.
Also protection is needed for all of the files potentailly used for replication 
and cluster high availability mechanism.

If this feature is not available, do you have it in the plan already?
Do you included it in Geode roadmap?

BR,
Mario

disable statistic archival

[Mario Kevo]

Hi,

We are trying to disable archiving statistic in the file by providing empty 
string to --statistic-archive-file. This option doesn't work.
>From the documentation it should work:
The file to which the running system member writes statistic samples. For 
example: “StatisticsArchiveFile.gfs”. An empty string disables archiving.
I opened ticket(GEODE-7714) 
and try to fix it, but without success.

As alter runtime command update properties and cache, it checks if any of these 
parameters change, but if we set this property to an empty string it failed 
with message
Please provide a relevant parameter(s).
We can change for this parameter that it can be an empty string but how this 
command works, it goes over all parameters and checks if it is changed. In that 
case if we provide something like
alter runtime --member=server it will be successiful but shouldn't as we didn't 
provide any parameter.

So the proposal is that we need to add a new parameter called 
--statistic-archiving-enabled  which can be true or false. In case it is true 
we need to provide also --statistic-archive-file.

Any thougths?

BR,
Mario