Re: [DISCUSS] Support For LTS Version Of Geode

[Michael Stolz]

I agree.

This is the most sensible way to achieve release alignment.


--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771

On Mon, Sep 30, 2019, 8:09 PM John Blum  wrote:

> Put simply, from my perspective, I would like to see LTS versions of Apache
> Geode align with the *Spring Data* (*Release Trains*) support for Apache
> Geode.
>
> For example:
>
> SDG Lovelace/2.1 is based on Apache Geode 1.6.x.
> SDG Moore/2.2 is based on Apache Geode 1.9.x.
>
> Therefore, both Apache Geode 1.6 and 1.9 would be LTS versions, with patch
> releases.
>
> The upcoming SD Neuman/2.3 (now in development given Moore has just went GA
> (i.e. 2.2.0.RELEASE) as of today), is currently based on 1.10, but is
> likely to move Apache Geode versions (e.g. 1.11, 1.12, or even 1.13) before
> SD Neuman reaches RC1.
>
> SD has longer lifecycles between release trains (1 to 1.5 years per SD
> Release Train) than Apache Geode's support cycle, on a particular
> major.minor version (e.g. 1.9), which always puts us in a
> precarious position.
>
> $0.02
> -John
>
>
>
> On Mon, Sep 30, 2019 at 3:55 PM Mark Bretl  wrote:
>
> > Hi All,
> >
> > It has come up a few times in recent weeks about the possibility of an
> LTS
> > version of Geode. Is this something the community would be interested in?
> >
> > There are advantages and disadvantages to supporting an LTS. Some
> > advantages may include:
> > - Stable release for downstream projects
> > - Include security and other maintenance related patches
> >
> > Disadvantages:
> > - Additional support for multiple distributions/versions
> > - Release management overhead
> >
> > Thoughts/Comments/Concerns?
> >
> > --Mark
> >
>
>
> --
> -John
> john.blum10101 (skype)
>

Re: [Proposal] Make gfsh "stop server" command synchronous

[Michael Stolz]

I understand that issue John, but masking it in an early return from the
shutdown command doesn't seem like the appropriate action.
Maybe we should consider that nearly all gfsh commands are not blocking,
and rather, have a way to determine which ones are still waiting for
completion?

--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771



On Tue, Sep 10, 2019 at 9:13 PM John Blum  wrote:

> @Anil-  I hear your argument when you are "scripting" with *Gfsh*, but
> blocking absolutely maybe less desirable when using *Gfsh* interactively.
> There are, after all, many non-cluster based commands.
>
> @Mark - I see.  I have generally found in my own testing purposes,
> especially automated, that a cache instance is not fully closed and has not
> properly released all it's resource even after cache.close() returns.
>
> -j
>
> On Tue, Sep 10, 2019 at 5:02 PM Mark Hanson  wrote:
>
> > Hi John,
> >
> > Kirk and I found that in our testing it was returning before it was fully
> > stopped. I have a change that seems viable that waits for the pid file to
> > disappear from the subdirectory of the server. I am not a fan. I would
> > prefer to wait for the pid to disappear, but that doesn’t seem like it
> will
> > be cross-platform friendly.
> >
> > Thanks,
> > Mark
> >
> > > On Sep 10, 2019, at 3:31 PM, John Blum  wrote:
> > >
> > > `stop server` is synchronous (with an option to break out of the wait
> > using
> > > CTRL^C) AFAIR.
> > >
> > > Way deep down inside, it simply relies on GemFireCache.close() to
> return
> > > (in-process).
> > >
> > > As Darrel mentioned, there is not "true" signal the the server was
> > > successfully stopped.
> > >
> > > -j
> > >
> > >
> > > On Tue, Sep 10, 2019 at 3:23 PM Darrel Schneider <
> dschnei...@pivotal.io>
> > > wrote:
> > >
> > >> I think it would be good for stop server to confirm in some way that
> the
> > >> server has stopped before returning.
> > >>
> > >> On Tue, Sep 10, 2019 at 3:08 PM Mark Hanson 
> wrote:
> > >>
> > >>> Hello All,
> > >>>
> > >>> I would like to propose that we make the gfsh “stop server” command
> > >>> synchronous. It is causing some issues with some tests as the rest of
> > the
> > >>> calls are blocking. Stop on the other hand immediately returns by
> > >>> comparison.
> > >>> This causes issues as shown in GEODE-7017 specifically.
> > >>>
> > >>> GEODE:7017 CI failure:
> > >>>
> org.apache.geode.launchers.ServerStartupValueRecoveryNotificationTest >
> > >>> startupReportsOnlineOnlyAfterRedundancyRestored
> > >>> https://issues.apache.org/jira/browse/GEODE-7017 <
> > >>> https://issues.apache.org/jira/browse/GEODE-7017>
> > >>>
> > >>>
> > >>> What do people think?
> > >>>
> > >>> Thanks,
> > >>> Mark
> > >>
> > >
> > >
> > > --
> > > -John
> > > john.blum10101 (skype)
> >
> >
>
> --
> -John
> john.blum10101 (skype)
>

Re: [DISCUSS] what region types to support in the new management rest api

[Michael Stolz]

I'm not at all sure why supporting the current set is more work than a
subset. Are we planning to fix issues in the current implementation in the
new API rather than the underlying (still needed) existing API? How is that
a good idea?



--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771

On Tue, Aug 20, 2019, 9:09 PM Anilkumar Gingade  wrote:

> My vote is for supporting all the region type currently supported. As mike
> was pointing, we have seen usecases where different regions are used for
> specific application needs.
>
>
>
> On Tue, Aug 20, 2019 at 5:09 PM Darrel Schneider 
> wrote:
>
> > gfsh create region currently does not support "distributed-no-ack" nor
> > "global". I did not find in jira a feature request for gfsh to support
> > these. So I think it would be safe for the Geode Management REST API to
> > also not support those scopes.
> >
> >
> > On Tue, Aug 20, 2019 at 12:10 PM Kirk Lund  wrote:
> >
> > > Here's my 2cents: The Geode Management REST API should definitely
> support
> > > "group" such that creation of a region may target zero, one, or more
> > > groups.
> > >
> > > On Tue, Aug 20, 2019 at 10:45 AM Darrel Schneider <
> dschnei...@pivotal.io
> > >
> > > wrote:
> > >
> > > > Is "group" support on the PCC roadmap or is the plan for the members
> > of a
> > > > cluster to always be uniform?
> > > >
> > > > On Tue, Aug 20, 2019 at 9:56 AM Jinmei Liao 
> wrote:
> > > >
> > > > > So, sound like we still need to support *PROXY types. It's OK to
> drop
> > > > > support for LOCAL* region types in management rest API?
> > > > >
> > > > > Also, regarding existing region shortcuts, we are also
> experimenting
> > > > using
> > > > > different object types to represent different types of region, for
> > > > example,
> > > > > redundantCopies property should only exists in partition regions.
> > > Instead
> > > > > of having a flat object that could have a type of any of these
> values
> > > and
> > > > > holds all sorts of properties that may/may not make sense for that
> > > type,
> > > > > should just have a factory method that given these region
> shortcuts,
> > we
> > > > > would return a specific region object that's determined by this
> type?
> > > > >
> > > > > On Tue, Aug 20, 2019 at 8:15 AM Jens Deppe 
> > wrote:
> > > > >
> > > > > > Currently, when deployed to the cloud (aka PCC) there is no
> ability
> > > > for a
> > > > > > user to group members thus it is also not possible to create
> > regions
> > > > (via
> > > > > > gfsh at least) that are separated by groups. Typically one would
> > > > create a
> > > > > > PROXY region against one group and the PARTITION region against
> > > another
> > > > > > group. However, without the ability to assign groups, that is not
> > > > > possible.
> > > > > >
> > > > > > --Jens
> > > > > >
> > > > > > On Tue, Aug 20, 2019 at 7:46 AM Michael Stolz  >
> > > > wrote:
> > > > > >
> > > > > > > I know that lots of folks use PROXY regions on the server side
> to
> > > > host
> > > > > > > logic associated with the region, but I think they always do
> that
> > > in
> > > > > > > conjunction with server groups so that the proxy is on some of
> > the
> > > > > server
> > > > > > > and the same region containing data is on others. Given the way
> > > > > cache.xml
> > > > > > > works they might not even bother with the server groups, but
> I'm
> > > not
> > > > > > sure.
> > > > > > >
> > > > > > > I think we should carry forward the existing shortcuts and not
> go
> > > > > > backward
> > > > > > > to the separate attributes.
> > > > > > >
> > > > > > > --
> > > > > > > Mike Stolz
> > > > > > > Principal Engineer, Pivotal Cloud Cache
> > > > > > > Mobile: +1-631-835-4771
> > > > > > >
> > > > &g

Re: [DISCUSS] Controlling event dispatch to AsyncEventListener (review by Aug 22)

[Michael Stolz]

Manual start has caused a lot of trouble over the years. We should
definitely circle back on those issues before traveling very far down this
road.

--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771



On Tue, Aug 20, 2019 at 11:56 AM Juan José Ramos  wrote:

> Hello Anil,
>
> +1 for the proposed solution.
> I'd change the method name from *pauseEventDispatchToListener* to something
> more meaningful and understandable for our users, maybe *startPaused*?,
> *setManualStart* (as we currently have for the *GatewaySenderFactory*)?,
> *startWithEventDispatcherPaused*?.
> Best regards.
>
>
>
> On Sat, Aug 17, 2019 at 12:55 AM Anilkumar Gingade 
> wrote:
>
> > I have updated the wiki based on Dan's comment.
> > Changes with api:
> >
> > *On "AsyncEventQueueFactory" interface - *
> >
> > *AsyncEventQueueFactory pauseEventDispatchToListener();  *// This causes
> > AEQ to be created with paused state.
> >
> >
> >
> >
> > On Fri, Aug 16, 2019 at 4:36 PM Anilkumar Gingade 
> > wrote:
> >
> > > Dan,
> > >
> > > If you look into the API; the AEQ will be created with the pause state.
> > > The user (application) has to call resume to dispatch the events.
> > >
> > > It will be slightly different from GatewaySender behavior; where
> > > GatewaySender will be created with run mode and then application has to
> > > call pause on it. Here in this case AEQ will be created with paused
> > state.
> > >
> > > -Anil.
> > >
> > >
> > > On Fri, Aug 16, 2019 at 4:31 PM Dan Smith  wrote:
> > >
> > >> Hi Anil,
> > >>
> > >> While I like the idea of matching the API of GatewaySender, I'm not
> > sure I
> > >> see how this solves the problem. Is it required of the user to call
> > pause
> > >> on the AsyncEventQueue as soon as it is created? How would someone do
> > that
> > >> when creating AEQs with xml or cluster configuration? Maybe it would
> be
> > >> better to not dispatch any events until we are done creating all
> > regions?
> > >>
> > >> -Dan
> > >>
> > >> On Fri, Aug 16, 2019 at 2:31 PM Anilkumar Gingade <
> aging...@pivotal.io>
> > >> wrote:
> > >>
> > >> > Proposal to support controlling capability with event dispatch to
> > >> > AsyncEventQueue Listener.
> > >> >
> > >> > Wiki proposal page:
> > >> >
> > >> >
> > >>
> >
> https://cwiki.apache.org/confluence/display/GEODE/%5BDraft%5D+Controlling+event+dispatch+to+AsyncEventListener
> > >> >
> > >> > Here is the details from the wiki page:
> > >> > *Problem*
> > >> >
> > >> > *The Geode system requires AEQs to be configured before regions are
> > >> > created. If an AEQ listener is operating on a secondary region, this
> > >> could
> > >> > cause listener to operate on a region which is not yet created or
> > fully
> > >> > initialized (for region with co-located regions) which could result
> in
> > >> > missing events or dead-lock scenario between region (co-located
> > region)
> > >> > creation threads. This scenario is likely to happen during
> persistence
> > >> > recovery; when AEQs are created in the start, the recovered AEQ
> events
> > >> are
> > >> > dispatched immediately, thus invoking the AEQ listeners.*
> > >> > Anti-Goals
> > >> >
> > >> > None
> > >> > *Solution*
> > >> >
> > >> > *The proposed solution is to provide a way to control dispatching
> AEQ
> > >> > events to the AEQ Listeners, this could be done by adding "pause"
> and
> > >> > "resume" capability to the AEQ, which will allow application to
> decide
> > >> when
> > >> > to dispatch events to the listeners. *
> > >> >
> > >> >
> > >> > *The proposal is similar to existing "pause" and "resume" behavior
> on
> > >> the
> > >> > GatewaySender, on which the AEQ is based on (AEQ implementation is a
> > >> > wrapper around GatewaySender). *
> > >> > Changes and Additions to Public Interfaces
> > >> >
> > >> > *The proposed APIs are:*
> > >> >
> > >> > *On "AsyncEventQueueFactory" interface - *
> > >> >
> > >> > *AsyncEventQueue pauseEventDispatchToListener();*
> > >> >
> > >> > *On "AsyncEventQueue" interface -*
> > >> >
> > >> > *boolean resumeEventDispatchToListener(); **returns true or false if
> > the
> > >> > event dispatch is resumed successfully.*
> > >> >
> > >> >
> > >> > *The constraints on the pauseEventDispatchToListener() will remain
> > >> similar
> > >> > to as in "GatewaySender.pause()" :*
> > >> >
> > >> > "It should be kept in mind that the events will still be getting
> > queued
> > >> > into the queue. The scope of this operation is the VM on which it is
> > >> > invoked. In case the AEQ is parallel, the AEQ will be paused on
> > >> individual
> > >> > node where this API is called and the AEQ on other VM's can still
> > >> dispatch
> > >> > events. In case the AEQ is not parallel, and the running AEQ on
> which
> > >> this
> > >> > API is invoked is not primary then primary AEQ will still continue
> > >> > dispatching events."
> > >> > Performance Impact
> > >> >
> > >> >
> > >> > *This will have similar performance and resource implication as with
> > the

Re: [DISCUSS] what region types to support in the new management rest api

[Michael Stolz]

I know that lots of folks use PROXY regions on the server side to host
logic associated with the region, but I think they always do that in
conjunction with server groups so that the proxy is on some of the server
and the same region containing data is on others. Given the way cache.xml
works they might not even bother with the server groups, but I'm not sure.

I think we should carry forward the existing shortcuts and not go backward
to the separate attributes.

--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771



On Mon, Aug 19, 2019 at 7:59 PM Darrel Schneider 
wrote:

> Keep in mind that the context of the regions in question is the cluster. So
> these regions would be created on servers.
> So, for example, does anyone see a need to create PROXY regions on the
> server? Even if we did not support them on the server, they would still be
> supported on clients.
>
>
> On Mon, Aug 19, 2019 at 4:26 PM Jinmei Liao  wrote:
>
> > Region type (in another word Region shortcut) defines a set of attributes
> > for a region. These are the list of region types we have:
> >
> > LOCAL,
> > LOCAL_PERSISTENT,
> > LOCAL_HEAP_LRU,
> > LOCAL_OVERFLOW,
> > LOCAL_PERSISTENT_OVERFLOW,
> >
> > PARTITION,
> > PARTITION_REDUNDANT,
> > PARTITION_PERSISTENT,
> > PARTITION_REDUNDANT_PERSISTENT,
> > PARTITION_OVERFLOW,
> > PARTITION_REDUNDANT_OVERFLOW,
> > PARTITION_PERSISTENT_OVERFLOW,
> > PARTITION_REDUNDANT_PERSISTENT_OVERFLOW,
> > PARTITION_HEAP_LRU,
> > PARTITION_REDUNDANT_HEAP_LRU,
> >
> > REPLICATE,
> > REPLICATE_PERSISTENT,
> > REPLICATE_OVERFLOW,
> > REPLICATE_PERSISTENT_OVERFLOW,
> > REPLICATE_HEAP_LRU,
> >
> > REPLICATE_PROXY,
> > PARTITION_PROXY,
> > PARTITION_PROXY_REDUNDANT,
> >
> > In region management rest api, especially in PCC world, we are wondering
> > 1) should we allow users to create LOCAL* regions through management rest
> > api?
> > 2) should we allow users to create *PROXY regions through management rest
> > api?
> > 3) for the rest of the PARTITION* and REPLICATE* types, should we strive
> to
> > keep the region type list the same as before, or only keep the type as
> > REPLICATE/PARTITION, but use other properties like "redundantCopy" and
> > "evictionAction" to allow different permutations of region attributes?
> >
> > comments appreciated!
> > --
> > Cheers
> >
> > Jinmei
> >
>

Re: Naming System for Regions - Active Jira Question

[Michael Stolz]

Changing the case sensitivity might break existing users' apps that use
Geode.


--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771

On Mon, Jul 15, 2019, 6:24 PM Jinmei Liao  wrote:

> I added the comment on this issue: Geode region names ARE case sensitive.
> Should not fix.
>
> On Mon, Jul 15, 2019 at 10:51 AM Alex Grisham  wrote:
>
> > Hello everyone,
> >
> > I’ve recently gotten into the Geode community and have been looking for a
> > good place to start contributing. I found this Jira regarding Duplicate
> > Region names (Geode-6286 <
> >
> https://issues.apache.org/jira/browse/GEODE-6286?jql=project%20=%20GEODE%20AND%20issuetype%20=%20Bug%20AND%20priority%20in%20(Minor,%20Low,%20Trivial)%20AND%20resolution%20=%20Unresolved%20ORDER%20BY%20priority%20ASC,%20updated%20DESC
> >)
> > and believe this would be a good place for me to start. Currently,
> creating
> > a region is case sensitive, so that “Region1” is different than
> “region1”.
> > However, this Jira is proposing that the check on region names not be
> case
> > sensitive, and only be focused on spelling. Before I begin working on
> this
> > Jira, I want to check to see if this case sensitive check was purposeful,
> > or if a spelling based check is preferred. I appreciate any
> clarification.
> >
> > Thanks!
> >
> > -Alex
>
>
>
> --
> Cheers
>
> Jinmei
>

Re: [PROPOSAL]: Improve OQL Method Invocation Security

[Michael Stolz]

One thing I will mention regarding DATA:READ:RegionName allowing query
behavior is that we have been asked by some users already to separate
DATA:READ:RegionName from DATA:QUERY:RegionName. This request is to protect
against arbitrary query execution by administrators that can cause huge
resource consumption.

So regardless of all the rest of the proposal, that's something we should
probably consider standardizing on.

--
Mike Stolz
Principal Engineer, Pivotal Cloud Cache
Mobile: +1-631-835-4771



On Thu, Jul 11, 2019 at 11:36 AM Juan José Ramos  wrote:

> Hello all,
>
> Friendly reminder regarding the deadline to rise concerns and/or objections
> regarding the *OQL Method InvocationSecurity Proposal [1]*, I'll go ahead
> and move it to *Development* on July 13th.
> Best regards.
>
> [1]:
>
> https://cwiki.apache.org/confluence/display/GEODE/OQL+Method+Invocation+Security#OQLMethodInvocationSecurity-PriorArt
>
>
> On Mon, Jul 8, 2019 at 3:29 PM Juan José Ramos  wrote:
>
> > Done [1]!.
> > Please remember that, if no major concerns arise before Friday this week,
> > I'll go ahead and move the proposal to *Development* on July 13th.
> > Best regards.
> >
> > [1]:
> >
> https://cwiki.apache.org/confluence/display/GEODE/OQL+Method+Invocation+Security#OQLMethodInvocationSecurity-PriorArt
> >
> > On Fri, Jul 5, 2019 at 3:48 PM Jacob Barrett 
> wrote:
> >
> >> Can you please add a Prior Art section to your proposal discussing these
> >> alternative solutions and why they are insufficient?
> >>
> >> Thanks,
> >> Jake
> >>
> >>
> >> > On Jul 5, 2019, at 10:41 AM, Juan José Ramos 
> wrote:
> >> >
> >> > Hello Jake,
> >> >
> >> > I've replied something similar *here [1]*.
> >> > Long story short, I haven't found anything that really applies to our
> >> use
> >> > case. The "most similar solution" is *Spring Method Security [2]*,
> which
> >> > basically implies annotating methods with explicit configuration about
> >> the
> >> > roles required to execute them. The same goes for *Shiro
> >> **Annotation-based
> >> > Authorization [3]*. The *AnnotationBasedMethodAuthorize**r [3]*
> approach
> >> > from the proposal is somewhat similar to this, but I've discarded it
> >> > because if forces the user to annotate classes with our own
> annotations,
> >> > basically forcing them to modify their domain model.
> >> > The proposal basically allows our users to use one of the default of
> the
> >> > box implementations and, if they don't like them for whatever reason,
> is
> >> > flexible enough so they can ultimately provide their own.
> >> > Hope this helps.
> >> > Cheers.
> >> >
> >> > [1]:
> >> >
> >>
> https://markmail.org/message/ekons7ixtz4jtf7n#query:+page:1+mid:snxgpsqd3yuppmsc+state:results
> >> > [2]:
> >> >
> >>
> https://docs.spring.io/spring-security/site/docs/5.1.5.RELEASE/reference/html/jc.html#jc-method
> >> > [3]:
> >> >
> >>
> https://shiro.apache.org/authorization.html#Authorization-AnnotationbasedAuthorization
> >> > [4]:
> >> >
> >>
> https://cwiki.apache.org/confluence/display/GEODE/OQL+Method+Invocation+Security#OQLMethodInvocationSecurity-AnnotationBasedMethodAuthorizer
> >> >
> >> > On Fri, Jul 5, 2019 at 1:46 PM Jacob Barrett 
> >> wrote:
> >> >
> >> >> So if we don’t want to use the Java built in SecurityManager to solve
> >> >> this, because we feel it's too big or too inflexible for our needs,
> >> have
> >> >> other projects implemented something we can borrow? We can’t be the
> >> first
> >> >> to need something like this if Java’s solution isn’t a good fit.
> >> >>
> >> >> Again I want to avoid inventing something new. What prior art is out
> >> there?
> >> >>
> >> >>
> >> >>> On Jul 4, 2019, at 1:29 PM, Juan José Ramos 
> >> wrote:
> >> >>>
> >> >>> Hello all,
> >> >>>
> >> >>> If you haven't added my email to the spam folder already :-), then
> I'd
> >> >> like
> >> >>> to let you know that I've update again the *Proposal [1]* and
> >> >> incorporated
> >> >>> most of the feedback provided, along with some additional
> information
> >> and
> >> >>> context I missed on the previous versions, thanks all that brought
> >> >> concerns
> >> >>> and suggestions to the discussion. Please take some time to review
> it
> >> >>> thoroughly, adding comments and/or concerns *only on this email
> >> thread*,
> >> >>> all feedback is more than welcome.
> >> >>> If no major concerns arise before July 12th 2019, I'll go ahead and
> >> mark
> >> >>> move the proposal to *Development* on July 13th.
> >> >>> Best regards.
> >> >>>
> >> >>> [1]:
> >> >>>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/GEODE/OQL+Method+Invocation+Security
> >> >>
> >> >>
> >> >
> >> > --
> >> > Juan José Ramos Cassella
> >> > Senior Technical Support Engineer
> >> > Email: jra...@pivotal.io
> >> > Office#: +353 21 4238611
> >> > Mobile#: +353 87 2074066
> >> > After Hours Contact#: +1 877 477 2269
> >> > Office Hours: Mon - Thu 08:30 - 17:00 GMT. Fri 08:30 - 16:00 GMT
> >> > How to upload artifacts:
> >> > https://sup

Re: [ANNOUNCE] Spring Boot for Apache Geode 1.0.0.RELEASE Available!

[Michael Stolz]

Congratulations John,

Great work as always.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771



On Mon, May 6, 2019 at 10:03 PM John Blum  wrote:

> It is my pleasure to announce the first GA release of Spring Boot for
> Apache Geode (SBDG) 1.0.0.RELEASE.
>
> See the official release announcement on spring.io for more details:
>
>
> https://spring.io/blog/2019/05/07/spring-boot-for-apache-geode-pivotal-gemfire-1-0-0-release-available
>
> Feedback appreciated and welcomed.
>
> Regards,
>
> --
> -John
>
>

Re: Copying pdx types via client

[Michael Stolz]

Cool! Thanks Jake!

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771



On Tue, Mar 26, 2019 at 2:42 PM Jacob Barrett  wrote:

>
>
> > On Mar 26, 2019, at 11:26 AM, Michael Stolz  wrote:
> >
> > I may be mistaken, but I think that this feature attempts to address
> cases where users want to selectively copy several items from one
> distributed system to another. Not sure if the two separate implementations
> of "ClientCache" actually achieves the same thing. Would like to discuss
> though.
> >
>
> It does support it and the PDX metadata would be correctly independent
> across both distributed systems. No issue of colliding PDX type IDs. It’s a
> much cleaner solution for that use case.
>
> -Jake
>
>

Re: Copying pdx types via client

[Michael Stolz]

I may be mistaken, but I think that this feature attempts to address cases 
where users want to selectively copy several items from one distributed system 
to another. Not sure if the two separate implementations of "ClientCache" 
actually achieves the same thing. Would like to discuss though.


> On Mar 26, 2019, at 2:07 PMEDT, Jacob Barrett  wrote:
> 
> Perhaps for others on this list that aren’t familiar with the context you 
> could fill in the blanks.
> 
> It is assumed you are asking about the “feature” where PDX metadata is 
> propagated from one connection pool to another in an attempt to support 
> multiple distributed systems in the same client. While the short answer is 
> yes, Geode Native (GN) goes further in not requiring one to share clients 
> across multiple distributed systems. You can create multiple independent 
> caches in the same process space with GN. Each of these instances can be 
> connected to independent distributed systems. The need to copy PDX metadata 
> goes away. The need to create multiple pools to different systems goes away. 
> All the pain and issues of supporting this “feature” just goes away. So if 
> there is someone out there wanting to do what was done before I would 
> strongly encourage to not and instead use independent caches.
> 
> -Jake
> 
> 
>> On Mar 26, 2019, at 9:50 AM, Michael Stolz  wrote:
>> 
>> Thanks. I'm happy to hear that.
>> 
>> 
>> --
>> Mike Stolz
>> Principal Engineer, GemFire Product Lead
>> Mobile: +1-631-835-4771
>> 
>> On Tue, Mar 26, 2019, 12:46 PM Blake Bender  wrote:
>> 
>>> Okay, after a conversation here at the office with folks who know, I can
>>> confirm the answer is "yes," NC has code to copy PDX types in the manner
>>> you're asking after.
>>> 
>>> Thanks,
>>> 
>>> Blake
>>> 
>>> 
>>> On Tue, Mar 26, 2019 at 8:43 AM Blake Bender  wrote:
>>> 
>>>> Hi Mike,
>>>> 
>>>> I'm not 100% certain what you're referring to, but... PdxSerializable in
>>>> NC has toData and fromData methods that allow you to copy objects via
>>>> serialization.  That what you're looking for?
>>>> 
>>>> Thanks,
>>>> 
>>>> Blake
>>>> 
>>>> 
>>>> On Mon, Mar 25, 2019 at 12:53 PM Michael Stolz 
>>> wrote:
>>>> 
>>>>> Does this functionality exist in the Native Client as well as the Java
>>>>> Client?
>>>>> 
>>>>> --
>>>>> Mike Stolz
>>>>> Principal Engineer, GemFire Product Lead
>>>>> Mobile: +1-631-835-4771
>>>>> 
>>>> 
>>> 
> 

Re: Copying pdx types via client

[Michael Stolz]

Thanks. I'm happy to hear that.


--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Tue, Mar 26, 2019, 12:46 PM Blake Bender  wrote:

> Okay, after a conversation here at the office with folks who know, I can
> confirm the answer is "yes," NC has code to copy PDX types in the manner
> you're asking after.
>
> Thanks,
>
> Blake
>
>
> On Tue, Mar 26, 2019 at 8:43 AM Blake Bender  wrote:
>
> > Hi Mike,
> >
> > I'm not 100% certain what you're referring to, but... PdxSerializable in
> > NC has toData and fromData methods that allow you to copy objects via
> > serialization.  That what you're looking for?
> >
> > Thanks,
> >
> > Blake
> >
> >
> > On Mon, Mar 25, 2019 at 12:53 PM Michael Stolz 
> wrote:
> >
> >> Does this functionality exist in the Native Client as well as the Java
> >> Client?
> >>
> >> --
> >> Mike Stolz
> >> Principal Engineer, GemFire Product Lead
> >> Mobile: +1-631-835-4771
> >>
> >
>

Copying pdx types via client

[Michael Stolz]

Does this functionality exist in the Native Client as well as the Java
Client?

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

Re: 1.9 release date

[Michael Stolz]

I think this is exactly the right balance. Yay!

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771



On Fri, Mar 1, 2019 at 11:48 AM Ryan McMahon  wrote:

> +1 to prioritizing quality over releasing on the desired cadence.  The
> quarterly release cadence is a good goal, but it shouldn't be a strict rule
> if there is more work to be done to ensure good quality.
>
> Ryan
>
> On Fri, Mar 1, 2019 at 8:02 AM Anthony Baker  wrote:
>
> > IMHO we start release work based on a quarterly schedule and we finish it
> > based on meeting quality goals.  So right now I’m less worried about when
> > the release will be done (because uncertainty) and more focused on
> ensuring
> > we have demonstrated stability on the release branch.  Hopefully that
> will
> > happen sooner than 4/1…but it could take longer too.
> >
> > Anthony
> >
> >
> > > On Feb 28, 2019, at 6:00 PM, Alexander Murmann 
> > wrote:
> > >
> > > Hi everyone,
> > >
> > > According to our wiki we were aiming for a March 1st release date for
> our
> > > 1.9 release. We cut the release branch about two weeks late and see
> > unusual
> > > amounts of merges still going into the branch. I propose that we give
> > > ourselves some more time to validate what's there. My proposal is to
> aim
> > > for last week of March or maybe even week of April 1st.
> > >
> > > What do you all think?
> >
> >
>

Re: Remove usages of deprecated classes

[Michael Stolz]

Good points, Ken.

I do think it is wise for us to begin a push to remove *USAGE* of
deprecated methods right at the time of deprecation, but the tests probably
should hang around until we're getting into the planning phase for the next
major.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771



On Wed, Jan 2, 2019 at 4:52 PM Ken Howe  wrote:

> I agree that we have too many uses of code deprecated in our own code
> base. I also agree with the idea that we should not introduce new usages of
> deprecated classes. So if someone is modifying a class that uses deprecated
> classes, should the deprecations be refactored out or is it OK to leave
> them? Seems hard to make firm “rules” (or should we call them guidelines?).
> Each case needs to be looked at with respect to code complexity, how
> extensive is the original code change and are the deprecated uses in the
> area of change, and will removing the deprecations require API changes to
> inject new dependencies to support removing the deprecations.
>
> We do have the guideline that deprecated classes can’t be removed until at
> least the major release after the release in which the class was
> deprecated. So currently we’re looking at Geode 2.0 to possibly remove
> deprecated classes.
>
> Now to get where it would be OK to remove a deprecated class in 2.0.0,
> then it seems to me that the initiative should be to remove all current
> uses of the class in the core product and in test code prior to when the
> decision is made to work toward a 2.0.0 release. In other words, do it
> sooner and avoid the rush to do it all just before the release.
>
> > On Jan 2, 2019, at 11:38 AM, Peter Tran  wrote:
> >
> > Hello Geode Dev,
> >
> > As a new contributor reviewing PRs I've learnt that it's acceptable to
> make
> > a PR that continues to use deprecated classes but not okay to introduce
> the
> > usage of a deprecated class.
> >
> > I wonder if there should be a systematic way to remove the usage of
> > deprecated classes. I'm concerned over time the code base will accumulate
> > more and more deprecated classes unless there is a system in place in
> which
> > they are removed.
> >
> > Has there been any initiatives in the past to do this? Are having a lot
> of
> > deprecated classes still in use a low risk thing?
> >
> > Thanks,
> >
> > Peter
> > --
> > Peter Tran
> > PCF Toronto
>
>

Re: Default branch for geode-examples

[Michael Stolz]

Consistency is best in my opinion.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Dec 12, 2018 9:59 AM, "Jacob Barrett"  wrote:

> Make the default “develop” to be consistent with the rest of our
> repositories. Put a statement in the https://urldefense.proofpoint.
> com/v2/url?u=http-3A__README.md&d=DwIFaQ&c=lnl9vOaLMzsy2niBC8-h_K-
> 7QJuNJEsFrzdndhuJ3Sw&r=us915XVL1kT8_Q-vmxAPAGcdk8eQkhBJKJGVCwFNFlA&
> m=2d4Fya8f3YbP-qfZHyIna7bxGf6bav_f6x5ygphVvmw&s=
> ATi5iKFzdX8ndZkkBDCbBHeidnwAGhMWjQaB3PMoI4k&e= that these examples for
> the 1.9.0-SNAPSHOT (develop). When cutting a release and merging to master
> change the say they are for 1.9.0 (rel/v1.9.0).
>
>
> > On Dec 12, 2018, at 9:07 AM, Anthony Baker  wrote:
> >
> > Alexander noticed that some recent PR’s against the geode-examples repo
> made against the master branch.  That breaks the gitflow approach where
> only released code is on master.  Should we update the default branch to be
> develop?
> >
> >
> > Anthony
> >
>
>

Re: Serializing Internal Classes

[Michael Stolz]

Jake thanks for bringing this up.
Could have been a big waste of effort if you hadnt.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Nov 30, 2018 8:51 AM, "Jacob Barrett"  wrote:

> Devs,
>
> Please be careful with how you are serializing internal classes between
> servers and clients. While you may think you are safe if you simply use
> DataSerializable you are wrong, very very wrong. If you define your
> internal class as DataSerializable then when serialized the fully qualified
> class name is serialized ahead of it to identify the class being
> serialized. This happens because there is no Instantiator for your internal
> class, so it can’t use a number and falls back to the class name. If you
> rename or move the class later you will run into backwards compatibility
> issues that have to be resolved by transforming the string name back and
> forth to the new name.
>
> DataSerializable should be reserved only for user defined classes. Please
> use DataSerializableFixedID for internal classes. This method uses a fixed
> integer to identify your class. Not only is it more efficient it also
> avoids issues with class or package renaming.
>
> Thanks,
> Jake
>
>

Re: Apache Geode Branch Housekeeping

[Michael Stolz]

The branch named "master" claims to be mine.
I don't exactly know how it became "mine".

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771



On Thu, Nov 8, 2018 at 7:55 PM Patrick Rhomberg 
wrote:

> Hello all!
>
>   We currently have 182 branches on the apache/geode repository.  This is
> excessive.
>   Please fork the main repository and save your branches there.  If you
> must use the apache/geode repository for some reason, please be diligent in
> your removal of dead branches.
>
>   *I would like to begin removing branches soon.*
>
>   Many branches appear to refer to JIRA tickets that are closed, or have
> pull requests against them that have been merged.  These seem safe to
> delete.
>   Many branches are thousands of commits behind develop or months to even
> years old.  Some of these branches have no commits since they were
> branched.  Release branches notwithstanding, these seem relatively safe to
> delete.  I suspect there are some that should be maintained for legal or
> legacy reasons, in which case please let me know.
>
>   This can be done most safely if you, as a responsible developer and
> reasonably-clean adult, were to pick up your own room.  If you have
> anything listed under "Your branches" at [1], please copy them to your fork
> and remove them from apache/geode.
>
> Thank you.
>
> Imagination is Change.
> ~ Patrick
>
> [1] https://github.com/apache/geode/branches/yours
>

Re: [VOTE] Time-based release schedule for minor releases

[Michael Stolz]

+1 Quarterly releases

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Oct 8, 2018 6:52 PM, "Sai Boorlagadda"  wrote:

> +1 for time-based minors (if patches are reserved for security fixes)
>
> On Mon, Oct 8, 2018 at 2:55 PM Anthony Baker  wrote:
>
> > We reserve the patch version number for when we want to issue a fix for a
> > security vulnerability or address a critical bug.  We did that with 1.1.1
> > and 1.2.1.
> >
> > I think the release schedule and SemVer are separate topics.  AFAICT this
> > proposal is just about putting a more deterministic schedule around what
> we
> > are already doing / have been doing.  Given that releases have
> significant
> > overhead, I think it makes sense to plan ahead.  If you want to know more
> > about that ask Naba, the last release manager :-)
> >
> >
> > Anthony
> >
> >
> > > On Oct 8, 2018, at 2:42 PM, Udo Kohlmeyer  wrote:
> > >
> > > -0
> > >
> > > It seems we have completely disregarded the *patch* version number.
> Does
> > this mean Geode versions will be *major*,*minor*? Can we then remove the
> > *patch* version on the release version?
> > >
> > > In addition to this, should the test coverage not be sufficient enough
> > to allow "release when green"? I must agree with @Jacob, I would prefer
> > something a lot less formalized. If  the community has contributed a
> > significant fix, should that not warrant an ad-hoc patch release? Or what
> > if the community has added functionality, that could "fill" a single
> minor
> > release by itself, should that not warrant a pre-emptive release.
> > >
> > > All these questions are not enough to warrant this effort to be
> blocked,
> > but I prefer those use cases to be considered for a more comprehensive
> > documentation effort, than what is currently on the wiki.
> > >
> > > In addition to that, is a release with only bug fixes in it, really
> > still a worthy of minor release number, or does it not count as a patch
> > release?
> > >
> > > --Udo
> > >
> > >
> > > On 10/8/18 14:27, Jacob Barrett wrote:
> > >> +0
> > >>
> > >> My preference is to release when there is something worth releasing
> > rather
> > >> than arbitrary points in time but I don't hold that preference
> strongly
> > >> enough to spike this effort.
> > >>
> > >> On Mon, Oct 8, 2018 at 2:24 PM Alexander Murmann  >
> > >> wrote:
> > >>
> > >>> Hi everyone,
> > >>>
> > >>> As discussed in "Predictable minor release cadence", I'd like us to
> > find
> > >>> agreement on releasing a new minor version every three months. There
> > are
> > >>> more details in the other thread and I should have captured
> everything
> > >>> relevant on the wiki:
> > >>> https://cwiki.apache.org/confluence/display/GEODE/Release+Schedule
> > >>>
> > >>> There are also some discussions about patch releases. Let's please
> > focus
> > >>> this vote on the proposed minor release schedule and carry on other
> > >>> discussions in the [DISCUSS] thread.
> > >>>
> > >>> Thank you all!
> > >>>
> > >
> >
> >
>

Re: [DISCUSS] Predictable minor release cadence

[Michael Stolz]

+1 on cutting in Nov.
Seems like community could benefit from one more release this year.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Oct 5, 2018 8:45 AM, "Anthony Baker"  wrote:

> I’ve been advocating for a fixed release schedule for a long time.  3
> months seems like a good rate given the release overhead.
>
> +1 on cutting the next release branch in November and shooting for an
> early December v1.8.0 release.
>
> Anthony
>
>
> > On Oct 4, 2018, at 6:48 PM, Sai Boorlagadda 
> wrote:
> >
> > I agree with Robert that we should release based on features that go in.
> >
> > I am not sure if Apache Kafka & Spark are a good reference for GEODE.
> > These tools were evolving rapidly in the last couple of years and
> frequent
> > releases would be good for a growing community.
> >
> > Should we do a patch release every few months to include bug fixes?
> >
> > Sai
> >
> >
> > On Thu, Oct 4, 2018 at 6:40 PM Robert Houghton 
> wrote:
> >
> >> I have found it refreshing that the geode released cadence has been
> based
> >> on features being done,  rather than a set schedule.
> >>
> >> I come from an environment where we had quarterly releases and monthly
> >> patches to all supported previous releases, and I can tell you that it
> >> became a grind. That being said, within that release cadence a one-month
> >> ramp for bug fixes on the release branch was almost always sufficient.
> >>
> >> On Thu, Oct 4, 2018, 18:32 Ryan McMahon  wrote:
> >>
> >>> +1 for scheduled releases and cutting the branch one month prior to
> >>> release. Given the time it took to fully root out, classify, and solve
> >>> issues with this 1.7 release, I think a month is the right amount of
> time
> >>> between cutting the branch and releasing.  If it ends up being too much
> >> or
> >>> too little, we can always adjust.
> >>>
> >>> I don’t feel strongly about the release cadence, but I generally favor
> >> more
> >>> frequent releases if possible (3 month over 6 month).  That way new
> >>> features can get into the hands of users sooner, assuming the feature
> >> takes
> >>> less than 3 months to complete.  Again, we can adjust the cadence if
> >>> necessary if it is too frequent or infrequent.
> >>>
> >>> Ryan
> >>>
> >>> On Thu, Oct 4, 2018 at 4:18 PM Alexander Murmann 
> >>> wrote:
> >>>
>  Anil, releasing every 3 months should give us 3 months of dev work.
> >> Don't
>  forget that there will be one month during which the next release is
>  already cut, but the vast majority of the work is going to the release
>  after that. So while we cut 1.7 one month ago and release 1.7 today,
> we
>  already have one month of work on develop again. It's not going to
> work
> >>> out
>  for this first release though, due to my suggestion to cut a month
> >> early
> >>> to
>  avoid holidays. If I recall correctly our past problem was that it
> took
>  longer to iron out issue on the release branch than expected. The
> >>> solution
>  to that would be to cut the release even earlier, but 1 month feels
> >> very
>  generous.
> 
>  On Thu, Oct 4, 2018 at 4:04 PM Anilkumar Gingade  >
>  wrote:
> 
> > If I remember from earlier discussion; the plan was to deliver a
> >>> release
> > once 3 months. But from the past release history we had difficulty in
> > achieving that, either the features are not completely ready or the
> > bug-fixes have taken more time. We need verify what is right for
> >> Apache
> > Geode, 3, 4 or 6 months; and there is any community dev/activity that
> > depends on Geode release.
> > My vote will be for 4 or 6 months, as it provides at least 3+ month
> >> for
>  dev
> > activity and 1 month for QA.
> >
> > -Anil.
> >
> >
> > On Thu, Oct 4, 2018 at 2:43 PM Dan Smith  wrote:
> >
> >> +1 I definitely like the idea of scheduled releases.
> >>
> >> I wonder if cutting the release branch a month ahead of time is
>  overkill,
> >> but I guess we do seem to keep finding issues after the branch is
> >>> cut.
> >>
> >> -Dan
> >>
> >> On Thu, Oct 4, 2018 at 1:25 PM Alexander Murmann <
> >>> amurm...@pivotal.io>
> >> wrote:
> >>
> >>> Hi everyone,
> >>>
> >>> I want to propose shipping Geode on a regular cadence. My
> >> concrete
> >> proposal
> >>> is to ship Geode every 3 months on the first weekday. To make
> >> sure
> >>> we
> > hit
> >>> that date we would cut the release 1 months prior to that day.
> >>>
> >>> *Why?*
> >>> Knowing on what day the release will get cut and on what day we
> >>> ship
> >> allows
> >>> community members to plan their contributions. If I want my
> >> feature
>  to
> > be
> >>> in the next release I know by when I need to have it merged to
>  develop
> >> and
> >>> can plan accordingly. As a user who is waiting for a particular
>  feature

Re: Queries on key fields

[Michael Stolz]

Usually key fields are faster
--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771



On Fri, Sep 28, 2018 at 2:49 PM siby_sekar  wrote:

> Which is more performant running queries on key fields or non key fields?
>
> we have an object in the form
>
>  sessionid :  { Listmessages}
>
> and message can be like {
>
> id1,
>
> sessionid,
>
> name
>
> }
>
> etc.
>
>
>
>
>
> --
> Sent from:
> http://apache-geode-incubating-developers-forum.70738.x6.nabble.com/
>

Re: [discuss] Should we evaluate at commit messages as part of PR review?

[Michael Stolz]

+1 to descriptive commit messages

Done well they will save time on every post commit access to that commit.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Sep 12, 2018 4:15 PM, "Alexander Murmann"  wrote:

> While our wiki page primarily calls out the 50/74 rule which is important,
> my bigger concern is in that I'd love to see commit messages that explain
> why a change was made. Sometimes that information is in the reference JIRA
> ticket, but not always. Especially with bug fixes we tend to not explain
> what actually caused the bug and how the code changes address it. It's also
> nice to minimize moving back and forth between editor and browsing JIRA and
> even better not to have to read 20+ messages long comment threads about a
> bug in JIRA to find out what the problem was. No hard rule can make sure we
> are doing that right, only human reviewers and care can. I don't think it's
> actually any extra work, since we are already reading the PR description
> and commit messages when we are doing a PR review. How else can you
> understand the PR? In fact better commit messages should save time here and
> not steal it.
>
> I totally agree that we don't always need a long text. "Fix typo in XYZ
> output" is totally fine. Both the why and what are obvious.
>
> On Wed, Sep 12, 2018 at 4:07 PM, Helena Bales  wrote:
>
> > I'm a fan of having useful commit messages. I would prefer this to be
> > another checkbox in our list of 6 things to do for Pull Requests as
> opposed
> > to something that is strictly enforced. There are cases where a simple
> > one-liner commit message is enough. I personally use commit messages
> often,
> > even when looking back at my own work. Additionally, I think it is a
> useful
> > exercise as it forces the dev to think back on the original problem and
> > think about how the solution addresses that.
> >
> > tl;dr -- +1 for commit messages
> >
> > ~Helena
> >
> > On Wed, Sep 12, 2018 at 11:50 AM, Pulkit Chandra 
> > wrote:
> >
> > > Have we thought about git hooks as a way to enforce policy
> > > https://git-scm.com/book/en/v2/Customizing-Git-An-Example-
> > > Git-Enforced-Policy
> > > ?
> > >
> > > *Pulkit Chandra*
> > >
> > >
> > > On Wed, Sep 12, 2018 at 2:46 PM Alexander Murmann  >
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > We have a wiki page
> > > >  > Commit+Message+Format
> > > >
> > > > that discusses why good commit messages matter and links to a even
> > better
> > > > article on the topic. In addition to what's described in those
> > documents,
> > > > better commit messages also would make it easier to have good PR
> > > messages.
> > > > Good commit and PR messages also provide more context to the reviewer
> > who
> > > > in turn now can do a better job at reviewing the pull request.
> > > >
> > > > Looking at our git log gives me the impression that we aren't always
> > > living
> > > > up to that standard. In fact we frequently aren't even close.
> > > >
> > > > I propose taking clear and well formatted commit messages into
> account
> > as
> > > > part of our PR review process. Lacking commit messages can be just as
> > bad
> > > > as bad naming in our code.
> > > >
> > > > Thoughts?
> > > >
> > >
> >
>

Re: [DISCUSS] Wrapping log calls in Conditionals like isDebugEnabled, isTraceEnabled, etc.

[Michael Stolz]

I doubt that the JIT can optimize the call out. That's what you're asking
it to do is optimize a method call away.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Mon, Sep 10, 2018 at 9:35 AM, Galen O'Sullivan 
wrote:

> I think that logging in hot paths/loops is probably something that should
> be avoided. And if it is necessary, I suspect that the JIT will
> short-circuit that call since debug levels don't generally change.
>
> There probably are a few hot paths that need to optimize logging, but
> that's not the majority.
>
> Can we agree to avoid this pattern in new code, since it adversely affects
> readability?
>
> Galen
>
>
> On Fri, Sep 7, 2018 at 1:46 PM, Nicholas Vallely 
> wrote:
>
> > I was just randomly looking into this a couple of days ago as a tangent
> to
> > 'observability' and came across this Stack Overflow:
> > https://stackoverflow.com/questions/6504407/is-there-a-
> need-to-do-a-iflog-
> > isdebugenabled-check
> > where the first answer is the most succinct in my opinion.
> >
> > In the geode code that I searched, we are not consistent with our use of
> > the if(statement) wrapper for debug, though most do have the wrapper.
> >
> > Nick
> >
> > On Fri, Sep 7, 2018 at 11:35 AM Jacob Barrett 
> wrote:
> >
> > > Checking with logger.isDebugEnabled() it still a good practice for hot
> > > paths to avoid the construction of intermediate object arrays to hold
> the
> > > var-args. Some logging libraries have fixed argument optimizations for
> > > var-args up to a specific limit. In very hot paths it is nice to
> > > check logger.isDebugEnabled() once into a temp boolean value and then
> > check
> > > that in all the subsequent debug logging statements in the hot path.
> > >
> > > -Jake
> > >
> > >
> > > On Fri, Sep 7, 2018 at 11:18 AM Dan Smith  wrote:
> > >
> > > > I think this pattern is a holdover from using log4j 1. In that
> version,
> > > you
> > > > added an if check to avoid unnecessary string concatenation if debug
> > was
> > > > enabled.
> > > >
> > > >
> > > >1. if (logger.isDebugEnabled()) {
> > > >2. logger.debug("Logging in user " + user.getName() + " with
> > > > birthday " + user.getBirthdayCalendar());
> > > >3. }
> > > >
> > > >
> > > > Log4j2 lets you pass a format string, so you can just do this:
> > > >
> > > > logger.debug("Logging in user {} with birthday {}", user.getName(),
> > > > user.getBirthdayCalendar());
> > > >
> > > >
> > > > These examples come from the log4j2 docs:
> > > >
> > > > https://logging.apache.org/log4j/2.0/manual/api.html
> > > >
> > > > -Dan
> > > >
> > > > On Fri, Sep 7, 2018 at 10:50 AM, Galen O'Sullivan <
> > gosulli...@apache.org
> > > >
> > > > wrote:
> > > >
> > > > > Hi all,
> > > > >
> > > > > I've noticed a pattern in Geode where we wrap a log call in a check
> > at
> > > > the
> > > > > same level, such as:
> > > > >
> > > > > if (logger.isDebugEnabled()) {
> > > > >   logger.debug("cleaning up incompletely started
> > > > > DistributionManager due to exception", r);
> > > > > }
> > > > >
> > > > > Is there any reason to do this? To my mind, it's an extra
> conditional
> > > > that
> > > > > should essentially be the first check inside `logger.debug(...)`
> > > anyways,
> > > > > and it complicates the code and makes it less readable. I've even
> > seen
> > > > > places in the code which have `if (logger.isDebugEnabled())
> > > > > logger.trace(...))` and such.
> > > > >
> > > > > I would like to propose that unless there is a compelling reason to
> > use
> > > > > this pattern, we remove all extra checks entirely.
> > > > >
> > > > > Galen
> > > > >
> > > >
> > >
> >
>

Re: [Proposal] SSL - hostname verification

[Michael Stolz]

I like this suggestion. +1

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Tue, Aug 14, 2018 at 1:00 PM, Anthony Baker  wrote:

> What if we require certs to have correct hostnames and automatically
> perform hostname verification.  No property required.  The only counter
> argument I can think of is that this change might require users to
> regenerate certificates when upgrading.  Perhaps we start by logging a
> warning for N releases, then make HN verification a hard requirement.
>
> Anthony
>
>
> > On Aug 14, 2018, at 8:59 AM, Jacob Barrett  wrote:
> >
> > On Tue, Aug 14, 2018 at 7:47 AM Sai Boorlagadda <
> sai_boorlaga...@apache.org>
> > wrote:
> >
> >> Geode currently does not implement hostname verification and is probably
> >> not required per TLS spec. But it can be supported on TLS as an
> additional
> >> check. The new proposed[1] implementation to use the default SSL context
> >> could cause certain man-in-the-middle attacks possible if there is no
> >> hostname verification.
> >
> >
> > The current SSL implementation is also susceptible to man-in-the-middle
> as
> > well. This proposal is really independent of those proposed changes.
> >
> >
> >> This is a proposal to add a new boolean SSL property
> >> `ssl-enable-endpoint-identification` which enables hostname
> verification
> >> for secure connections.
> >
> >
> > Are you proposing that we ship a custom trust manager that verifies hosts
> > on all TLS connections? I would rather shy away from yet another
> confusing
> > SSL property. Is there a proposed why for consumers to provide their own
> > trust manager and host verification process? If so, I assume that is yet
> > another property, can we merge those properties somehow?
> >
> > At this point with all theses system properties can we come up with a
> > better way to configure SSL?
> >
> > -Jake
>
>

Re: Spring Boot for Apache Geode 1.0.0.M1 Released!

[Michael Stolz]

Great stuff John!

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jun 26, 2018 12:42 AM, "John Blum"  wrote:

> Apache Geode Community-
>
> It is my please to announce the first milestone release of Spring Boot for
> Apache Geode, version 1.0.0.M1.
>
> See the official Spring Blog Post [1] for more details.
>
> Documentation [2] is available as are a couple of Examples [3].
>
> Feedback appreciated and welcomed.
>
> Much more to follow soon; stay tuned!
>
> Regards,
>
> --
> -John
>
> [1]
> https://spring.io/blog/2018/06/26/spring-boot-for-apache-
> geode-pivotal-gemfire-1-0-0-m1-released
> [2]
> https://spring.io/blog/2018/06/26/spring-boot-for-apache-
> geode-pivotal-gemfire-1-0-0-m1-released#documentation
> [3]
> https://spring.io/blog/2018/06/26/spring-boot-for-apache-
> geode-pivotal-gemfire-1-0-0-m1-released#examples
>

Re: Objections to removing 'upgrade offline-disk-store' command

[Michael Stolz]

I know for a fact there are still users who have not upgraded from 7.0.2
yet. Will this be a problem for them?

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Fri, May 25, 2018 at 12:51 PM, Jens Deppe  wrote:

> The gfsh command 'upgrade offline-disk-store' exists in the code in order
> to upgrade diskstores from pre-Apache days (GemFire 6.x and 7.x).
>
> I don't believe the command is relevant/necessary anymore and, unless there
> are objections, I'd like to go ahead and remove it.
>
> --Jens
>

Apache Geode Logo

[Michael Stolz]

Can someone who has sufficient mojo please update the Apache Geode logo on
geode.apache.org with the attached version with the TM added?



--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.

Re: [DISCUSS] Apache Geode 1.7.0 release branch created

[Michael Stolz]

+1

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Wed, May 23, 2018 at 12:24 PM, Barbara Pruijn  wrote:

> +1
>
> > On May 23, 2018, at 8:33 AM, Joey McAllister 
> wrote:
> >
> > +1 for including these. They are documentation-only changes that are
> > applicable to 1.7.
> >
> > On Wed, May 23, 2018 at 8:24 AM Karen Miller  wrote:
> >
> >> Geode devs,  I think that my merges of commits for GEODE-5071 and
> >> GEODE-5242 really
> >> belong in Geode 1.7.  They just missed making it in before the release
> >> branch was cut.  I'm going to
> >> cherry pick them into the 1.7 release branch.  If anyone disagrees with
> >> this, let's discuss why, and we
> >> can always revert the commits.  Thanks!
> >>
> >>
> >> On Mon, May 21, 2018 at 4:39 PM, Nabarun Nag  wrote:
> >>
> >>> Hello Geode dev community,
> >>>
> >>> We have created a release branch for Apache Geode 1.7.0 -
> "release/1.7.0"
> >>>
> >>> Please do review and raise any issue with the release branch.
> >>>
> >>> If no concerns are raised we will start with voting for release
> candidate
> >>> within a week.
> >>>
> >>> Regards
> >>> Nabarun Nag
> >>>
> >>
>
>

Re: geode.apache.org and GDPR

[Michael Stolz]

+1 to remove it

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Mon, May 21, 2018 at 11:31 AM, Greg Chase  wrote:

> Part of GDPR: Don’t collect or store data you don’t intend to use.
>
> +1 to remove it.
>
> This email encrypted by tiny buttons & fat thumbs, beta voice recognition,
> and autocorrect on my iPhone.
>
> > On May 21, 2018, at 8:29 AM, Anthony Baker  wrote:
> >
> > The geode website uses Google Analytics [1].  IANAL but there are some
> constraints that will apply when GPDR takes effect [2].  Specifically,
> users can opt-out of data collection.  Given how little use we’ve made of
> this data, I think it would be simpler to just remove the GA tracker from
> our website.
> >
> > Thoughts?
> >
> >
> > Anthony
> >
> > [1] https://github.com/apache/geode-site/blob/master/
> website/layouts/header.html#L40  geode-site/blob/master/website/layouts/header.html#L40>
> > [2] https://issues.apache.org/jira/browse/MSKINS-143 <
> https://issues.apache.org/jira/browse/MSKINS-143>
> >
>

Re: Quarterly report - DRAFT for your review

[Michael Stolz]

Looks fine to me

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Tue, May 8, 2018 at 1:46 PM, Nabarun Nag  wrote:

> LGTM
>
> Regards
> Naba
>
> On Tue, May 8, 2018 at 10:44 AM Joey McAllister 
> wrote:
>
> > Looks good to me, too. Thanks, Dave!
> >
> > On Tue, May 8, 2018 at 10:41 AM Mark Bretl  wrote:
> >
> > > Looks good! Thanks Dave!
> > >
> > > --Mark
> > >
> > > On Tue, May 8, 2018 at 10:13 AM, Dave Barnes 
> wrote:
> > >
> > > > Here's the near-final draft of our quarterly report. Responses
> received
> > > by
> > > > 5pm PDT today (May 8) will be incorporated.
> > > > Thanks,
> > > > Dave Barnes
> > > >
> > > > ## Description:
> > > >  - Apache Geode provides a database-like consistency model, reliable
> > > >transaction processing and a shared-nothing architecture to
> maintain
> > > >very low latency performance with high concurrency processing.
> > > >
> > > > ## Issues:
> > > >  - There are no issues requiring board attention at this time.
> > > >
> > > > ## Activity:
> > > >  - Two full product releases, Apache Geode versions 1.5 and 1.6
> > > >  - Community member Addison Huddy presented "What the Heck is an
> > > In-Memory
> > > > Data Grid?" at DataEngConf SF '18
> > > >  - Launched an investigation into adding PR status updates based on
> CI
> > > jobs
> > > > (see INFRA-16409).
> > > >  - In connection with SpringOne Platform, Pivotal, Inc., proposed to
> > > host a
> > > > Geode community event, prompting a discussion among the PMC regarding
> > > logo
> > > > use
> > > >
> > > > ## Health report:
> > > >  - Mailing lists remain active and productive.
> > > >  - JIRA tickets show that issues continue to be identified and
> > resolved.
> > > >  - We’re continuing to work on attracting new contributors and making
> > it
> > > > easier to participate in the community.
> > > >
> > > > ## PMC changes:
> > > >
> > > >  - Currently 46 PMC members.
> > > >  - New PMC members:
> > > > - Dick Cavender was added to the PMC on Tue Feb 20 2018
> > > > - Lynn Hughes-Godfrey was added to the PMC on Mon Feb 19 2018
> > > > - Lynn Gallinat was added to the PMC on Mon Feb 19 2018
> > > >
> > > > ## Committer base changes:
> > > >
> > > >  - Currently 87 committers.
> > > >  - No new committers added in the last 3 months
> > > >  - Last committer addition was Michael W. Dodge at Wed Jan 24 2018
> > > >
> > > > ## Releases:
> > > >
> > > >  - Apache Geode 1.5.0 was released on Fri Apr 06 2018. Highlights
> > > include:
> > > >
> > > > - Support for arithmetic operators in the WHERE clause of OQL
> > queries
> > > > - Expanded gfsh (cli) functionality in the form of new commands
> and
> > > > enhancements to existing commands
> > > > - REST interface improvements
> > > > - Improved deadlock prevention
> > > > - Improved client capabilities, including enhancements to load
> > > > balancing and subscription behavior
> > > > - 18 new features
> > > > - 270 enhancements and improvements
> > > > - 226 bug fixes
> > > >
> > > > A full list of issues that were resolved in Apache Geode 1.5 can be
> > found
> > > > at
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12318420&version=12342395
> > > >
> > > >  - Apache Geode 1.6.0 was released on Fri May 04 2018 under the
> > watchful
> > > > eye of Mike Stolz, in his debut as an Apache Geode Release Manager.
> > > > Highlights include:
> > > >
> > > > - Serialization refinements
> > > > - JDBC connector improvements
> > > > - gfsh (cli) commands to configure and manage JNDI bindings
> > > > - Cluster configuration enhancements
> > > > - 17 new features
> > > > - 139 enhancements and improvements
> > > > - 148 bug fixes
> > > >
> > > > A full list of issues that were resolved in Apache Geode 1.6 can be
> > found
> > > > at
> > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > projectId=12318420&version=12342867
> > > >
> > > >
> > > > ## Mailing list activity:
> > > >
> > > > Mailing lists have remained active and have maintained consistent
> usage
> > > > levels.
> > > >
> > > >  - dev@geode.apache.org:
> > > > - 182 subscribers (up 6 in the last 3 months):
> > > > - 818 emails sent to list (706 in previous quarter)
> > > >
> > > >  - iss...@geode.apache.org:
> > > > - 54 subscribers (up 0 in the last 3 months):
> > > > - 7028 emails sent to list (4960 in previous quarter)
> > > >
> > > >  - u...@geode.apache.org:
> > > > - 251 subscribers (up 12 in the last 3 months):
> > > > - 151 emails sent to list (228 in previous quarter)
> > > >
> > > >
> > > > ## JIRA activity:
> > > >
> > > >  - 612 JIRA tickets created in the last 3 months
> > > >  - 701 JIRA tickets closed/resolved in the last 3 months
> > > >
> > >
> >
>

Re: Reviewing our JIRA's

[Michael Stolz]

Good first step!

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Sun, May 6, 2018 at 10:38 AM, Anthony Baker  wrote:

> As a first step, I closed 30 issues that hadn’t been updated in 2 years.
>
> project = GEODE AND issuetype = Bug AND resolution = Unresolved AND
> (labels in (CI, Ci, ci, Flaky, flaky) OR summary ~ ci) and updated <= -104w
> ORDER BY created DESC, priority DESC, updated DESC
>
> Anthony
>
>
> > On Apr 26, 2018, at 6:24 PM, Lynn Hughes-Godfrey <
> lhughesgodf...@pivotal.io> wrote:
> >
> > Modifying your filter to look at jiras that haven't been updated in a
> year
> > (vs. created in the past year) ... there are 114 to review.
> > That probably means there were updates for 34 of those when they
> reproduced
> > in CI, etc, so we wouldn't want to close those.
> >
> > Looking specifically at GEODE-552 ... GEODE-640 was a duplicate of this
> and
> > has been marked closed (use port 0 so we use next available port vs.
> > default port) ... so really this one looks like a bookkeeping issue
> > (GEODE-552 should be closed as a duplicate of GEODE-640).
> > Same for GEODE-554 ... it is the same as GEODE-552, GEODE-640 (and also
> > open).
> >
> > I will probably take some more time tomorrow to look through the
> remaining
> > 112  to see if I can see any reason why we shouldn't just resolve
> them
> > now.
> > I will send you more feedback then.
> >
> >
> >
> >
> > On Thu, Apr 26, 2018 at 11:53 AM, Galen O'Sullivan <
> gosulli...@pivotal.io>
> > wrote:
> >
> >> I'm for it. Less noise is a good thing, and I don't think they're likely
> >> to get prioritized anyways. If we close as WONTFIX or similar, we can
> >> always look back for them later if we want.
> >>
> >>
> >>
> >> On 4/26/18 10:39 AM, Anthony Baker wrote:
> >>
> >>> Thanks Lynn!
> >>>
> >>> As I first step I’d like to focus on issues labeled as ‘CI’.  There are
> >>> 220 open issues and 148 [1] of those have been open for > 1 year.  If I
> >>> look at the metrics jobs [2, 3, 4] I see a clear mismatch between
> failures
> >>> that are currently relevant and our JIRA backlog.  That is, a bunch of
> >>> tests that used to fail don’t anymore.  Perhaps that’s because of the
> >>> transition away from Jenkins or something else, but it makes it hard to
> >>> figure out what is important.  GEODE-552 [5] is a good example—is this
> >>> still a problem and if so is it worth doing compared to more recent
> issues?
> >>>
> >>> So I’d like to make a radical proposal:  let’s close out all 148 of
> those
> >>> stale CI issues.  If a test failure recurs, we can always reopen the
> ticket.
> >>>
> >>> Why I think this is important:  I’ve noticed a few reports from users
> >>> that did not get timely attention and caused frustration.  I think
> reducing
> >>> the sheer volume of issues will help us focus on the most important
> issues.
> >>>
> >>> Let me know what you think.
> >>>
> >>> Thanks,
> >>> Anthony
> >>>
> >>> [1]https://issues.apache.org/jira/issues/?filter=12343689&jq
> >>> l=project%20%3D%20GEODE%20AND%20issuetype%20%3D%20Bug%20AND%
> >>> 20resolution%20%3D%20Unresolved%20AND%20(labels%20in%20(CI%
> >>> 2C%20Ci%2C%20ci%2C%20Flaky%2C%20flaky)%20OR%20summary%20~%
> >>> 20ci)%20and%20created%20%3C%3D%20%20-52w%20ORDER%20BY%
> >>> 20created%20DESC%2C%20priority%20DESC%2C%20updated%20DESC
> >>> [2] https://concourse.apachegeode-ci.info/teams/main/pipelines/d
> >>> evelop-metrics/jobs/GeodeDistributedTestMetrics/builds/66
> >>> [3] https://concourse.apachegeode-ci.info/teams/main/pipelines/d
> >>> evelop-metrics/jobs/GeodeIntegrationTestMetrics/builds/66
> >>> [4] https://concourse.apachegeode-ci.info/teams/main/pipelines/d
> >>> evelop-metrics/jobs/GeodeFlakyTestMetrics/builds/66
> >>> [5] https://issues.apache.org/jira/browse/GEODE-552
> >>>
> >>> On Apr 20, 2018, at 3:19 PM, Lynn Hughes-Godfrey <
>  lhughesgodf...@pivotal.io> wrote:
> 
>  I can help with that.
> 
> 
>  On Fri, Apr 20, 2018 at 1:46 PM, Anthony Baker 
>  wrote:
> 
>  I surfed through our JIRA backlog and cleaned up a bunch of old
> > issues—primarily issues that we missed resolving when the fix was
> > made.  In
> > some cases I asked for help determining if the issue should be
> closed.
> > If
> > you got one of these requests please try and follow up in the next
> week
> > or
> > so and close if needed.
> >
> > There are a number of issues remaining that probably deserve a deeper
> > review.  Some of these include:
> >
> > - Bugs that have insufficient detail and can’t be reproduced
> > - Tasks that may no longer be relevant
> > - Ideas that are good but we may never get around to doing them
> > - CI failures that no longer occur
> >
> > Ideally I’d like to close out issues where appropriate to make the
> > backlog
> > m

Re: [VOTE] Apache Geode 1.6.0 RC1

[Michael Stolz]

Yeah, looks like a copy and paste issue that caused the link to get wrapped
to a new line.
Sorry.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Thu, Apr 26, 2018 at 5:17 PM, Diane Hardman  wrote:

> Here is the correct link to the Release notes:
> https://cwiki.apache.org/confluence/display/GEODE/
> Release+Notes#ReleaseNotes-1.6.0
>
> On Thu, Apr 26, 2018 at 2:13 PM, Diane Hardman 
> wrote:
>
> > The link to the Release notes seems to be incorrect.
> >
> >
> >
> > On Thu, Apr 26, 2018 at 11:05 AM, Mike Stolz 
> wrote:
> >
> >> This is the first release candidate for Apache Geode, version 1.6.0.
> >> Thanks to all the community members for their contributions to this
> >> release!
> >>
> >> *** Please download, test and vote by Monday, April 30, 1500 hrs US
> >> Pacific. ***
> >>
> >> It fixes 157 issues. Release notes can be found at:
> >> https://cwiki.apache.org/confluence/display/GEODE/
> >> Release+Notes#ReleaseNotes-1.6.0.
> >>
> >> Note that we are voting upon the source tags: rel/v1.6.0.RC1
> >> https://github.com/apache/geode/tree/rel/v1.6.0.RC1
> >> https://github.com/apache/geode-examples/tree/rel/v1.6.0.RC1
> >>
> >> Commit ID:
> >> b4ba77f5131018d36b79608ef007dd3cbd761cd9 (geode)
> >> 45d174a1280e539108341b286ff79938f9729bc7 (geode-examples)
> >>
> >> Source and binary files:
> >> https://dist.apache.org/repos/dist/dev/geode/1.6.0.RC1
> >>
> >> Maven staging repo:
> >> https://repository.apache.org/content/repositories/orgapachegeode-1041
> >>
> >>
> >>
> >> Geode's KEYS file containing PGP keys we use to sign the release:
> >> https://github.com/apache/geode/blob/develop/KEYS
> >>
> >> Release Signed with Fingerprint:
> >>
> >> pub   rsa4096 2018-04-12 [SC] [expires: 2022-04-12]
> >>
> >>  876331B45A97E382D1BDFB820F9CABF4396F
> >>
> >
> >
>

Geode 1.6.0 RC1

[Michael Stolz]

The concourse pipeline is green.
Can I go ahead and start the build for RC1?

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.

Re: [Proposal]: behavior change when region doesn't exist in cluster configuration

[Michael Stolz]

Ok. Yes we do have to take the leap :)
Let's keep thinking that way.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.
<https://content.pivotal.io/ebooks/scaling-data-services-with-pivotal-gemfire>

On Thu, Apr 19, 2018 at 6:29 PM, Jinmei Liao  wrote:

> but this proposed change is one of the effort toward "deprecating
> cache.xml". I think we've got to take the leap at one point.
>
> On Thu, Apr 19, 2018 at 3:14 PM, Michael Stolz  wrote:
>
> > Hmmm...I think I liked the old behavior better. It was a kind of bridge
> to
> > cluster config.
> >
> > I still think we need to be putting much more effort into deprecating
> > cache.xml and much less effort into fixing the (possibly) hundreds of
> bugs
> > related to using both cache.xml and cluster configuration at the same
> time.
> > If we can make cluster config complete enough, and deprecate cache.xml,
> > people will stop using cache.xml.
> >
> >
> >
> > --
> > Mike Stolz
> > Principal Engineer, GemFire Product Lead
> > Mobile: +1-631-835-4771
> > Download the GemFire book here.
> > <https://content.pivotal.io/ebooks/scaling-data-services-
> > with-pivotal-gemfire>
> >
> > On Thu, Apr 19, 2018 at 5:58 PM, Jinmei Liao  wrote:
> >
> > > Scenario:
> > > a locator with cluster configuration enabled and a server started with
> a
> > > cache.xml that has /regionA defined and connected to this locator. So
> the
> > > initial state is the locator has an empty cluster configuration for the
> > > cluster, but the server has a region defined in it's cache.
> > >
> > > Old behavior:
> > > when user execute "create index --region=/regionA " command using
> > gfsh,
> > > the index creation is successful on the server, and the server returns
> a
> > > xml section that contains both  and  elements, CC is
> > updated
> > > with this xml, so the end result is: both region and index end up in
> the
> > > cluster configuration.
> > >
> > > Problem with old behavior:
> > > Not sure if the region is a cluster wide configuration. What if a
> region
> > > with the same name, but different type exists on different servers? the
> > xml
> > > returned by different server might be different.
> > >
> > > New behavior:
> > > when user execute "create index --region=/regionA " command using
> > gfsh,
> > > the index creation is successful on the server. We failed to find the
> > > region in the existing cluster configuration, so cluster configuration
> > will
> > > NOT be updated.
> > >
> > > I would also suggest that this would not apply to index alone, any
> > element
> > > inside region would have the same behavior change if we approve this.
> > >
> > > Thanks!
> > >
> > > --
> > > Cheers
> > >
> > > Jinmei
> > >
> >
>
>
>
> --
> Cheers
>
> Jinmei
>

Re: [Proposal]: behavior change when region doesn't exist in cluster configuration

[Michael Stolz]

Hmmm...I think I liked the old behavior better. It was a kind of bridge to
cluster config.

I still think we need to be putting much more effort into deprecating
cache.xml and much less effort into fixing the (possibly) hundreds of bugs
related to using both cache.xml and cluster configuration at the same time.
If we can make cluster config complete enough, and deprecate cache.xml,
people will stop using cache.xml.



--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the GemFire book here.


On Thu, Apr 19, 2018 at 5:58 PM, Jinmei Liao  wrote:

> Scenario:
> a locator with cluster configuration enabled and a server started with a
> cache.xml that has /regionA defined and connected to this locator. So the
> initial state is the locator has an empty cluster configuration for the
> cluster, but the server has a region defined in it's cache.
>
> Old behavior:
> when user execute "create index --region=/regionA " command using gfsh,
> the index creation is successful on the server, and the server returns a
> xml section that contains both  and  elements, CC is updated
> with this xml, so the end result is: both region and index end up in the
> cluster configuration.
>
> Problem with old behavior:
> Not sure if the region is a cluster wide configuration. What if a region
> with the same name, but different type exists on different servers? the xml
> returned by different server might be different.
>
> New behavior:
> when user execute "create index --region=/regionA " command using gfsh,
> the index creation is successful on the server. We failed to find the
> region in the existing cluster configuration, so cluster configuration will
> NOT be updated.
>
> I would also suggest that this would not apply to index alone, any element
> inside region would have the same behavior change if we approve this.
>
> Thanks!
>
> --
> Cheers
>
> Jinmei
>

re: 1.6.0 release

[Michael Stolz]

Yes please. I'm holding the build til this gets in. Please notify me here
when it's ready

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Apr 18, 2018 8:05 PM, "Bruce Schuchardt"  wrote:

> A couple of people have reported running into GEODE-5085 <
> https://issues.apache.org/jira/browse/GEODE-5085>, which prevents a
> server from rejoining the cluster if it gets kicked out and a
> SecurityManager has been configured.
>
> Is this something we could get into the 1.6 release?  The fix is a single
> commit and it's been through precheckin testing a few times now.
>
>
>

Re: Index on Region

[Michael Stolz]

Correct. We can "deprecate" any time we like as long as we have provided an
alternative, but we should only "remove" on a major release.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Tue, Apr 17, 2018 at 10:51 AM, Anthony Baker  wrote:

> Deprecation is a signal that a user should begin migrating to an
> alternative because that thing may be removed in a future release.
> Following the SemVer practice gives our users confidence that we won’t
> break stuff in a minor release.
>
> Anthony
>
>
> > On Apr 16, 2018, at 3:11 PM, Kirk Lund  wrote:
> >
> > We should probably target removal of the "deprecated" cache xsd
> > types/elements in Geode 2.0. I'm not sure if we can introduce
> cache-2.0.xsd
> > before Geode 2.0 or not (anyone know?).
> >
> > On Mon, Apr 16, 2018 at 2:59 PM, Jinmei Liao  wrote:
> >
> >> Simply searching "deprecated" in cache-1.0.xsd, we have 15 hits. Would
> it
> >> make sense to start creating a cache-2.0.xsd? or better yet, a
> >> server-cache-2.0.xsd and a client-cache-2.0.xsd?
> >>
> >>
> >> On Mon, Apr 16, 2018 at 2:55 PM, Patrick Rhomberg  >
> >> wrote:
> >>
> >>> Some types / fields have their deprecation noted in their
> documentation,
> >>> within the  block.  Alternatively / in addition,
> some
> >>> xsd:element have the block
> >>>
> >>> 
> >>>  deprecated
> >>> 
> >>>
> >>>
> >>> Although I don't know if these annotations count as "visible," but they
> >> are
> >>> there.
> >>>
> >>> On Mon, Apr 16, 2018 at 1:49 PM, Jinmei Liao 
> wrote:
> >>>
>  I don't think we have a process for deprecating elements in cache.xml
>  yet All the changes we've had so far are additions, not removal.
> 
>  The reason I am asking is that we are creating POJO's (started by JAXB
> >>> tool
>  from xsd file) that would generate the cluster configuration xml
>  automatically. As long as we know there are "new" ways to do things,
> we
>  should have these POJO's only generate XML that's in the new style.
> 
>  Thanks!
> 
>  On Mon, Apr 16, 2018 at 12:01 PM, Jason Huynh 
> >> wrote:
> 
> > Hi Jinmei,
> >
> > I am not sure whether these elements were deprecated or not.  I know
> >>> that
> > they were at one time valid and a user could specify the following in
>  their
> > app at one point:
> >
> > 
> >   
> > 
> >
> > I believe the "new" way to do this would be:
> >
> >  >> expression="ID"/>
> >
> > How would deprecation for this work? Would your roll a new version of
> > the new definition/scheme?
> >
> >
> > On Mon, Apr 16, 2018 at 10:28 AM Jinmei Liao 
> >>> wrote:
> >
> >> From the cache-1.0.xsd, we noticed that an index can have element
> >>> like
> >> "functional" and "primary-key", but the docs did not mention
> >> anything
> > about
> >> it (
> >>
> >> https://geode.apache.org/docs/guide/13/reference/topics/
> > cache_xml.html#region
> >> ).
> >> I am wondering if these are deprecated? Would it be better for the
> >>> the
> > xml
> >> created by the cluster configuration not consist any of these?
> >>
> >> 
> >>  
> >>
> >>  
> >>A functional type of index needs a from-clause, expression
> >> which are mandatory.
> >>The import string is used for specifying the type of Object
> >>> in
> >> the region or
> >>the type of Object which the indexed expression evaluates
> >> to.
> >>  
> >>
> >>
> >>   >>> use="required"
> > />
> >>    use="required"
> > />
> >>   >> use="optional"
> >>> />
> >>
> >>  
> >>
> >>  
> >>
> >>  
> >>A primary-key type of index needs a field attribute which
> >> is
> >> mandatory.
> >>There should be only one or zero primary-index defined for
> >> a
> > region
> >>  
> >>
> >>
> >>   >> />
> >>
> >>  
> >> 
> >>
> >>
> >>
> >> --
> >> Cheers
> >>
> >> Jinmei
> >>
> >
> 
> 
> 
>  --
>  Cheers
> 
>  Jinmei
> 
> >>>
> >>
> >>
> >>
> >> --
> >> Cheers
> >>
> >> Jinmei
> >>
>
>

Geode 1.6.0 release branch has been created

[Michael Stolz]

Can someone please create a concourse pipeline for this release?


--
Mike Stolz

Geode 1.6.0

[Michael Stolz]

Hi,

We are going to cut a release branch for 1.6.0 soon. Probably today or
tomorrow.

This is in keeping with our desire to get onto a monthly release cycle.

The Geode pipeline is green, so will not be an impediment to cutting this
branch.

There are a couple of minor issues that we might squeeze into this release,
so we may have a couple of changes that need to be done both on the release
branch and the develop branch.

--
Mike Stolz

Re: [DISCUSS] Release 1.6.0

[Michael Stolz]

I would really appreciate the opportunity to learn how to do the release
Manager role.

Would someone mentor me?

--
Mike Stolz

On Apr 6, 2018 6:57 PM, "Alexander Murmann"  wrote:

> Hi everyone!
>
> It's great that 1.5.0 just got released and we already have 108 changes
> ready for 1.6.0. I think we should get all those awesome changes to our
> users.
>
> What does everyone think about starting the release process for 1.6.0?
>
> Is anyone interested in taking on the release manager job for 1.6.0?
>

Re: [PROPOSAL]: deprecating gfsh command option --load-cluster-config-from-dir for 'start locator'

[Michael Stolz]

There could be. For instance, in a cloud environment, the working directory
could be on transient disk. The cluster configuration should, however,
always be on permanent disk.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Thu, Apr 5, 2018 at 4:22 PM, Jinmei Liao  wrote:

> +1 for deprecating "--load-cluster-configuration-dir" option in favor of
> "import cluster-configuration" for a more explicit intention.
>
> Also, since we are deprecating "load-cluster-configuration-dir", should
> we also deprecate "-cluster-config-dir" as well. Is there any need for
> users to persist the cluster configuration in a place other than the
> working dir?
>
> On Thu, Apr 5, 2018 at 12:16 PM, Sai Boorlagadda <
> sai_boorlaga...@apache.org> wrote:
>
>> All,
>>
>> Currently this option takes (true/false, defaults to false) to let
>> locator load cluster configuration from a specified directory provided with
>> other option '--cluster-config-dir'. Also `--cluster-config-dir` is used to
>> create the persistent disk store to store configuration region entries.
>>
>> There are couple of issues on how these two parameters work together. When
>> starting a locator to join an existing cluster -
>>
>> 1) if option `--load-cluster-config-from-dir` is set to true, then users
>> have to provide a directory to load the configuration, failing to provide
>> `--cluster-config-dir` would cause the newly started locator to wipe out
>> existing configuration if no configuration is found (`--cluster-config-dir`
>> defaults to locators working directory).
>>
>> 2) users have to leave `load-cluster-configuration-dir=false` when
>> specifying a directory for persistent region to be used. Otherwise the
>> existing configuration is wiped out by the newly started locator as it
>> finds the directory provided by `--cluster-config-dir` is empty. (here
>> user's intent to provide a specific directory is to create the disk store
>> and not to load configuration from it). Unlike issue #1, users does not
>> need to be cautious in this case as the default value for
>> load-cluster-configuration-from-dir is 'false'.
>>
>> Its not very intuitive how these two parameters work together, so in
>> favor of simplicity this command option will be deprecated and users can
>> rely on 'import cluster-configuration' command to import configuration
>> while bootstrapping a cluster.
>>
>> In the past this option has created few issues: GEODE-3237, GEODE-4218.
>>
>> Sai
>>
>
>
>
> --
> Cheers
>
> Jinmei
>

Re: [VOTE] Apache Geode 1.5.0.RC2

[Michael Stolz]

Mine was a +1 for both actually

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.
<https://content.pivotal.io/ebooks/scaling-data-services-with-pivotal-gemfire>

On Wed, Apr 4, 2018 at 4:08 PM, Jinmei Liao  wrote:

> +1 for release.
>
> On Wed, Apr 4, 2018 at 12:24 PM, Joey McAllister 
> wrote:
>
> > Mike: Just for clarification, is yours a +1 for the release or a +1 to
> > extending the vote?
> >
> > Thanks,
> > Joey
> >
> >
> >
> > On Wed, Apr 4, 2018 at 12:21 PM Dan Smith  wrote:
> >
> > > +1 to an extension of time
> > >
> > > +1 to this release - ran geode-release-check, looks good to me.
> > >
> > > -Dan
> > >
> > > On Wed, Apr 4, 2018 at 11:20 AM, Michael Stolz 
> > wrote:
> > >
> > > > +1
> > > >
> > > > --
> > > > Mike Stolz
> > > >
> > > > On Apr 4, 2018 2:06 PM, "Joey McAllister" 
> > > wrote:
> > > >
> > > > > +1 to Anthony's recommendation of extending the vote by 24 hours,
> > since
> > > > > some community members might have ignored this one, thinking that
> the
> > > > > negative vote would lead to a new release candidate/vote.
> > > > >
> > > > >
> > > > > On Wed, Apr 4, 2018 at 10:59 AM Anthony Baker 
> > > wrote:
> > > > >
> > > > > > Bruce updated his vote (retracting the -1) in a different thread.
> > > That
> > > > > > leaves us with one vote cast thus far which is not sufficient.  I
> > > > suggest
> > > > > > we extend the vote window an extra 24h to allow additional time
> to
> > > > review
> > > > > > and vote.
> > > > > >
> > > > > > Anthony
> > > > > >
> > > > > >
> > > > > > > On Apr 2, 2018, at 3:13 PM, Bruce Schuchardt <
> > > bschucha...@pivotal.io
> > > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > I vote -1 in light of GEODE-4989 <
> > > > > > https://issues.apache.org/jira/browse/GEODE-4989>. This issue
> can
> > > > cause
> > > > > > CQ queries to be slow or even hang.  I can see that the
> > serialization
> > > > > work
> > > > > > that caused this problem is in the release/1.5.0 branch.
> > > > > > >
> > > > > > >
> > > > > > > On 3/30/18 12:15 PM, Swapnil Bawaskar wrote:
> > > > > > >> Both issues mentioned in the RC1 vote thread
> > > > > > >> <
> > > > > > https://lists.apache.org/thread.html/
> > f5e21be3bb5985836a7931b43d992a
> > > > > f49fa6d7f079bdfa6aaf0fa0bb@%3Cdev.geode.apache.org%3E
> > > > > > >
> > > > > > >> have been resolved.
> > > > > > >>
> > > > > > >> This is the second release candidate for Apache Geode, version
> > > > 1.5.0.
> > > > > > >> Thanks to all the community members for their contributions to
> > > this
> > > > > > >> release!
> > > > > > >>
> > > > > > >> *** Please download, test and vote by Wednesday, April 4, 1200
> > hrs
> > > > > > >> US Pacific. ***
> > > > > > >>
> > > > > > >> It fixes 235 issues. release notes can be found at:
> > > > > > >>
> > > > > > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> > > > > projectId=12318420&version=12342395
> > > > > > >>
> > > > > > >> Note that we are voting upon the source tags: rel/v1.5.0.RC1
> > > > > > >> https://github.com/apache/geode/tree/rel/v1.5.0.RC2
> > > > > > >> https://github.com/apache/geode-examples/tree/rel/v1.5.0.RC2
> > > > > > >>
> > > > > > >> Commit ID:
> > > > > > >> 1be57f3d18e5a97dbcc784d6df7724f5f6ced39b (geode)
> > > > > > >> 4941f05c86d928949fbcdb3fb12295ccecc219eb (geode-examples)
> > > > > > >>
> > > > > > >> Source and binary files:
> > > > > > >> https://dist.apache.org/repos/dist/dev/geode/1.5.0.RC2
> > > > > > >>
> > > > > > >> Maven staging repo:
> > > > > > >> https://repository.apache.org/content/repositories/
> > > > > orgapachegeode-1039
> > > > > > >>
> > > > > > >>
> > > > > > >> Geode's KEYS file containing PGP keys we use to sign the
> > release:
> > > > > > >> https://github.com/apache/geode/blob/develop/KEYS
> > > > > > >>
> > > > > > >> Release Signed with Key: pub 4096R/18F902DB 2016-04-07
> > > > > > >> Fingerprint: E1B1 ABE3 4753 E7BA 8097 4285 8F8F 2BCC 18F9 02DB
> > > > > > >>
> > > > > > >
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> Cheers
>
> Jinmei
>

Re: 1.5.0 needs a bugfix

[Michael Stolz]

+1

I believe this means we have 3 or 4 +1 votes and zero -1 votes.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Apr 4, 2018 1:49 PM, "Bruce Schuchardt"  wrote:

> Well, I went to merge this into a 1.5.0 based branch and found that I was
> wrong.  The serialization changes that caused GEODE-4989 are /not/ in the
> release/1.5.0 branch.
>
> I'm taking back my negative vote on the 1.5.0 release candidate. Sorry for
> the alarm folks.
>
>
> On 4/4/18 9:49 AM, Bruce Schuchardt wrote:
>
>> I've checked in the fix for GEODE-4989 and it needs to be in the 1.5.0
>> release.  The SHA is 3c263e9220cd56486e3ec4d39cdd0a694482fb49
>>
>>
>

Re: [VOTE] Apache Geode 1.5.0.RC2

[Michael Stolz]

+1

--
Mike Stolz

On Apr 4, 2018 2:06 PM, "Joey McAllister"  wrote:

> +1 to Anthony's recommendation of extending the vote by 24 hours, since
> some community members might have ignored this one, thinking that the
> negative vote would lead to a new release candidate/vote.
>
>
> On Wed, Apr 4, 2018 at 10:59 AM Anthony Baker  wrote:
>
> > Bruce updated his vote (retracting the -1) in a different thread.  That
> > leaves us with one vote cast thus far which is not sufficient.  I suggest
> > we extend the vote window an extra 24h to allow additional time to review
> > and vote.
> >
> > Anthony
> >
> >
> > > On Apr 2, 2018, at 3:13 PM, Bruce Schuchardt 
> > wrote:
> > >
> > > I vote -1 in light of GEODE-4989 <
> > https://issues.apache.org/jira/browse/GEODE-4989>. This issue can cause
> > CQ queries to be slow or even hang.  I can see that the serialization
> work
> > that caused this problem is in the release/1.5.0 branch.
> > >
> > >
> > > On 3/30/18 12:15 PM, Swapnil Bawaskar wrote:
> > >> Both issues mentioned in the RC1 vote thread
> > >> <
> > https://lists.apache.org/thread.html/f5e21be3bb5985836a7931b43d992a
> f49fa6d7f079bdfa6aaf0fa0bb@%3Cdev.geode.apache.org%3E
> > >
> > >> have been resolved.
> > >>
> > >> This is the second release candidate for Apache Geode, version 1.5.0.
> > >> Thanks to all the community members for their contributions to this
> > >> release!
> > >>
> > >> *** Please download, test and vote by Wednesday, April 4, 1200 hrs
> > >> US Pacific. ***
> > >>
> > >> It fixes 235 issues. release notes can be found at:
> > >>
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12318420&version=12342395
> > >>
> > >> Note that we are voting upon the source tags: rel/v1.5.0.RC1
> > >> https://github.com/apache/geode/tree/rel/v1.5.0.RC2
> > >> https://github.com/apache/geode-examples/tree/rel/v1.5.0.RC2
> > >>
> > >> Commit ID:
> > >> 1be57f3d18e5a97dbcc784d6df7724f5f6ced39b (geode)
> > >> 4941f05c86d928949fbcdb3fb12295ccecc219eb (geode-examples)
> > >>
> > >> Source and binary files:
> > >> https://dist.apache.org/repos/dist/dev/geode/1.5.0.RC2
> > >>
> > >> Maven staging repo:
> > >> https://repository.apache.org/content/repositories/
> orgapachegeode-1039
> > >>
> > >>
> > >> Geode's KEYS file containing PGP keys we use to sign the release:
> > >> https://github.com/apache/geode/blob/develop/KEYS
> > >>
> > >> Release Signed with Key: pub 4096R/18F902DB 2016-04-07
> > >> Fingerprint: E1B1 ABE3 4753 E7BA 8097 4285 8F8F 2BCC 18F9 02DB
> > >>
> > >
> >
> >
>

Re: New APIs, read-serialized and CacheListeners, PostProcessor etc

[Michael Stolz]

in addition to not breaking existing callbacks, new getvalue mechanisms
should be created that support either getting the serialized or
deserialized value on the call itself rather than at the thread level or
the entire cache level.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Tue, Mar 27, 2018 at 3:06 AM, Galen O'Sullivan 
wrote:

> Hi all,
>
> We're working on a new protocol for Geode, which means wrapping Geode APIs
> and exposing them via the new protocol. For performance and to avoid
> class/serialization logic, we want to get these objects in serialized form.
> This means calling cache.setReadSerializedForCurrentThread(true) while
> we're operating.
>
> However, what side effects might this have on CacheListeners or other
> user-defined callbacks? Looking at the API for CacheListener, it's not
> clear whether EntryEvent.getOldValue respects the value of
> pdx-read-serialized or whether it always deserialized (since there is an
> associated getSerializedOldValue method here). There is also PostProcessor
> and may be other APIs that will be affected as well.
>
> So, how do we make sure we don't break callbacks? Is there an API I don't
> know about for getting entries in PDX serialized format (or whatever format
> they're currently in in the region)?
>
> Thanks,
> Galen
>

Re: Trying to cleanup dunit use of client cache

[Michael Stolz]

I haven't encountered a customer using a persistent region on a client in
many years. Maybe that entire construct should be deprecated on a
ClientCache.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Mar 9, 2018 3:39 PM, "Kirk Lund"  wrote:

> And I can't invoke createRegionFactory(RegionShortcut.REPLICATE_
> PERSISTENT)
> because ClientCache throws UnsupportedOperationException for it...
>
>   /**
>* @since GemFire 6.5
>*/
>   @Override
>   public  RegionFactory createRegionFactory(RegionShortcut
> shortcut) {
> if (isClient()) {
>   throw new UnsupportedOperationException("operation is not supported
> on a client cache");
> } else {
>   return new RegionFactoryImpl<>(this, shortcut);
> }
>   }
>
> ...which means you can only use deprecated code in the client unless you
> don't create a ClientCache.
>
> On Fri, Mar 9, 2018 at 3:32 PM, Kirk Lund  wrote:
>
> > I keep running into things that the dunit tests are doing in a client
> > cache that cannot be done with non-deprecated APIs.
> >
> > Is there a non-deprecated way to create a PERSISTENT_REPLICATE region in
> > a client cache?
> >
> >   AttributesFactory factory = new AttributesFactory();
> >   factory.setDataPolicy(DataPolicy.PERSISTENT_REPLICATE);
> >   factory.setPoolName(pool.getName());
> >   factory.setScope(Scope.DISTRIBUTED_ACK);
> >
> > ...compared to...
> >
> >   ClientRegionFactory crf = clientCacheRule.getClientCache().
> > createClientRegionFactory(ClientRegionShortcut.NOTHING_FITS);
> >   crf.setPoolName(pool.getName());
> >
> >
>

Re: [PROPOSAL]: concurrent bucket moves during rebalance

[Michael Stolz]

We should be very careful about how much resource we dedicate to
rebalancing.

One of our competitors rebalances *much* faster than we do, but in doing so
they consume all available resources.

At one bank that caused significant loss of incoming market data that was
coming in on a multicast feed, which had a severe adverse effect on the
pricing and risk management functions for a period of time. That bank
removed the competitor's product and for several years no distributed
caching was allowed by the chief architect at that bank. Until he left and
a new chief architect was named they didn't use any distributed caching
products. When they DID go back to using them, it pre-dated Geode, so they
used GemFire largely because GemFire does not consume all available
resources while rebalancing.

I do think we need to improve our rebalancing such that it iterates until
it achieves balance, but not in a way that will consume all available
resources.

--
Mike Stolz


On Thu, Mar 8, 2018 at 2:25 PM, Nick Reich  wrote:

> Team,
>
> The time required to undertake a rebalance of a geode cluster has often
> been an area for improvement noted by users. Currently, buckets are moved
> one at a time and we propose that creating a system that moved buckets in
> parallel could greatly improve performance for this feature.
>
> Previously, parallelization was implemented for adding redundant copies of
> buckets to restore redundancy. However, moving buckets is a more
> complicated matter and requires a different approach than restoration of
> redundancy. The reason for this is that members could be potentially both
> be gaining buckets and giving away buckets at the same time. While giving
> away a bucket, that member still has all of the data for the bucket, until
> the receiving member has fully received the bucket and it can safely be
> removed from the original owner. This means that unless the member has the
> memory overhead to store all of the buckets it will receive and all the
> buckets it started with, there is potential that parallel moving of buckets
> could cause the member to run out of memory.
>
> For this reason, we propose a system that does (potentially) several rounds
> of concurrent bucket moves:
> 1) A set of moves is calculated to improve balance that meet a requirement
> that no member both receives and gives away a bucket (no member will have
> memory overhead of an existing bucket it is ultimately removing and a new
> bucket).
> 2) Conduct all calculated bucket moves in parallel. Parameters to throttle
> this process (to prevent taking too many cluster resources, impacting
> performance) should be added, such as only allowing each member to either
> receive or send a maximum number of buckets concurrently.
> 3) If cluster is not yet balanced, perform additional iterations of
> calculating and conducting bucket moves, until balance is achieved or a
> possible maximum iterations is reached.
> Note: in both the existing and proposed system, regions are rebalanced one
> at a time.
>
> Please let us know if you have feedback on this approach or additional
> ideas that should be considered.
>

Re: Starter tickets

[Michael Stolz]

I'd love to tag all the pulse issues as newbie if we could.



On Fri, Mar 2, 2018 at 1:34 PM, Anthony Baker  wrote:

> Good idea!  We he have a few other labels relevant to this as well:
>
> - newbie
> - low-hanging-fruit
>
> Anthony
>
>
> > On Mar 2, 2018, at 10:03 AM, Alexander Murmann 
> wrote:
> >
> > Hi all,
> >
> > I think we could make it easier for people to find tasks that can be
> their
> > first contribution to Geode. The wiki page on how to contribute
> > 
> has a
> > list of suggested projects. Most of those are pretty ambitious and likely
> > would be overwhelming to first time contributors. There also is a link
> > to tickets
> > labeled with "Starter"
> >  3D%20Geode%20AND%20labels%20%3D%20starter%20AND%20status%
> 20in%20%28Open%2C%20%22In%20Progress%22%2C%20Reopened%29>.
> > I think the later is probably a much better angle for someone to start
> > contributing to Geode. Unfortunately there was less than a handful of
> > tickets labeled as starter tickets.
> >
> > My suggestion is to find tickets that strike a healthy balance between
> being
> >
> >   - simple enough that someone new to the code base can realistically
> take
> >   them on
> >   - rewarding enough that someone can feel a sense of accomplishment
> after
> >   having their PR merged
> >   - small enough that it can be accomplished in an evening or at most a
> >   long weekend
> >   - unlikely to result in conflicts with larger efforts that someone is
> >   already undertaking
> >
> > It would be great to have a variety of tickets that have different levels
> > of complexity and required effort to allow new contributors of varying
> > experience levels to start contributing meaningfully and maybe even
> support
> > somewhat of a progression curve for someone who wants to become a regular
> > contributor or even committer.
> >
> > I'd also suggest to not allow the list to grow too large. If the list
> gets
> > bigger than 20-30 tickets it might get too hard to keep a clear grasp on
> it
> > and the risk of having tickets that aren't relevant anymore increases.
> > Nothing would be more frustrating to a new contributor than investing
> their
> > personal time just to find out that it was wasted. So let's be mindful of
> > not using this as a dumping ground for everything we'd like to see, but
> > know we'll never get to.
> >
> > I already labeled a few more tickets as starter. Please feel free to
> > validate that I didn't violate my own suggestions in doing so.
> >
> > Thoughts?
>
>

Re: New PMC members - Lynn Gallinat, Lynn Hughes-Godfrey, and Dick Cavender

[Michael Stolz]

WELCOME!

--
Mike Stolz

On Wed, Feb 21, 2018 at 6:28 PM, Anthony Baker  wrote:

> Welcome all!
>
> > On Feb 21, 2018, at 1:50 PM, Dan Smith  wrote:
> >
> > We recently added a few new members to the Geode PMC. These three have
> all
> > been working on geode since it became an apache project, but through some
> > process oversight were never added to the PMC. Please join me in
> welcoming
> > these three to the PMC now!
> >
> > Lynn Gallinat
> > Lynn Hughes-Godfrey
> > Dick Cavender
> >
> > Thanks!
> > -Dan
>
>

Re: [DISCUSS] changes to registerInterest API

[Michael Stolz]

st passing behavior in
> > >>>>>>>>>> registerInterest()
> > >>>>>>>>>> - add registerInterestAllKeys();
> > >>>>>>>>>> - add registerInterest(T... keys) and
> > >>>> registerInterest(Iterable
> > >>>>>>>> keys)
> > >>>>>>>>>> and
> > >>>>>>>>>> not have one specifically for List or specific collections.
> > >>>>>>>>>>
> > >>>>>>>>>> The Iterable version would handle any collection type by
> > >>> having
> > >>>>>> the
> > >>>>>>>> user
> > >>>>>>>>>> pass in the iterator for the collection.
> > >>>>>>>>>>
> > >>>>>>>>>> On Fri, Nov 17, 2017 at 11:32 AM Jacob Barrett <
> > >>>>>> jbarr...@pivotal.io>
> > >>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>>> I am failing to see where registerInterest(List keys)
> > >> is
> > >>> an
> > >>>>>>> issue
> > >>>>>>>>> for
> > >>>>>>>>>>> the key type in the region. If our region is
> > >> Region
> > >>>>>> then I
> > >>>>>>>>> would
> > >>>>>>>>>>> expect registerInterest(List). If the keys are
> > >>> unknown
> > >>>>>> or a
> > >>>>>>>> mix
> > >>>>>>>>>>> then you should have Region and thus
> > >>>>>>>>>> registerInterest(List > >>>>>>>>>>>
> > >>>>>>>>>>> I echo John's statements on VarArgs and type erasure as
> > >> well
> > >>>> as
> > >>>>>> his
> > >>>>>>>>>>> argument for Iterable.
> > >>>>>>>>>>>
> > >>>>>>>>>>> Also, List does not restrict you from List indexes. The
> > >>>>>> region
> > >>>>>>>> would
> > >>>>>>>>>> be
> > >>>>>>>>>>> Region> with registerInterest > >>>>>> ing>>().
> > >>>>>>>>>>>
> > >>>>>>>>>>> -Jake
> > >>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>> On Fri, Nov 17, 2017 at 10:04 AM John Blum <
> > >>> jb...@pivotal.io>
> > >>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>>> Personally, I prefer the var args method
> > >>>>>> (registerInterest(T...
> > >>>>>>>>> keys))
> > >>>>>>>>>>>> myself.  It is way more convenient if I only have a few
> > >>> keys
> > >>>>>> when
> > >>>>>>>>>> calling
> > >>>>>>>>>>>> this method then to have to add the keys to a List,
> > >>>> especially
> > >>>>>>> for
> > >>>>>>>>>>> testing
> > >>>>>>>>>>>> purposes.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>> But, I typically like to pair that with a
> > >>>>>>>>> registerInterest(Iterable
> > >>>>>>>>>>>> keys) method
> > >>>>>>>>>>>> as well.  By having a overloaded Iterable variant, then
> > >> I
> > >>>> can
> > >>>>>>> pass
> > >>>>>>>> in
> > >>>>>>>>>> any
> > >>>>>>>>>>>> Collection type I want (which shouldn't be restricted to
> > >>>> just
> > >>>>>>>> List).
> > >>>>>>>>>> It
> > >>>>>>>>>

Re: [ANNOUNCE] Apache Geode 1.4.0

[Michael Stolz]

Super!

Congratulations

--
Mike Stolz

On Feb 5, 2018 8:36 PM, "Akihiro Kitada"  wrote:

> Awesome!
>
>
> --
> Akihiro Kitada  |  Staff Customer Engineer |  +81 80 3716 3736
> <+81%2080-3716-3736>
> Support.Pivotal.io   |  Mon-Fri  9:00am to
> 5:30pm JST  |  1-877-477-2269 <(877)%20477-2269>
> [image: support]  [image: twitter]
>  [image: linkedin]
>  [image: facebook]
>  [image: google plus]
>  [image: youtube]
> 
>
>
> 2018-02-05 19:10 GMT+09:00 Swapnil Bawaskar :
>
>> The Apache Geode community is pleased to announce the availability of
>> Apache Geode 1.4.0.
>>
>> Apache Geode is a data management platform that provides a database-like
>> consistency model, reliable transaction processing and a shared-nothing
>> architecture to maintain very low latency performance with high concurrency
>> processing.
>>
>> Geode 1.4.0 contains a number of improvements and bug fixes. This release
>> includes a simple JDBC connector which enables read-through and write
>> behind to any RDBMS without any custom code, enables indexing nested fields
>> for lucene indexes, optimizations to reduce memory footprint of compact
>> range indexes and optimizations to reduce CPU consumption by eviction
>> algorithm. Users are encouraged to upgrade to the latest release.
>>
>> For the full list of changes please review the release notes:
>> https://cwiki.apache.org/confluence/display/GEODE/Release+No
>> tes#ReleaseNotes-1.4.0
>>
>> The release artifacts can be downloaded from the project website:
>> http://geode.apache.org/releases/
>>
>> The release documentation is available at:
>> http://geode.apache.org/docs/guide/14/about_geode.html
>>
>> We would like to thank all the contributors that made the release
>> possible.
>>
>> Regards,
>> Swapnil Bawaskar on behalf of the Apache Geode team
>>
>
>

Re: C# native client problem

[Michael Stolz]

Also could be that he's hitting the Cache singleton. Hard to tell from what
he wrote, but I think he's attempting to new multiple Caches in the same
process.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Feb 6, 2018 9:35 AM, "Jacob Barrett"  wrote:

> Since there is no official release of the Geode Native C# client can you
> please tell us what SHA you have compiled?
>
> Quickly I can say you can’t create two regions with the same name in the
> same cache.
>
> -Jake
>
>
> > On Feb 6, 2018, at 1:30 AM, Rupert St John Webster <
> rup...@impress-solutions.com> wrote:
> >
> > Dear All,
> >
> > Per https://stackoverflow.com/questions/48628342/ I have a C# class
> Geode.cs that sets up a native client local cache, pool and region and does
> some work.
> >
> > I want to reuse the class and have 2 or more local caches that do work
> (ultimately to have different callbacks) but very simply like in the
> Program.cs
> >
> > new Geode();
> > new Geode();
> >
> > The 2nd time around to create a new local cache and find the server
> connection pool is fine, but to initialise a region at the following code:
> >
> > IRegion test = rf.SetPoolName("myPool").Create string>("test");
> >
> > I get the error:
> >
> > 
> >
> > I have attached a simple test program to show this. Please can you let
> me know what’s wrong?
> >
> > Thanks 😊
> >
> >
> > Kind regards
> >
> > Rupert St John Webster
> > FX Consultant
> > 
> > 
>

Re: MigrationClient and MigrationServer

[Michael Stolz]

Yeah we made a change to our diskstore format right around 6.0 that caused
customers to have to migrate all their persistent stores.
I don't think we have any plans to attempt any migrations from pre-6.0 any
more ;)

+1 for deleting these

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Mon, Feb 5, 2018 at 1:08 PM, Anilkumar Gingade 
wrote:

> Its used for data migration from one version to another...Don't see any
> mention of this in docs or any tools built around it.
>
> +1 for removal.
>
> -Anil.
>
>
> On Fri, Feb 2, 2018 at 4:38 PM, Kirk Lund  wrote:
>
> > I just ran across two classes in org.apache.geode.internal package:
> > * MigrationClient
> > * MigrationServer
> >
> > They have the javadoc @since GemFire 6.0.1 so I suspect these are no
> longer
> > useful or relevant. Can I delete these classes?
> >
> > Thanks,
> > Kirk
> >
>

Re: [ANNOUNCE] New committers and PMC members - Michael Dodge and Brian Rowe

[Michael Stolz]

Welcome Brian and Sarge!​

Re: Proposal: GEODE-4367 - Return PDXInstance when Domain Object can't be found

[Michael Stolz]

I should have said Geode 2.0 rather than GemFire 10...
Sorry. My GemFire roots are showing ;(

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.
<https://content.pivotal.io/ebooks/scaling-data-services-with-pivotal-gemfire>

On Wed, Jan 24, 2018 at 2:40 PM, Michael Stolz  wrote:

> I agree that we should have api level choice for whether to read
> serialized or not.
> We can deprecate read-serialized and get rid of it in GemFire 10.
>
> --
> Mike Stolz
> Principal Engineer, GemFire Product Lead
> Mobile: +1-631-835-4771 <(631)%20835-4771>
> Download the new GemFire book here.
> <https://content.pivotal.io/ebooks/scaling-data-services-with-pivotal-gemfire>
>
> On Wed, Jan 24, 2018 at 12:21 PM, Dan Smith  wrote:
>
>> Is this really just a workaround for the fact that the read-serialized
>> flag
>> applies to the whole cache? I can see that if you have mix of regions with
>> and without domain classes on the server you might want this feature. Can
>> you provide some more background on your use case?
>>
>> IMO we should get rid of read-serialized in favor of APIs that let the
>> user
>> decide whether they get a domain class or a PdxInstance.
>>
>> -Dan
>>
>> On Wed, Jan 24, 2018 at 9:58 AM, Galen O'Sullivan 
>> wrote:
>>
>> > Hi Addison,
>> >
>> > What kind of setup do you have that is causing you to have PDX
>> serialized
>> > objects that cannot be deserialized? Do you have classes that are
>> present
>> > on some servers but not others?
>> >
>> > This change would break the contract of region.get() , which is that it
>> > returns a value of a type that can be put into the region.
>> >
>> > Returning something that isn't what the user is expecting to be in the
>> > region would require users to check for PdxInstances every time they
>> get a
>> > value -- even though the type annotations say that you can't get a
>> > PdxInstance back (except for Region ).
>> >
>> > I think it would be better to have a second API that allows users to get
>> > and put PdxInstance objects in regions. That way, if they want to handle
>> > the exceptional case where they have a serialized object that can't be
>> > deserialized, they can catch the ClassNotFound exception and get the
>> > underlying PdxInstance.
>> >
>> > I do think that the possibility of a ClassNotFoundException should be
>> > documented in the Region API.
>> >
>> > Cheers,
>> > Galen
>> >
>> > On Tue, Jan 23, 2018 at 2:56 PM, Addison Huddy 
>> wrote:
>> >
>> > > Hi Geode Devs,
>> > >
>> > > I'm proposing the following change to how we handle deserialization
>> when
>> > > Domain Objects can't be found and pdx-serialize=false.
>> > >
>> > > https://issues.apache.org/jira/browse/GEODE-4367
>> > >
>> > > Looking forward to the discussion.
>> > >
>> > > \ah
>> > >
>> >
>>
>
>

Re: Proposal: GEODE-4367 - Return PDXInstance when Domain Object can't be found

[Michael Stolz]

I agree that we should have api level choice for whether to read serialized
or not.
We can deprecate read-serialized and get rid of it in GemFire 10.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Wed, Jan 24, 2018 at 12:21 PM, Dan Smith  wrote:

> Is this really just a workaround for the fact that the read-serialized flag
> applies to the whole cache? I can see that if you have mix of regions with
> and without domain classes on the server you might want this feature. Can
> you provide some more background on your use case?
>
> IMO we should get rid of read-serialized in favor of APIs that let the user
> decide whether they get a domain class or a PdxInstance.
>
> -Dan
>
> On Wed, Jan 24, 2018 at 9:58 AM, Galen O'Sullivan 
> wrote:
>
> > Hi Addison,
> >
> > What kind of setup do you have that is causing you to have PDX serialized
> > objects that cannot be deserialized? Do you have classes that are present
> > on some servers but not others?
> >
> > This change would break the contract of region.get() , which is that it
> > returns a value of a type that can be put into the region.
> >
> > Returning something that isn't what the user is expecting to be in the
> > region would require users to check for PdxInstances every time they get
> a
> > value -- even though the type annotations say that you can't get a
> > PdxInstance back (except for Region ).
> >
> > I think it would be better to have a second API that allows users to get
> > and put PdxInstance objects in regions. That way, if they want to handle
> > the exceptional case where they have a serialized object that can't be
> > deserialized, they can catch the ClassNotFound exception and get the
> > underlying PdxInstance.
> >
> > I do think that the possibility of a ClassNotFoundException should be
> > documented in the Region API.
> >
> > Cheers,
> > Galen
> >
> > On Tue, Jan 23, 2018 at 2:56 PM, Addison Huddy 
> wrote:
> >
> > > Hi Geode Devs,
> > >
> > > I'm proposing the following change to how we handle deserialization
> when
> > > Domain Objects can't be found and pdx-serialize=false.
> > >
> > > https://issues.apache.org/jira/browse/GEODE-4367
> > >
> > > Looking forward to the discussion.
> > >
> > > \ah
> > >
> >
>

Re: Addition of rmi-io library

[Michael Stolz]

There is going to be a 9.3.1 shortly after 9.3.0. Lets not hold 9.3.0 for
this.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jan 24, 2018 10:58 AM, "Jinmei Liao"  wrote:

yeah, Jens just found that out too. It's opening up a new port in either
server/server and gfsh/jmManager cases. I think he has a solution to it and
we will get it in soon.

On Wed, Jan 24, 2018 at 9:47 AM, Dan Smith  wrote:

> >
> > the content is going over the wire on whatever port that was port
before.
>
>
> From what I see, DownloadJarFunction is calling
> SimpleRemoteInputStream.export() which will call
> UnicastRemoteObject.exportObject. That's an RMI call to start a tcp server
> socket listening for connections to interact with that object.
>
> -Dan
>
> On Tue, Jan 23, 2018 at 6:15 PM, Jinmei Liao  wrote:
>
> > As far as I can see, we are utilizing the streaming capability provided
> by
> > the rmi-io, the content is going over the wire on whatever port that was
> > port before. When streaming content from the gfsh to the jmxManager,
it's
> > using the jmx port; when getting jars between locator/servers, it's
using
> > the FunctionService, so it's whatever communication channel that
> > FunctionService is using.
> >
> > All the FileContent are saved in temp folder, and get cleaned up after
> each
> > deployment.
> >
> > On Tue, Jan 23, 2018 at 3:17 PM, Dan Smith  wrote:
> >
> > > I don't have an issue with the dependency. But if we are opening up
new
> > > ports for RMI connections, that seems like a potential security risk.
> If
> > > someone has enabled cluster SSL we shouldn't be opening up an insecure
> > port
> > > for RMI connections.
> > >
> > > We should also make sure this is not leaking open sockets/file
> > decriptors.
> > > How does this SimpleRemoteInputStream we are creating get shutdown and
> > > cleaned up?
> > >
> > > -Dan
> > >
> > > On Tue, Jan 23, 2018 at 2:36 PM, Jens Deppe 
> > wrote:
> > >
> > > > Apologies that this was not raised earlier in discussion but I'm
> happy
> > to
> > > > describe it now.
> > > >
> > > > *Background:*
> > > >
> > > > When deploying jars into Geode they are moved through the system as
> > > simple
> > > > byte[] blobs. This obviously consumes memory. The various affected
> > areas
> > > > are:
> > > >
> > > > - gfsh reads the jars into memory
> > > > - the jars are pushed to the locator (via a jmx call) - again
> creating
> > a
> > > > byte[] blob on the locator
> > > > - from the locator, the jars are pushed to all servers via a
function
> > > call
> > > > (also sending the jars as byte[] blobs).
> > > >
> > > > Obviously if the jar is small this would not be a problem, however
in
> > > > memory constrained systems or with large jars this is obviously
going
> > to
> > > > put pressure on memory and possibly result in OOM situations. In
> fact,
> > > the
> > > > reason this came up was that some folks were unable to deploy a 40Mb
> > jar
> > > to
> > > > a 512Mb (heap) locator.
> > > >
> > > > *rmi-io:*
> > > >
> > > > After doing some research, it seemed that the ideal solution would
be
> > > > something that allows for serializing Input/OutputStreams. Java
> doesn't
> > > > provide anything natively.
> > > >
> > > > One library that stood out as being robust and feature complete was
> > > rmi-io
> > > > [1]. This allows for serializing a remote Input/OutputStream object
> > which
> > > > then lets us completely avoid having to pull deploying jars into
> memory
> > > > everywhere. Under the covers it uses RMI and allows for either
> > 'pulling'
> > > or
> > > > 'pushing' data. The reference page [2] has nice sequence diagrams.
> > > >
> > > > If anyone sees any issues with this, please do raise them. The
> current
> > > > usage of this has not changed any user-facing interaction so
> ultimately
> > > > changing the actual implemented fix for this problem (if we needed
> to)
> > > > would not have any external effect.
> > > >
> > > > Thanks
> > > > --Jens
> > > >
> > > > [1] http://openhms.sourceforge.net/rmiio
> > > > [2] http://openhms.sourceforge.net/rmiio/class_reference.html
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> >
> > Jinmei
> >
>



--
Cheers

Jinmei

Re: Spring Data for Apache Geode 2.0.3.RELEASE available!

[Michael Stolz]

Congratulations. Great job as always.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jan 24, 2018 10:35 AM, "John Blum"  wrote:

> Apache Geode Community-
>
> I am very pleased to follow-up to the announcement by Mark Paluch on the
> release of Spring Data Kay SR3.  This includes Spring Data for Apache Geode
> 2.0.3.RELEASE.
>
> The 3rd service release of SDG includes several import fixes and
> enhancements, particular around the new Annotation config model.  You can
> see a complete list of changes in the changelog (https://docs.spring.io/
> spring-data/geode/docs/2.0.3.RELEASE/changelog.txt).
>
> See the official announcement here (https://spring.io/blog/2018/
> 01/24/spring-data-ingalls-sr10-and-kay-sr3-released).
>
> Feedback welcomed.
>
> Happy coding!
>
> --
> -John
> john.blum10101 (skype)
>

Re: Spring Session for Apache Geode 2.0.0.RELEASE Available!

[Michael Stolz]

Congrats John

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Thu, Jan 18, 2018 at 2:16 PM, John Blum  wrote:

> Apache Geode Community-
>
> I am pleased to announce the General Availability (GA) of *Spring Session
> for Apache Geode* 2.0.0.RELEASE (SSDG), which is now available from Maven
> Central [1].
>
> SSDG 2.0 GA builds on...
>
> * Spring Framework 5.0.2.RELEASE
> * Spring Data for Apache Geode 2.0.2.RELEASE
> * Spring Session core 2.0.0.RELEASE
> * Apache Geode 1.2.1
>
> See the official announcement [2] for further details.
>
> Happy coding!
>
> Regards,
> --
> -John
>
> [1] http://search.maven.org/#search%7Cga%7C1%7Cspring-session
> [2] http://spring.io/blog/2018/01/16/spring-session-2-0-0-released
>

Re: Monitor the neighbour JVM using neihbour's member-timeout

[Michael Stolz]

Pardon my ignorance, but is this something that should be brought up on the
JGroups community?

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771
Download the new GemFire book here.


On Wed, Jan 17, 2018 at 2:37 AM, Aravind Musigumpula <
aravind.musigump...@amdocs.com> wrote:

> Hi Everyone,
>
> Consider a Geode cluster in which some nodes contain a particular type of
> data which is critical to the business and hosts a large amount of data.
> Some nodes may host data which is not critical to the business and hosts
> less amount of data compared to the previous type of nodes.
>
> If both the type of nodes are going through some operation which is making
> them unresponsive, the former type of node may take a couple of seconds
> extra than the later to respond.
>
> In this scenario is it fair to give the same member-timeout to all the
> members?
> What if we want to wait for a little longer time for such nodes.
>
> In the present configuration in geode, we cannot wait a little longer for
> some nodes when compared to do this although we can configure different
> member-timeout for all the nodes. But i think no one will ever configure
> different timeouts for each node because those member-timeouts will be used
> to monitor their neighbors.
>
> In this solution, we all do is wait for the suspected member-timeout
> instead of its own timeout during final check.
> It has no backward implications also, if somebody wants to use the
> existing behavior they will continue to use the same member-timeouts for
> all the nodes. So the behavior of the system is preserved.
>
> If you have any concerns in this solution, please let me know.
>
>
> Thanks,
> Aravind Musigumpula
>
>
> -Original Message-
> From: Aravind Musigumpula
> Sent: Monday, December 18, 2017 6:55 PM
> To: dev@geode.apache.org
> Subject: RE: Monitor the neighbour JVM using neihbour's member-timeout
>
> Hi Community,
>
> Can you please give your suggestions on the below solution.
>
> I have raised a pull request for the same : https://github.com/apache/
> geode/pull/1075 .
>
>
> Thanks,
> Aravind Musigumpula
>
> -Original Message-
> From: Aravind Musigumpula
> Sent: Friday, November 03, 2017 3:23 PM
> To: dev@geode.apache.org
> Subject: RE: Monitor the neighbour JVM using neihbour's member-timeout
>
> Thanks Bruce for suggestions, I will change the new variables from
> InternalDistributedMember to NetView and do changes related to backward
> compatibility.
>
> Now I know that there is another way that member can be removed from the
> view i.e if any member is sending a message and waits for
> ack-wait-threshold, if there is no response from the target the sender will
> do final check and remove it from the view if there is still no response.
> But I don't understand how deprecating the settings member-timeout,
> ack-wait-threshold, ack-severe-alert-threshold into one will solve the
> problem. The main problem is that we want a member to survive in the view
> for longer time than others.
>
> If we deprecate the settings into one setting and pass the setting to
> monitoring member(say A), then it will use the target member(say B which we
> want to survive in view for longer time) timeout for health monitoring and
> ack-wait-threshold to wait for the response for any message before doing
> final check.
> But what if some other member(say C) which is monitoring any other
> member(say D) have the member-timeout and ack-wait-threshold some smaller
> values. So if member C messages to B, C uses the smaller value of
> ack-wait-threshold(which is of member D) to get a response and does the
> final check again on basis of smaller member-timeout. So still member B can
> be kicked out of the view in small amount of time.
>
> I think this can be solved simply if we use the member-timeout of
> suspected member in the final check where we establish TCP connection. We
> don't need to club those three settings as well. We can set the
> member-timeout of a particular member to a higher value and the member
> which monitors it uses its own member-timeout as it is now, but during the
> final check it uses the suspected member-timeout(which is a greater value).
> The final check is common place in both the no heartbeat scenario and no
> response for a message scenario.
>
> Are there any concerns around this new proposal ?
>
>
> Thanks,
> Aravind Musigumpula
>
> -Original Message-
> From: Bruce Schuchardt [mailto:bschucha...@pivotal.io]
> Sent: Thursday, September 07, 2017 10:42 PM
> To: dev@geode.apache.org
> Subject: Re: Monitor the neighbour JVM using neihbour's member-timeout
>
> I think this might be an acceptable change though I doubt many people
> would find it useful.
>
> It's already possible to set different member-timeouts on each node of the
> distributed system but the meaning of the setting is the inverse of what's
> proposed here, 

Board Reports

[Michael Stolz]

It looks like board reports stopped being posted to the Geode wiki back in
July of 2016.

Where can I see the latest board reports now?

Re: [PDE East] Geode Conference - Double the Expected Attendance

[Michael Stolz]

This was really a great day in the history of Apache Geode.

We were concerned that we might not have any turn-out at all for this
session because it was disjointed from the rest of Spring 1 Platform. The
rest of the show starts tomorrow.

We were concerned that we had no way to measure the popularity because
there was no registration page.
We knew we had way too many great speaker submissions to turn down all but
7 of them in the main show starting Tuesday.
In fact we still had to turn down quite a few (including mine :)


Thank goodness the Spring 1 app has an "add to my schedule" feature.


We got wind a few days ago that there were 80+ "add to my schedule" hits.
The room was only configured to handle 65 people.

Jagdish worked with the folks running the conference to reconfigure the
room to accommodate about 170 people by changing the layout.

We filled all 170 seats 30 minutes before the session started. We
proactively moved briefcases off of seats, moved people to the sides of the
room and made every seat count. We still couldn't fit everybody.

I want to thank all the speakers, the PDE team, the event team and Jagdish
for all of the efforts that went into making this the best attended Geode
event ever. Clearly Geode is up and coming. More than double last year's
attendance. And we had to turn some away.

Let's start looking at how we want to continue to grow the community next
year. And lets not wait until December. We have lots of opportunity to grow
the Geode community in the early part of 2018.

Thanks all for the great work leading up to this fantastic outcome.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Dec 4, 2017 at 10:01 PM, Jagdish Mirani  wrote:

> Yes, we promoted the book. As Wes noted, we totally exceeded our
> expectations. ☺
>
> On Mon, Dec 4, 2017 at 1:33 PM, Marshall Presser 
> wrote:
>
>> Please promote Mike's book if appropriate.
>> MEP
>>
>> On Mon, Dec 4, 2017 at 4:16 PM, Wes Williams 
>> wrote:
>>
>>>
>>>
>>>
>>>
>>>
>>> Regards,
>>> Wes Williams
>>> Sent from mobile phone
>>>
>>
>>
>>
>> --
>> Marshall Presser
>> Pivotal Data Engineering
>> mpresser@pivotal .io
>> 240.401.1750 <(240)%20401-1750>
>>
>>
>
>
> --
> Regards
> Jag
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

Anything that breaks data on disk is also a big PITA. This change would
break data on disk.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Dec 4, 2017 at 1:52 PM, Dan Smith  wrote:

> The new protocol is currently translating from PDX->JSON before sending
> results to the clients so the client doesn't have to understand PDX or
> DataSerializable.
>
> There is a lot more to DataSerializable than just how a String is
> serialized. And it's not even documented that I am aware of. Just tweaking
> the string format is not going to make that much better. Your hypothetical
> ruby developer is in trouble with or without this proposed change.
>
> Breaking compatibility is a huge PITA for our users. We should do that when
> we are actually giving them real benefits. In this case if we were
> switching to some newer PDX format that was actually easy to implement
> deserialization logic I could see the argument for breaking compatibility.
> Just changing the string format without fixing the rest of the issues
> around DataSerializable isn't providing real benefits.
>
> You can't assume that a client in one language will only be serializing
> > strings for it's own consumption.
> >
>
> I wasn't making that assumption. The suggestion is that the C++ client
> would have to deserialize all 4 valid formats, but it could just always
> serialize data using the valid UTF-16 format. All other clients should be
> able to deserialize that.
>
> -Dan
>
> On Fri, Dec 1, 2017 at 5:35 PM, Jacob Barrett  wrote:
>
> > On Fri, Dec 1, 2017 at 4:59 PM Dan Smith  wrote:
> >
> > > I think I'm kinda with Mike on this one. The existing string format
> does
> > > seem pretty gnarly. But the complexity of implementing and testing all
> of
> > > the backwards compatibility transcoding that would be required in order
> > to
> > > move to the new proposed format seems to be way more work with much
> more
> > > possibility for errors. Do we really expect people to be writing new
> > > clients that use DataSerializable? It hasn't happened yet, and we're
> > > working on a new protocol that uses protobuf right now.
> > >
> >
> > Consider that any new clients written would have to implement all these
> > encodings. This is going to make writing new clients using the upcoming
> new
> > protocol laborious. The new protocol does not define object encoding, it
> > strictly defines message encoding. Objects sent over the protocol will
> have
> > to be serialized in some format, like PDX or data serializer. We could
> > alway develop a better serialization format than what we have now. If we
> > don't develop something new then we have to use the old. Wouldn't it be
> > nice if the new clients didn't have to deal with legacy encodings?
> >
> > If the issue is really the complexity of serialization from the C++
> client,
> > > maybe the C++ client could always write UTF-16 strings?
> > >
> >
> > You can't assume that a client in one language will only be serializing
> > strings for it's own consumption. We have many people using strings in
> PDX
> > to transform between C++, .NET and Java.
> >
> > The risk is high not to remove this debt. If I am developing a new Ruby
> > client I am forced to deal with all 4 of these encodings. Am I really
> going
> > to want to build a Ruby client for Geode, am I going to get these
> encodings
> > correct? I can tell you that getting them correct may be a challenge if
> the
> > current C++ client is any indication, it has a few incorrect assumptions
> in
> > its encoding of ASCII and modified UTF-8.
> >
> > I am fine with a compromise that deprecates but doesn't remove the old
> > encodings for a few releases. This would give time for users to update.
> New
> > clients written would not be be able to read this old data but could read
> > and write new data.
> >
> >
> >
> > -Jake
> >
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

It could always WRITE UTF-16 strings, but it will still need to be able to
read the others.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Fri, Dec 1, 2017 at 7:58 PM, Dan Smith  wrote:

> I think I'm kinda with Mike on this one. The existing string format does
> seem pretty gnarly. But the complexity of implementing and testing all of
> the backwards compatibility transcoding that would be required in order to
> move to the new proposed format seems to be way more work with much more
> possibility for errors. Do we really expect people to be writing new
> clients that use DataSerializable? It hasn't happened yet, and we're
> working on a new protocol that uses protobuf right now.
>
> If the issue is really the complexity of serialization from the C++ client,
> maybe the C++ client could always write UTF-16 strings?
>
> -Dan
>
> On Fri, Dec 1, 2017 at 4:17 PM, Michael Stolz  wrote:
>
> > My opinion is that risk/reward on this on is not worth it
> >
> > --
> > Mike Stolz
> > Principal Engineer - Gemfire Product Manager
> > Mobile: 631-835-4771
> >
> > On Dec 1, 2017 5:19 PM, "Jacob Barrett"  wrote:
> >
> > > On Fri, Dec 1, 2017 at 1:48 PM Michael Stolz 
> wrote:
> > >
> > > > There also would likely be Disk Stores that would need to be
> converted.
> > > > That would be real ugly too.
> > >
> > >
> > > Disk store could transcode each entry on demand or all at once on first
> > > load. Not saying it will be easy but progress rarely is.
> > >
> > > -Jake
> > >
> >
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

My opinion is that risk/reward on this on is not worth it

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Dec 1, 2017 5:19 PM, "Jacob Barrett"  wrote:

> On Fri, Dec 1, 2017 at 1:48 PM Michael Stolz  wrote:
>
> > There also would likely be Disk Stores that would need to be converted.
> > That would be real ugly too.
>
>
> Disk store could transcode each entry on demand or all at once on first
> load. Not saying it will be easy but progress rarely is.
>
> -Jake
>

Re: DISCUSS: Deprecating and replacing current serializable string encodings

[Michael Stolz]

There also would likely be Disk Stores that would need to be converted.
That would be real ugly too.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Dec 1, 2017 4:25 PM, "Bruce Schuchardt"  wrote:

> I'm wondering how we would maintain backward compatibility with this
> change?  Geode accepts serialized data from a client and keeps it in
> serialized form and might transmit this serialized data to an older version
> peer or client.  An older peer or client wouldn't be able to handle a new
> string encoding if it tried to access and deserialize the data.
>
> On 12/1/17 1:13 PM, Jacob Barrett wrote:
>
>> So what I would like to propose is that we deprecate all these methods and
>> replace them with standard UTF-8 prefixed with uint64 length. It is
>> preferable that the length be run length encoded to reduce the overhead of
>> encoding small strings. Why such a large length, well consider that
>> different languages have different limits as well as Java stores strings
>> internally as UTF-16.
>>
>
>

Font for geode.apache.org

[Michael Stolz]

Does anybody know what font was used for the tagline on geode.apache.org?

--
Mike Stolz

Re: Next release: 1.4.0

[Michael Stolz]

I'd like to see it include the recently completed work on indexing nested
objects in the Apache Lucene integration.
Early Dec sounds good to me.

--
Mike Stolz


On Wed, Nov 22, 2017 at 4:48 PM, Anthony Baker  wrote:

> We released Geode 1.3.0 at the end of October.  Our next release will be
> 1.4.0.  Questions:
>
> 1) Who wants to volunteer as a release manager?
> 2) What do we want to include in the release?
> 3) When do we want to do this?
>
> IMO, let's should shoot for an early Dec release.
>
> Anthony
>
>

Re: [Discussion] Geode-Native Removing Stats from Public API

[Michael Stolz]

I'm not clear on why we are removing stats gathering capability.
Do we know that customers aren't using this?
Is it badly broken?

What is actually driving this work?

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Nov 20, 2017 at 11:42 AM, Bruce Schuchardt 
wrote:

> Should this be done for the Java caches as well?
>
>
> On 11/17/17 11:48 AM, David Kimura wrote:
>
>> I agree, a statistics interface seems beyond the scope of Geode Native
>> client responsibility.  Hiding or removing seems appropriate to me.
>>
>> Thanks,
>> David
>>
>> On Fri, Nov 17, 2017 at 11:29 AM, Ernest Burghardt
>>  wrote:
>>
>>> +1 for removal
>>>
>>> On Thu, Nov 16, 2017 at 1:46 PM, Jacob Barrett 
>>> wrote:
>>>
>>> I want to open a discussion regarding the removal of StatisticsFactory
 and
 related APIs from the public API. I can't see that we would want the
 Geode
 Native client to be a first class statistics/metrics gathering API.
 There
 are plenty of other first class players in this space. If this isn't a
 feature of the client then I suggest it be moved internally. It’s highly
 unlikely it’s being used but in the case that it is we can consider
 moving
 it back after some serious refactoring as it relies on an over
 abundance of
 raw pointers. Rather than spend time refactoring it now let’s just hide
 it
 away.

 -Jake





>

Re: [DISCUSS] changes to registerInterest API

[Michael Stolz]

I don't like the vararg option.
If i'm maintaining a list of keys i'm interested in, I want to be able to
pass that List in.
Varargs is a poor substitute. It might even cause problems of pushing in
multiple different types. Keys must all be of one type for a given Region.


I'm very much in favor of deprecating the ALL_KEYS string in favor of
something that is typed specially if you refer to ALL_KEYS.


If that works, then we don't necessarily need the additional API
registerInterestAllKeys(). But if ALL_KEYS can't be a special type to get
over the compilation issues then we should go with the new API.



--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Thu, Nov 16, 2017 at 7:02 PM, Anilkumar Gingade 
wrote:

> +1 Deprecating ALL_KEYS option; I believe this is added before we supported
> regex support.
>
>  Doesn't seems like a new API is needed. The regex java doc clearly
> specifies the effect of ".*".
>
> +1 for deprecating list argument; and replacing with new API.
>
> -Anil.
>
>
>
> On Thu, Nov 16, 2017 at 3:36 PM, Jason Huynh  wrote:
>
> > For GEODE-3813 :
> Region
> > registerInterest API usage of type parameters is broken
> > 
> >
> >
> > The current API to registerInterest allows a special string token
> > “ALL_KEYS” to be passed in as the parameter to registerInterest(T key).
> > This special token causes the registerInterest to behave similar to
> > registerInterestRegex(“.*”).  As the ticket states, if the region has
> been
> > typed to anything other than Object or String, the usage of “ALL_KEYS”
> as a
> > parameter results in a compilation error.
> >
> >
> > Proposals:
> >
> > I would like to deprecate the special string “ALL_KEYS” and document a
> > workaround of using registerInterestRegex(“.*”) or we can add a new API
> > called registerInterestAllKeys()
> >
> >
> > I think we should also deprecate passing a List Object of keys into
> > registerInterest.  It has the same compilation restrictions as “ALL_KEYS”
> > when the region is key constrained/typed.  The reason why List would be
> > used is to allow registering multiple keys at once.  Instead, we can add
> a
> > new var arg API like registerInterest(T… keys).  This problem and
> solution
> > was also documented in the ticket by the ticket creator (Kirk Lund)
> >
> >
> >
> > Thanks,
> >
> > -Jason
> >
>

Re: Removing the old Authenticator/AccessControl interfaces

[Michael Stolz]

Yep. That's the normal path.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Oct 30, 2017 at 7:19 PM, Mark Bretl  wrote:

> If we are talking about the following:
> https://github.com/apache/geode/blob/develop/geode-core/
> src/main/java/org/apache/geode/security/Authenticator.java
> https://github.com/apache/geode/blob/develop/geode-core/
> src/main/java/org/apache/geode/security/AccessControl.java
>
> They are both marked '@deprecated since Geode 1.0' which I interpret to
> mean they were deprecated in Geode 1.0 and need to stay until 2.0, if we
> keep with semver.
>
> --Mark
>
>
>
> On Mon, Oct 30, 2017 at 3:55 PM, Jinmei Liao  wrote:
>
> > Yes, it's deprecated in 1.0.
> >
> > On Mon, Oct 30, 2017 at 3:39 PM, Dan Smith  wrote:
> >
> > > Are you saying these interfaces were already deprecated in 1.0? If so,
> > then
> > > I think we can remove them. If they have been deprecated after 1.0, I
> > think
> > > we have to wait until 2.0 to remove them.
> > >
> > > -Dan
> > >
> > > On Mon, Oct 30, 2017 at 1:18 PM, Jinmei Liao 
> wrote:
> > >
> > > > The old security framework, basically Authentcator/AccessControl
> > > interfaces
> > > > and the usage of them inside GEODE is deprecated since 1.0. I am
> > > wondering
> > > > since 1.3 is out already, can we start removing these interfaces and
> > > > usages? It would enable of deleting bunch of unmanageable tests.
> > > >
> > > >
> > > > --
> > > > Cheers
> > > >
> > > > Jinmei
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> >
> > Jinmei
> >
>

Re: [DISCUSS] Storage-class memory ecosystem program

[Michael Stolz]

I'd like to participate in this as well.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Oct 24, 2017 2:59 PM, "Swapnil Bawaskar"  wrote:

> Thanks Roman!
>
> I have volunteered to be on that group, if anyone else is interested,
> please chime in on that email thread.
>
> On Tue, Oct 24, 2017 at 10:57 PM Roman Shaposhnik 
> wrote:
>
> > FYI
> >
> >
> > -- Forwarded message --
> > From: Gang(Gary) Wang 
> > Date: Tue, Oct 24, 2017 at 10:10 AM
> > Subject: Re: [DISCUSS] Storage-class memory ecosystem program
> > To: gene...@incubator.apache.org
> > Cc: d...@mnemonic.incubator.apache.org
> >
> >
> > Add *Hive/LLAP and RocketMQ*
> >
> >- *Ignite* represented by Denis Magda
> >- *Arrow *represented by Wes McKinney
> >- *Hbase *represented by Anoop John
> >- *Crail* represented by *Patrick Stuedi*
> >- *Hive/LLAP *represented by *Gopal Vijayaraghavan*
> >- *RocketMQ  *represented by *Von Gosling*
> >- *ORC *represented by* Owen O'Malley*
> >- *Mnemonic *represented by* Gary*
> >
> > With above projects, we could cover Storage-class memory oriented
> > *Distributed
> > Database, KV Store, Columnar Structured Dataset, Distributed Data Store,
> > Columnar Storage, **Live Long And Process, Distributed Messaging and
> > Streaming, **Durable Object Model, Durable Computing Model* for new
> > generation high-performance applications, e.g. data querying, processing,
> > and analytics.
> >
> > On Mon, Oct 23, 2017 at 9:32 PM, Von Gosling 
> > wrote:
> >
> > > I'm happy to represent Apache RocketMQ on this  storage collaboration
> > > effort. This is a very key in low latency messaging.
> > >
> > > Best Regards,
> > > Von Gosling
> > >
> > >
> > >
> > > > 在 2017年10月20日，02:55，Gang(Gary) Wang  写道：
> > > >
> > > > Hi all,
> > > >
> > > > We can expect more and more projects will take the huge potential
> > > > advantages of storage-class memory for data processing and analytics
> > > > because silicon companies are able to produce high capacity
> > non-volatile
> > > > memory on a large scale, this hardware technology will fundamentally
> > > change
> > > > the way to construct high performance applications similar to what
> > > happened
> > > > when replacing tape with disk technology since the 1980s. so if
> > > possible, I
> > > > advocate establishing an Apache working group to enhance the
> > > collaboration
> > > > and synergies mentioned by Patrick Stuedi for storage-class memory
> > > > technology-oriented projects.
> > > >
> > > > Best.
> > > > Gary.
> > >
> > >
> >
>

Re: [Discuss] CliStrings

[Michael Stolz]

I know we did at one time go through the exercise to collect all of the
strings and put them in one place and have them translated to at least
Japanese.

I don't know where the Japanese version of the file is though.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Thu, Oct 19, 2017 at 6:05 PM, Jared Stewart  wrote:

> I wanted to kick off a discussion about the usage of CliStrings.  For
> those unfamiliar, it’s a java class that contains about ~3000 lines of
> String constants and has a javadoc explaining that it is an attempt at i18n
> localization.  Does anyone know if this localization is actually
> implemented in practice?
>
> If not, I would like suggest that we try to move away from this pattern
> going forward.  We have ended up with many constants in CliStrings like
> this:
> CliStrings.CREATE_REGION__MSG__ONLY_ONE_OF_REGIONSHORTCUT_
> AND_USEATTRIBUESFROM_CAN_BE_SPECIFIED
> The constant is only used in CreateRegionCommand, so I would be happier to
> see it as a member of CreateRegionCommand (where there would be no need for
> the unwieldy "CREATE_REGION__MSG__” prefix) unless there is localization
> being done which I am unaware of.
>
> Thoughts?
>
> Thanks,
> Jared

Re: [ANNOUNCE] Spring Data Geode 2.0.0.RELEASE (Kay GA) Available...

[Michael Stolz]

Wow! Great stuff John!

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Thu, Oct 5, 2017 at 6:35 PM, John Blum  wrote:

> Dear Geode Community-
>
> After almost 1 year of radio silence on all things related to *Spring
> Data Geode* for Apache Geode, it is my pleasure to inform you that *Spring
> Data Geode* *2.0.0.RELEASE* (Kay GA) is now available! [1]
>
> Many things have happened since my last announcement.
>
> First, *Spring Data Geode* 2.0 joins the *Spring Data Release Train* [2]
> as another top-level *Spring Data* module in the *Spring Data* portfolio.
> [3]  This is significant for few reasons, but most importantly, you can
> expect a predictable and regular series of SDG releases going forward, and
> announcement from me when they occur.
>
> Next, *Spring Data Geode* 2.0 encompasses some key updates...
>
> * Upgrades to *Apache Geode 1.2.1*.
>
> * Uses *Java 8* as the baseline.
>
> * Upgrades to *Spring Framework 5.0 GA*.
>
> * Includes a new and very well-polished *Annotation-based configuration
> model* [4] for getting started with Apache Geode quickly and easily,
> especially when using *Spring Boot*.  You will find this [5], this [6]
> (DATAGEODE-33) and then this [7] (DATAGEODE-34) particularly interesting.
>
> * Improves support when using Apache Geode with other transactional
> resources in a JTA transaction by including *Annotation configuration for
> Geode's JCA Resource Adapter*; DATAGEODE-16. [8]
>
> * Adds support for conveniently enabling *client-side Security* when
> using the @EnableSecurity annotation, DATAGEODE-24 [9].  Some of you may
> remember this blog post [10] where I discussed SDG's support of Apache
> Geode's new *Integrated Security* framework, which focused on server-side
> Security. [11]  Now, the same annotation covers client-side Security as
> well [12].
>
> * Improves support of Apache Geode's *Continuous Query* feature using
> Annotations, DATAGEODE-38 [13], complete with associated documentation
> [14].  Works similarly to the core *Spring Framework's* POJO method
> annotated message listeners.
>
> One other notable is DATAGEODE-18 [15], which is *the making of a test
> framework* for greatly simplifying the development of both *Unit* and 
> *Integration
> Tests* for Apache Geode applications in a *Spring* context.  Every user
> here knows how daunting a task writing effective Unit/Integration tests can
> be for Apache Geode; I have been doing this with GemFire/Geode (with
> *Spring*) for well over 6 years.
>
> The SDG testing framework aims to introduce new Annotations annotating
> your test classes that will help in simplifying mocking GemFire components
> in *Unit Tests* and as well manage servers tied to the *Spring's*
> TestContext framework/container lifecycle inside your testing provider
> (e.g. *JUnit*) in client/server-based integration tests.
>
> For instance, here is 1 example [16] and an earlier preview of using the
> new @EnableGemFireMockObjects annotation in *Unit Tests*.
>
> For a complete list of changes in this release, have a look in the
> *changelog* [17].
>
> So many goodies to share, not enough time, so... expect a series of blog
> posts to follow and startup covering all the new developments in *Spring*
> on the Apache Geode front.
>
> I hope you will enjoy using all the new features in this release, and, as
> always, feedback is very much appreciated and welcomed.
>
> Stay tuned for more!  Until next time...
>
> Cheers!
> --
> -John
>
> [1] https://spring.io/blog/2017/10/02/spring-data-
> release-train-kay-goes-ga
> [2] https://github.com/spring-projects/spring-data-commons/
> wiki/Release-Train-Kay#participating-modules
> [3] http://projects.spring.io/spring-data/
> [4] https://docs.spring.io/spring-data/geode/docs/current/reference/html/#
> bootstrap-annotation-config
> [5] https://docs.spring.io/spring-data/geode/docs/current/reference/html/#
> bootstrap-annotation-config-regions
> [6] https://docs.spring.io/spring-data/geode/docs/current/reference/html/#
> bootstrap-annotation-config-caching
> [7] https://docs.spring.io/spring-data/geode/docs/current/reference/html/#
> bootstrap-annotation-config-cluster
> [8] https://jira.spring.io/browse/DATAGEODE-16
> [9] https://jira.spring.io/browse/DATAGEODE-24
> [10] https://spring.io/blog/2016/11/10/spring-data-geode-
> 1-0-0-incubating-release-released
> [11] https://docs.spring.io/spring-data/geode/docs/
> current/reference/html/#bootstrap-annotation-config-security-server
> [12] https://docs.spring.io/spring-data/geode/docs/
> current/reference/html/#bootstrap-annotation-config-security-client
> [13] https://jira.spring.io/browse/DATAGEODE-38
> [14] https://docs.spring.io/spring-data/geode/docs/
> current/reference/html/#bootstrap-annotation-config-continuous-queries
> [15] https://jira.spring.io/browse/DATAGEODE-18
> [16] https://github.com/spring-projects/spring-data-
> geode/blob/master/src/test/java/org/springframework/data/gemfire/client/
> ClientCacheI

Re: New client/server protocol - seeking feedback

[Michael Stolz]

We should check that it is actually safe to add fields.
If it isn't we're likely to have a lot of versioning to do.

--
Mike Stolz
Principal Engineer, GemFire Product Lead
Mobile: +1-631-835-4771

On Mon, Sep 25, 2017 at 5:25 PM, Galen O'Sullivan 
wrote:

> Replies inline.
>
> On Mon, Sep 25, 2017 at 12:13 PM, Udo Kohlmeyer 
> wrote:
>
> > Replies inline
> > On Mon, Sep 25, 2017 at 11:21 AM, Dan Smith  wrote:
> >
> > > This actually brings up another point I was going to ask about. I don't
> > see
> > > any version information in the protocol. How will we handle adding new
> > > features to this protocol? Do the clients and servers negotiate which
> > > version of the protocol to use somehow?
> > >
> > > I think there should be a plan in place for making changes to the
> > protocol
> > > and supporting old clients. Given that, we shouldn't add fields that
> > aren't
> > > actually used into the existing version of the protocol. When we
> > introduce
> > > new features into the protocol, that's the point at which we should add
> > the
> > > fields to support that feature.
> > >
> >
> > [UK] - Protobuf allows for the amending of messages. As soon as the
> > protocol changes significantly the "magic" number will have to be
> > incremented, indicating a new (non-backward compatible) version of the
> > protocol. This of course has bigger problems, where Java does not allow
> for
> > multiple versions of the same class to be loaded, so a server could run
> > only 1 version of Protobuf messages at a time.
> >
>
> We have to be careful about how we extend protobuf messages, though. I'm
> not sure exactly what's safe to do, but it should at least be safe to add
> fields (assuming they don't change existing behavior -- we'll have to think
> about this) and ignore old fields (which is how you would remove a
> now-unused field). It's fairly simple to add new operations without any
> interesting breakages - they'll fail with older servers and not be sent
> with older clients. I think adding new operations is a pretty good way to
> add features that don't require a real rework of the protocol. For those
> that do, we can version the initial byte.
>

Re: New Geode PMC Member: Galen O'Sullivan

[Michael Stolz]

Where can I see the list of PMC members?

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Sep 26, 2017 7:49 PM, "Mark Bretl"  wrote:

> The Apache Geode Project Management Committee has invited Galen
> O'Sullivan to join the Geode PMC and we are pleased to announce he has
> accepted.
>
> Please join me in welcoming Galen!
>
> Best regards,
>
> Mark
> On behalf of the Apache Geode PMC
>

Re: [DISCUSS] authorizing function execution

[Michael Stolz]

Yeah that's the right level of authorization. The Function Author should
get to decide.
Listeners, Loaders and Writers should only require DATA:READ and DATA:WRITE
as appropriate.
It is up to the authors what they do inside of those.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Thu, Sep 14, 2017 at 2:29 PM, Swapnil Bawaskar 
wrote:

> Indeed, the function author has a higher level of privilege than someone
> who is invoking a function, that is why the proposal here is to let the
> function author choose what level of permissions are required to invoke a
> function.
>
> On Thu, Sep 14, 2017 at 11:20 AM Anilkumar Gingade 
> wrote:
>
> > >> from reaching into internal classes
> > If thats the case; one could do anything, even with read
> permission...Isn't
> > it...
> >
> >
> >
> > On Thu, Sep 14, 2017 at 10:43 AM, Jared Stewart 
> > wrote:
> >
> > > There is nothing to prevent code in a function that's executing on a
> > > server from reaching into internal classes and bypassing the public
> > region
> > > APIs. I think a function's author should ultimately determine the
> > > permissions required to execute it.
> > >
> > > > On Sep 14, 2017, at 10:34 AM, Anilkumar Gingade  >
> > > wrote:
> > > >
> > > > When a function is accessing/modifying region; the function will be
> > doing
> > > > so by region apis, don't we have credential check with region apis;
> if
> > > not
> > > > can we add those checks here...instead of having it in the
> function...
> > > >
> > > > -Anil.
> > > >
> > > >
> > > >
> > > > On Wed, Sep 13, 2017 at 11:22 AM, Jared Stewart  >
> > > wrote:
> > > >
> > > >> After some more investigation into the implementation details, here
> is
> > > our
> > > >> updated proposal to add to the Function interface:
> > > >>
> > > >> default Collection getRequiredPermissions(
> > > Optional
> > > >> onRegion) {
> > > >>  return Collections.singletonList(ResourcePermissions.DATA_WRITE);
> > > >> }
> > > >>
> > > >> This method can be overridden by Function authors who want to
> require
> > > >> permissions other than DATA:WRITE.. The onRegion parameter will be
> > > present
> > > >> only when a Function is executed via FunctionService.onRegion, and
> is
> > > >> intended to allow Function authors to require different permissions
> > > >> depending on the Region which Function will be executed on.  We pass
> > the
> > > >> region name into this method rather than the full FunctionContext
> > > because
> > > >> the latter would be much more expansive to implement.
> > > >>
> > > >> Any feedback is appreciated.
> > > >>
> > > >> Thanks,
> > > >> Jared
> > > >>
> > > >>> On Aug 17, 2017, at 1:42 AM, Swapnil Bawaskar <
> sbawas...@pivotal.io>
> > > >> wrote:
> > > >>>
> > > >>> Discuss fix for GEODE-2817
> > > >>> 
> > > >>>
> > > >>> Currently to execute a function, you will need "data:write"
> > permission,
> > > >> but
> > > >>> it really depends on what the function is doing. For example, if a
> > > >> function
> > > >>> is just reading data, the function author might want users with
> > > DATA:READ
> > > >>> permissions to execute the function. The two options mentioned in
> the
> > > >>> ticket are:
> > > >>>
> > > >>> 1) externalize SecurityService so that function author can use it
> in
> > > the
> > > >>> function.execute code to check authorization.
> > > >>> 2) add a method to function interface to tell the framework what
> > > >> permission
> > > >>> this function needs to execute, so that the framework will check
> the
> > > >>> permission before executing the function.
> > > >>>
> > > >>> I vote for #2 because, I think, a function author will be able to
> > > easily
> > > >>> discover a method on the Function interface, rather than trying to
> > look
> > > >> for
> > > >>> SecurityService.
> > > >>>
> > > >>> I propose that we add the following new method to Function:
> > > >>>
> > > >>> default public List requiredPermissions() {
> > > >>>  // default DATA:WRITE
> > > >>> }
> > > >>>
> > > >>> In order to preserve existing behavior, the default required
> > permission
> > > >>> would be DATA:WRITE.
> > > >>
> > > >>
> > >
> > >
> >
>

Re: [DISCUSS] Addition of isValid API to Index interface

[Michael Stolz]

Great, that's exactly the behavior I would expect.

Thanks.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Mon, Sep 11, 2017 at 4:34 PM, Jason Huynh  wrote:

> Hi Mike, I think the concern was less about the security portion but rather
> if any exception occurs during index update, right now, the region gets
> updated and the rest of the system (index/wan/callbacks) may or may not be
> updated.  I think Naba just tried to provide an example where this might
> occur, but that specific scenario is invalid.
>
> I believe Nabarun has opened a ticket for rolling back the put operation
> when an index exception occurs. GEODE-3589.  It can probably be modified to
> state any exception instead of index exceptions.
>
> To summarize my understanding:
> -Someone will need to implement the rollback for GEODE-3589.  This means
> that if any exception occurs during a put, geode it will propagate back to
> the user and it is expected the rollback mechanism will clean up any
> partial put.
>
> GEODE-3520 should be modified to:
> -Add the isValid() api to index interface
> -Mark an index as invalid during async index updates but not for
> synchronous index updates.  The synchronous index updates will rely on a
> rollback mechanism
>
>
>
>
> On Mon, Sep 11, 2017 at 1:23 PM Michael Stolz  wrote:
>
> > I think there was an intention of having CREATION of an index require a
> > higher privilege than DATA:WRITE, but it shouldn't affect applying the
> > index on either of put or get operations.
> >
> > If we are requiring something like CLUSTER:MANAGE for put on an indexed
> > region, that is an incorrect requirement. Only DATA:WRITE should be
> > required to put an entry and have it be indexed if an index is present.
> >
> > --
> > Mike Stolz
> > Principal Engineer, GemFire Product Manager
> > Mobile: +1-631-835-4771 <(631)%20835-4771>
> >
> > On Fri, Sep 8, 2017 at 6:04 PM, Anilkumar Gingade 
> > wrote:
> >
> > > Indexes are critical for querying; most of the databases doesn't allow
> > > insert/update if there is any failure with index maintenance...
> > >
> > > As Geode OQL supports two ways (sync and async) to maintain the
> indexes,
> > we
> > > need be careful about the error handling in both cases...
> > >
> > > My take is:
> > > 1. For synchronous index maintenance:
> > > If there is any failure in updating any index (security/auth or logical
> > > error) on the region; throw an exception and rollback the cache
> update/op
> > > (index management id done under region.entry lock - we should be able
> to
> > > revert the op). If index or cache is left in bad state, then its a bug
> > that
> > > needs to be addressed.
> > >
> > > Most of the time, If there is any logical error in index, it will be
> > > detected as soon as index is created (on existing data) or when first
> > > update is done to the cache.
> > >
> > > 2. For Asynchronous index maintenance:
> > > As this is async (assuming) user has good understanding of the risk
> > > involved with async, any error with index maintenance, the index should
> > be
> > > invalidated...
> > >
> > >  About the security/auth, the user permission with region read/write
> > needs
> > > to be applied for index updates, there should not be different
> permission
> > > on index.
> > >
> > > -Anil.
> > >
> > >
> > >
> > > On Fri, Sep 8, 2017 at 2:01 PM, Nabarun Nag  wrote:
> > >
> > > > Hi Mike,
> > > >
> > > > Please do find our answers below:
> > > > *Question:* What if there were multiple indices that were in flight
> and
> > > > only the third
> > > > one errors out, will they all be marked invalid?
> > > >
> > > > *Answer:* Only the third will be marked invalid and only the third
> one
> > > will
> > > > not be used for query execution.
> > > >
> > > > *Question/Statement:* If anything goes wrong with the put it should
> > > > probably still throw back to
> > > > the caller. Silent invalidation of the index is probably not
> desirable.
> > > >
> > > > *Answer: *
> > > > In our current design this the flow of execution of a put operation:
> > > > entry put into region -> update index -> other wan related
> executions /
> > > > callbacks etc.
> > >

Re: [DISCUSS] Addition of isValid API to Index interface

[Michael Stolz]

I think there was an intention of having CREATION of an index require a
higher privilege than DATA:WRITE, but it shouldn't affect applying the
index on either of put or get operations.

If we are requiring something like CLUSTER:MANAGE for put on an indexed
region, that is an incorrect requirement. Only DATA:WRITE should be
required to put an entry and have it be indexed if an index is present.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Fri, Sep 8, 2017 at 6:04 PM, Anilkumar Gingade 
wrote:

> Indexes are critical for querying; most of the databases doesn't allow
> insert/update if there is any failure with index maintenance...
>
> As Geode OQL supports two ways (sync and async) to maintain the indexes, we
> need be careful about the error handling in both cases...
>
> My take is:
> 1. For synchronous index maintenance:
> If there is any failure in updating any index (security/auth or logical
> error) on the region; throw an exception and rollback the cache update/op
> (index management id done under region.entry lock - we should be able to
> revert the op). If index or cache is left in bad state, then its a bug that
> needs to be addressed.
>
> Most of the time, If there is any logical error in index, it will be
> detected as soon as index is created (on existing data) or when first
> update is done to the cache.
>
> 2. For Asynchronous index maintenance:
> As this is async (assuming) user has good understanding of the risk
> involved with async, any error with index maintenance, the index should be
> invalidated...
>
>  About the security/auth, the user permission with region read/write needs
> to be applied for index updates, there should not be different permission
> on index.
>
> -Anil.
>
>
>
> On Fri, Sep 8, 2017 at 2:01 PM, Nabarun Nag  wrote:
>
> > Hi Mike,
> >
> > Please do find our answers below:
> > *Question:* What if there were multiple indices that were in flight and
> > only the third
> > one errors out, will they all be marked invalid?
> >
> > *Answer:* Only the third will be marked invalid and only the third one
> will
> > not be used for query execution.
> >
> > *Question/Statement:* If anything goes wrong with the put it should
> > probably still throw back to
> > the caller. Silent invalidation of the index is probably not desirable.
> >
> > *Answer: *
> > In our current design this the flow of execution of a put operation:
> > entry put into region -> update index -> other wan related executions /
> > callbacks etc.
> >
> > If an exception happens while updating the index, the cache gets into a
> bad
> > state, and we may end up getting different results depending on the index
> > we are using. As the failure happens half way in a put operation, the
> > regions / cache are now in a bad state.
> > --
> > We are thinking that if index is created  over a method invocation in an
> > empty region and then we do puts, but method invocation is not allowed as
> > per security policies. The puts will now be successful but the index will
> > be rendered invalid. Previously the puts will fail with exception and put
> > the entire cache in a bad state.
> >
> >
> >
> > Regards
> > Nabarun
> >
> >
> >
> >
> >
> > On Fri, Sep 8, 2017 at 10:43 AM Michael Stolz  wrote:
> >
> > > Just to help me understand, the index is corrupted in a way beyond just
> > the
> > > field that errors out?
> > > What if there were multiple indices that were in flight and only the
> > third
> > > one errors out, will they all be marked invalid?
> > > If anything goes wrong with the put it should probably still throw back
> > to
> > > the caller. Silent invalidation of the index is probably not desirable.
> > >
> > > --
> > > Mike Stolz
> > > Principal Engineer, GemFire Product Manager
> > > Mobile: +1-631-835-4771 <(631)%20835-4771>
> > >
> > > On Fri, Sep 8, 2017 at 12:34 PM, Dan Smith  wrote:
> > >
> > > > +1
> > > >
> > > > -Dan
> > > >
> > > > On Thu, Sep 7, 2017 at 9:14 PM, Nabarun Nag  wrote:
> > > >
> > > > > *Proposal:*
> > > > > * Index interface will include an API - isValid() which will return
> > > true
> > > > if
> > > > > the index is still valid / uncorrupted, else will return false if
> it
> > > > > corrupted / invalid.
> > > > > * gfsh command "list index&q

Re: [DISCUSS] Addition of isValid API to Index interface

[Michael Stolz]

Just to help me understand, the index is corrupted in a way beyond just the
field that errors out?
What if there were multiple indices that were in flight and only the third
one errors out, will they all be marked invalid?
If anything goes wrong with the put it should probably still throw back to
the caller. Silent invalidation of the index is probably not desirable.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Fri, Sep 8, 2017 at 12:34 PM, Dan Smith  wrote:

> +1
>
> -Dan
>
> On Thu, Sep 7, 2017 at 9:14 PM, Nabarun Nag  wrote:
>
> > *Proposal:*
> > * Index interface will include an API - isValid() which will return true
> if
> > the index is still valid / uncorrupted, else will return false if it
> > corrupted / invalid.
> > * gfsh command "list index" will have one more column "Is Valid" which
> will
> > state the status as "true" or "false".
> > * Invalid indexes will not be used during query executions.
> > * In case the index is found to be invalid, the user will be able to
> > remove/destroy the index.
> > * When a put operation corrupts an index, it will be logged.
> >
> > *Reasoning:*
> > * Currently if a put operation raises an exception while updating the
> > index, the put operation fails with an exception to the putter.
> > * For example, if an index is created on an object method, and that
> method
> > causes an exception while updating the index as a part of a put
> operation,
> > then the put operation fails for that particular entry and the index is
> > left in a bad state.
> > * This may occur also due to lack of permission to update index but have
> > permission to do puts.
> > * We are proposing that in the above mentioned scenarios, the put
> succeeds
> > in putting the entry in the region but the index which it was trying to
> > update will be marked invalid and will not be used for query executions.
> > * This can be justified because the corrupted index will not correctly
> > represent the region entries.
> >
> > *Status Quo:*
> > * Index creation will still fail if we are trying to create an index
> over a
> > region containing an entry/entries  which will cause an exception while
> > loading the entry into the index.
> >
> > Regards
> > Nabarun Nag
> >
>

Re: [Discuss] Use of -1 as Infinite/All for retry related functions...

[Michael Stolz]

Just don't break existing users.
Timeouts are tricky, and we have lots of users who have tuned them very
carefully over the years.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Wed, Sep 6, 2017 at 1:18 PM, Mark Hanson  wrote:

> So the basic challenge here is for specific API, do we want to focus on
> providing a
> wait forever approach or just use the standard MAX_UNSIGNED and duration.
> I
> think that variable delay is something someone would implement in their own
> API
> and would not be done in the public API where consistency is valuable. I
> could be
> wrong in that belief.
>
> Further, it is sounding like a move away from overloading is the desired
> direction.
> Do we have any points against it??
>
> On Fri, Sep 1, 2017 at 2:45 PM, Michael Stolz  wrote:
>
> > We would certainly rather have the time-out set correctly, but one of the
> > things I've noticed is, sometimes there is just one query or one function
> > that takes a really long time, and because we keep retrying it with the
> > same timeout, it keeps timing out each retry. It would probably be much
> > smarter to use some sort of increasing timeout on the retries until we
> give
> > up.
> >
> > --
> > Mike Stolz
> > Principal Engineer, GemFire Product Manager
> > Mobile: +1-631-835-4771
> >
> > On Fri, Sep 1, 2017 at 2:07 PM, Mark Hanson  wrote:
> >
> > > I actually don’t really have a strong opinion one way or the other. I
> get
> > > both approaches. If we want to tailor the code to use a timeout instead
> > of
> > > retry attempts I guess that is fine. It seems kind of like we are
> > > perpetuating the same API problem, that the LCD approach alleviates,
> but
> > ok.
> > >
> > > It is more complicated to code because now you need to push everything
> > > through poll or select. Such as the connect. Not that that is a bad
> > thing,
> > > because it is not. It is just more complicated.
> > >
> > > Thanks,
> > > Mark
> > >
> > > http://developerweb.net/viewtopic.php?id=3196 <
> http://developerweb.net/
> > > viewtopic.php?id=3196>
> > > > On Aug 31, 2017, at 3:47 PM, Jacob Barrett 
> > wrote:
> > > >
> > > > None of the time spent performing the request is deterministic that’s
> > > why there are timeouts. I don’t follow your rational for claiming it
> > > complicated to code.
> > > >
> > > >> On Aug 31, 2017, at 3:27 PM, Mark Hanson 
> wrote:
> > > >>
> > > >> The only problem with that is the time to connect to another server
> is
> > > non-deterministic. So,  the code one would have to write to enable this
> > > would involve a select and a bit of not fun code, but in general could
> be
> > > not very useful as an API.
> > > >>
> > > >> I would say the lowest common denominator approach or the server
> based
> > > approach is better.
> > > >>
> > > >> Just two cents.
> > > >>
> > > >> Thanks,
> > > >> Mark
> > > >>> On Aug 31, 2017, at 1:41 PM, Jacob Barrett 
> > > wrote:
> > > >>>
> > > >>> I believe what Bruce was saying is that the behavior should be
> > covered
> > > by
> > > >>> timeouts not iteration attempts. If the client is able to
> > successfully
> > > send
> > > >>> the command to a server but a failure occurs waiting for a reply we
> > > would
> > > >>> not retry. If the client is unable to send the request to a sever
> > > because
> > > >>> the connection closes then we would try the next server, and the
> > next,
> > > up
> > > >>> to the timeout value.
> > > >>>
> > > >>>> On Thu, Aug 31, 2017 at 1:31 PM Mark Hanson 
> > > wrote:
> > > >>>>
> > > >>>> I can also see why the user doing the retries themselves has
> value.
> > > As a
> > > >>>> lowest common denominator approach, pulling the API is sound.
> > > >>>>
> > > >>>>> On Thu, Aug 31, 2017 at 1:26 PM, Mark Hanson  >
> > > wrote:
> > > >>>>>
> > > >>>>> I think the setRetryAttempts really harks back to the case that
> > > Bruce was
> > > >>>>> alluding to in which the server goes down. Whi

Re: [VOTE] Apache Geode release - v1.2.1 RC1

[Michael Stolz]

GEODE-3249 is a breaking change to the client/server protocol, but it has a
property that can override whether or not to turn the breaking change on.

The plan was to release Geode 1.2.1 with the breaking change turned on, but
I spent a lot of time thinking about this over the weekend, and I came to
the conclusion that introducing a *breaking change in a patch release* is a
bad idea. Protocol breaking changes should really only occur in major
releases.

That said, because this is an important change for some users, the right
thing to do is probably to set the default for the controlling property to
enable backward compatibility, and then the users who want the change can
OPT-IN, rather than requiring users who are taking the patch release and
don't want the breaking change having to OPT-OUT.

We can switch that default behavior on the next major release.


--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

Re: [Discuss] Use of -1 as Infinite/All for retry related functions...

[Michael Stolz]

We would certainly rather have the time-out set correctly, but one of the
things I've noticed is, sometimes there is just one query or one function
that takes a really long time, and because we keep retrying it with the
same timeout, it keeps timing out each retry. It would probably be much
smarter to use some sort of increasing timeout on the retries until we give
up.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Fri, Sep 1, 2017 at 2:07 PM, Mark Hanson  wrote:

> I actually don’t really have a strong opinion one way or the other. I get
> both approaches. If we want to tailor the code to use a timeout instead of
> retry attempts I guess that is fine. It seems kind of like we are
> perpetuating the same API problem, that the LCD approach alleviates, but ok.
>
> It is more complicated to code because now you need to push everything
> through poll or select. Such as the connect. Not that that is a bad thing,
> because it is not. It is just more complicated.
>
> Thanks,
> Mark
>
> http://developerweb.net/viewtopic.php?id=3196  viewtopic.php?id=3196>
> > On Aug 31, 2017, at 3:47 PM, Jacob Barrett  wrote:
> >
> > None of the time spent performing the request is deterministic that’s
> why there are timeouts. I don’t follow your rational for claiming it
> complicated to code.
> >
> >> On Aug 31, 2017, at 3:27 PM, Mark Hanson  wrote:
> >>
> >> The only problem with that is the time to connect to another server is
> non-deterministic. So,  the code one would have to write to enable this
> would involve a select and a bit of not fun code, but in general could be
> not very useful as an API.
> >>
> >> I would say the lowest common denominator approach or the server based
> approach is better.
> >>
> >> Just two cents.
> >>
> >> Thanks,
> >> Mark
> >>> On Aug 31, 2017, at 1:41 PM, Jacob Barrett 
> wrote:
> >>>
> >>> I believe what Bruce was saying is that the behavior should be covered
> by
> >>> timeouts not iteration attempts. If the client is able to successfully
> send
> >>> the command to a server but a failure occurs waiting for a reply we
> would
> >>> not retry. If the client is unable to send the request to a sever
> because
> >>> the connection closes then we would try the next server, and the next,
> up
> >>> to the timeout value.
> >>>
>  On Thu, Aug 31, 2017 at 1:31 PM Mark Hanson 
> wrote:
> 
>  I can also see why the user doing the retries themselves has value.
> As a
>  lowest common denominator approach, pulling the API is sound.
> 
> > On Thu, Aug 31, 2017 at 1:26 PM, Mark Hanson 
> wrote:
> >
> > I think the setRetryAttempts really harks back to the case that
> Bruce was
> > alluding to in which the server goes down. Which is the one valid
> case
>  for
> > this kind of API in theory. Are we say that in that case we don't
> retry?
> > Seems like we are making the API a little less nice for people.
> > As a developer using an API, I want to do as little as possible and
> get
> > the most robust solution possible. This seems to go the wrong
> direction
>  of
> > that kind of intent in a way. I want the client to automatically try
>  every
> > server. I don't ever want to configure the value. I could limit with
> this
> > API and force it to never retry or I could cause it to retry more
> times
> > than I care for it to.  If we are going to get rid of this API in
> > particular, I would favor having it automatically try some number of
> > servers or all, but not retrying at all would not be my choice.
> >
> > On Thu, Aug 31, 2017 at 1:08 PM, Jacob Barrett 
> > wrote:
> >
> >>> On Thu, Aug 31, 2017 at 1:00 PM Mark Hanson 
> wrote:
> >>>
> >>> I would have to go looking, but the key concept is that this is a
>  bigger
> >>> problem.
> >>>
> >>> interval such as the time between retries
> >>> wait as in how long to wait for a response...
> >>>
> >>
> >> All time intervals should be expressed in terms of
> std::chrono::duration
> >> values. A value of std::chrono::duration::zero means don't wait. I
> would
> >> suggest that a negative time not be allowed and that some very
> large,
> >> MAXINT, value could take the place of "forever". There is a ticket
>  already
> >> open and in progress to replace all time based values with
> std::chrono.
> >>
> >>
> >>> retry as how many times to retry after a failure
> >>> attempts as in how many times to do a thing before giving up
> >>> Set of objects as in the setRetryAttempts code which , will try a
> >> number of
> >>> servers before giving up. where n is the number, -1 equals all,
> and 0
> >> means
> >>> (1 server, no retries).
> >>>
> >>
> >> If there are other examples of "iteration" then we should consider
> them
> >> based on what they iterate. I think the consensus on
> 

Re: [VOTE] Apache Geode release - v1.2.1 RC1

[Michael Stolz]

I think in number 3 you mean remove flag and restart servers.
That should work.

Ok, +1

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Tue, Aug 29, 2017 at 4:38 PM, Anthony Baker  wrote:

> Hi Mike!
>
> Thanks for your feedback.  IMHO I don't consider GEODE-3249 to break
> client/server compatibility since there's a mechanism to disable the
> protocol change to allow clients to be upgraded.  The process would
> look like this:
>
> 1) Upgrade servers and set the
> 'geode.allow-internal-messages-without-credentials=true' flag.
> 2) Upgrade clients.
> 3) Remove flag and upgrade clients.
>
> Do you agree?
>
> Anthony
>
> On Tue, Aug 29, 2017 at 12:48 PM, Michael Stolz  wrote:
> > -1 for breaking change in client/server protocol.
> >
> > --
> > Mike Stolz
> > Principal Engineer, GemFire Product Manager
> > Mobile: +1-631-835-4771
> >
>

Re: [VOTE] Apache Geode release - v1.2.1 RC1

[Michael Stolz]

-1 for breaking change in client/server protocol.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Tue, Aug 29, 2017 at 3:07 PM, Anthony Baker  wrote:

> I encourage the PMC members to review and vote so we can close out this
> release.  Of course, all feedback is welcome.
>
> Thanks,
> Anthony
>
> > On Aug 25, 2017, at 3:20 PM, Anthony Baker  wrote:
> >
> > This is the first release candidate for Apache Geode, version 1.2.1.
> > Thanks to all the community members for their contributions to this
> > release!
> >
> > *** Please download, test and vote by Wednesday, August 25, 0800 hrs
> > US Pacific. ***
> >
> > It fixes the following issues:
> >  https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12318420&version=12341124
> >
> > Note that we are voting upon the source tags:  rel/v1.2.1.RC1
> >  https://git-wip-us.apache.org/repos/asf?p=geode.git;a=commit;h=
> 1bb1cae34814bdca298e476f8b88d19e4bc0dd51
> >  https://git-wip-us.apache.org/repos/asf?p=geode-examples.
> git;a=commit;h=5d034de588a43cdefb8fbb3a6259579785768340
> >
> > Commit ID:
> >  1bb1cae34814bdca298e476f8b88d19e4bc0dd51 (geode)
> >  5d034de588a43cdefb8fbb3a6259579785768340 (geode-examples)
> >
> > Source and binary files:
> >  https://dist.apache.org/repos/dist/dev/geode/1.2.1.RC1
> >
> > Maven staging repo:
> >  https://repository.apache.org/content/repositories/orgapachegeode-1021
> >
> > Geode's KEYS file containing PGP keys we use to sign the release:
> >  https://git-wip-us.apache.org/repos/asf?p=geode.git;a=blob_
> plain;f=KEYS;hb=HEAD
> >
> > pub  4096R/C72CFB64 2015-10-01
> >  Fingerprint=948E 8234 14BE 693A 7F74  ABBE 19DB CAEE C72C FB64
> >
> > Anthony
>
>

Re: [GitHub] geode issue #719: GEODE-3447 Implement client authorization for the new prot...

[Michael Stolz]

A hash is not guaranteed unique so is not suitable as a security token.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Fri, Aug 25, 2017 at 4:49 PM, galen-pivotal  wrote:

> Github user galen-pivotal commented on the issue:
>
> https://github.com/apache/geode/pull/719
>
> @metatype We need the `StreamAuthenticator` to receive and send
> (Protobuf-encoded) messages containing the credentials that get passed to
> the `SecurityManager`. I would think that ideally it's nothing more than
> this.
>
> I wonder if it would be better to send a hash that gets put into the
> Properties that SecurityManager uses, rather than having a message that
> explicitly contains username and password.
>
>
> ---
> If your project is set up for it, you can reply to this email and have your
> reply appear on GitHub as well. If your project does not have this feature
> enabled and wishes so, or if the feature is enabled but not working, please
> contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
> with INFRA.
> ---
>

Re: [Discuss] Enable region set operations to start JTA transactions

[Michael Stolz]

+1 for revising set ops to bootstrap by default. Document the behavior
change in release notes along with property to revert to old behavior.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Fri, Aug 25, 2017 at 6:41 PM, Anilkumar Gingade 
wrote:

> +1 Having a consistent behavior for set operation within the transaction
> (wherever its invoked). And having a gemfire property to allow old way.
>
> On Fri, Aug 25, 2017 at 2:49 PM, Eric Shu  wrote:
>
> > Hi Team,
> >
> > Currently in GEODE JTA implementation, if GEODE transaction is not yet
> > bootstrapped (there is no TXState created yet for the transaction),
> region
> > set operations will not bootstrap the JTA transaction. In other word, if
> > the first operation of the JTA transaction is region set operation, the
> set
> > is not in a transactional view (TXState). However, if the JTA
> > UserTransaction has been bootstrapped (first operation is a get or put
> > operation, etc), the following region set operation is transactional (in
> > TXState). To some users, this seems to be a bug. Please see the GEODE
> > ticket (https://issues.apache.org/jira/browse/GEODE-3521).
> >
> > Although we intentionally implemented this way to alleviate the memory
> > requirements for users (each transaction will have a new copy of region
> > data put into TXState but not if the set op is the first op), we think
> the
> > new user has a valid point as well. We propose to allow region set op to
> > bootstrap the transaction. To allow backward compatibility, we will
> > introduce a new GemFire property to allow the old behavior.
> >
> > The question is what should be a default behavior. Do we allow region set
> > op to bootstrap transaction as default? If so, existing customers need to
> > set the new property to get the old behavior. Or we maintain the old
> > behavior, and require the new users to set the property.
> >
> > Please comment if you have any suggestions.
> >
> > Thanks,
> > Eric
> >
>

Re: Adding parallel import/export of snapshots to gfsh

[Michael Stolz]

One other idea that hasn't been mentioned is making parallel the only way
for Partitioned Regions, and having --file serve the purpose of defining
both a path and a filename pattern where the bucket ID or whatever we're
using gets automatically inserted before the .gfd extension.

No need for a new option (--parallel).
No need for a new option (--path).

In fact, no need for a change to gfsh command at all.


--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Tue, Aug 22, 2017 at 2:15 PM, Nick Reich  wrote:

> Parallel export will write the data to files on the bucket primary for each
> bucket, distributing the work (and therefore files) to all the members.
> That would be a big enough deviation from the current behavior (single file
> on single machine), that I think it makes it worth having the additional
> options (but I agree: less options is generally better).
>
> On Tue, Aug 22, 2017 at 1:59 PM, Jacob Barrett 
> wrote:
>
> > On Tue, Aug 22, 2017 at 1:49 PM Nick Reich  wrote:
> >
> > > The idea of deprecating —file in favor of path is interesting. I wonder
> > if
> > > instead of making them mutually exclusive to start, having —path be
> able
> > to
> > > support both modes from the start would be better? That way —file could
> > > still be used for the existing mode, but —path could be used instead
> (and
> > > override —file is both given?): that would provide a clear path forward
> > for
> > > how the command should be used, while fully supporting existing
> > workflows.
> > >
> >
> > This is what I meant by deprecating. Maybe even providing a message that
> if
> > --file is set that it is deprecated for --path.
> >
> >
> > > We need to continue to support both modes, as only Partitioned Regions
> > can
> > > make use of parallel export (it is parallelized on a bucket level).
> > >
> >
> > Ok, so why not just make parallel the only mode for partitioned. Then you
> > remove the need for --parallel and --path would work for any region,
> > non-partitioned would create a single file at that path and partitioned
> > would create several? I am all for less options. ;)
> >
> > -Jake
> >
>

Tombstones

[Michael Stolz]

Do tombstones show up in operations that indicate presence of keys like
Region.keyset(), containsKey(key), etc?
They probably shouldn't.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

Re: Question regarding node failure scenario

[Michael Stolz]

Without subscription-redundancy you are running the risk that some of the
data isn't being pushed to you in the event of a server failure.


--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Sun, Jul 23, 2017 at 10:06 PM, Akihiro Kitada  wrote:

> Hello Roi,
>
> I want to confirm actual your configuration.
>
> >- I have a replicated Node, say N1 and its replicated N2 (N2 gets
> activated when N1 is down) and they are configured to send updates via
> continuous query to my process which then reports on these updates.
>
> Do node N1 and N2 replicate each data based on the same Replicated region
> configuration in the same distributed system? If not, how do N1 and N2
> replicated data? Could you attach actual cache configuration (such as
> cache.xml) for N1 and N2?
>
> Who update the data, some specific Geode client application or Geode peer
> (cache server internally)?
>
> Thanks.
>
>
>
>
> --
> Akihiro Kitada  |  Staff Customer Engineer |  +81 80 3716 3736
> Support.Pivotal.io   |  Mon-Fri  9:00am to
> 5:30pm JST  |  1-877-477-2269
> [image: support]  [image: twitter]
>  [image: linkedin]
>  [image: facebook]
>  [image: google plus]
>  [image: youtube]
> 
>
>
> 2017-07-23 22:04 GMT+09:00 Roi Apelker :
>
> > Hi,  (Bear with me I am a bit new here :))
> >
> > I have the following scenario, I wonder if anyone can comment on it - is
> > it a known issue, maybe it was solved already in later version, etc. (I
> am
> > using version 1.0.0)
> > Or maybe you can point me to somewhere in the code.
> > I have posted this question once before (thanks Dan S. for relating to
> > it), however I was away for 3 weeks therefore posting again.
> >
> > - I have a replicated Node, say N1 and its replicated N2 (N2 gets
> > activated when N1 is down) and they are configured to send updates via
> > continuous query to my process which then reports on these updates.
> >
> > - N1 is working all the time and serves as a server, and data is written
> > to it continuously from external clients. If 1000 events are written to
> N1,
> > the final report I am referring to will indicate 1000.
> >
> > - When N1 gets killed for any reason, the client connects to N2 which
> > continues to send the continuous query results.
> >
> > - But sometimes, the report is inaccurate, e.g. after running 1000
> events,
> > my report says 950, as if some events of the continuous query do not
> reach
> > the client (the actual data does arrive its destination, just the report
> is
> > qrong)
> >
> > The fact is, that the report is not accurate, and I only assume that
> > something is wrong in the update mechanism, somewhere in the area of
> > continuous query, or somewhere in the replication between the nodes.
> >
> > Right now, the parameter of subscription-redundancy is not configured.
> But
> > subscription-enabled="true" .
> >
> > What is the significance of not configuring the subscription-redundancy ?
> > does it mean that client disconnection may cause continuous query events
> to
> > be discarded?
> > And if so, is it "may be discarded" or "will be discarded", meaning, will
> > the result be always the same or not?
> >
> >
> > Thanks,
> >
> > Roi
> > This message and the information contained herein is proprietary and
> > confidential and subject to the Amdocs policy statement,
> >
> > you may review at https://www.amdocs.com/about/email-disclaimer <
> > https://www.amdocs.com/about/email-disclaimer>
> >
> >
>

Re: New Committer: Deepak Dixit

[Michael Stolz]

Welcome

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jul 14, 2017 7:46 PM, "Mark Bretl"  wrote:

> The Apache Geode Project Management Committee has invited Deepak Dixit to
> be committer on the project. We are pleased to announce he has accepted.
>
> Please join me in welcoming Deepak!
>
> Thanks,
>
> Mark
> On behalf of the Apache Geode PMC
>

Re: Stored procedures on Apache Geode.

[Michael Stolz]

Pivotal provides a closed-source connector to link commercial GemFire to
commercial Greenplum.
Pivotal has no intention of opening that connector at this time.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Fri, Jul 14, 2017 at 2:33 AM, marios390  wrote:

> Hi John,
>
> Just a quick one,
> Geode could be integrated with green plum and if not what alternatives for
> this.
>
> Thanks
> Ms
> 
> From: John Blum [via Apache Geode (Incubating) Developers Forum] [
> ml+s70738n24325...@n6.nabble.com]
> Sent: Thursday, July 13, 2017 11:33 PM
> To: Marios Sofocleous/IT/CREDITSAFE
> Subject: Re: Stored procedures on Apache Geode.
>
> Hi Marios-
>
> It seems you and your team could be / mostly are likely dealing with a
> high-volume of sensitive information, but without knowing a lot about your
> UC(s) or particular application requirements/SLAs, I would recommend
> starting small, simple and scale based on need.
>
> Geode is a highly concurrent and distributed system with strong consistency
> guarantees.  Prematurely breaking the logic down into many individual
> microservices early (especially for individual Stored Procedures... how
> complex are these?) might unduly add complexity to your application and
> system architecture.
>
> So my advice is to really evaluate the need to create individual
> microservices first (which usually involves a platform like [Pivotal's]
> CloudFoundry on an IaaS (or private infra) to manage effectively) vs.
> starting small and just converting the Stored Procs into Geode Functions.
>
> Geode Function executions can be distributed across the cluster (similarly
> to Map-Reduce, but far more robust) in a highly available and reliable
> fashion.
>
> You might want to also read up on Geode's Partitioned Regions [1] for
> effectively managing (partitioning and distributing/arranging your data).
>
> Hope this helps.
>
> -John
>
> [1]
> http://gemfire90.docs.pivotal.io/geode/developing/
> partitioned_regions/chapter_overview.html yrjldnY3TtixZu9rDvLgvAx0M0bPn0iR_AzXTlMikrJPQ7v4gcrUCAFodHRwOi8
> vc2Nhbm1haWwudHJ1c3R3YXZlLmNvbS8_Yz03NDIyJmQ9aXRubjJWd0otbE1xQm
> NKbU01T3Q1RU1ZSHRrUWgyc1E4Q08zemVsZjZnJnU9aHR0cCUzYSUyZiUyZm
> dlbWZpcmU5MCUyZWRvY3MlMmVwaXZvdGFsJTJlaW8lMmZnZW9kZSUyZmRldm
> Vsb3BpbmclMmZwYXJ0aXRpb25lZCU1ZnJlZ2lvbnMlMmZjaGFwdGVyJTVmb3
> ZlcnZpZXclMmVodG1s>
>
>
> On Thu, Jul 13, 2017 at 1:09 PM, marios390 <[hidden
> email]
> > wrote:
>
> >
> > 
> > From: John Blum [via Apache Geode (Incubating) Developers Forum] [
> > [hidden email]]
> > Sent: Thursday, July 13, 2017 9:16 PM
> > To: Marios Sofocleous/IT/CREDITSAFE
> > Subject: Re: Stored procedures on Apache Geode.
> >
> > Right.
> >
> > You can also review the Apache Geode documentation on Function Execution
> > [1].  And if you are a *Spring* user, you can use *Spring Data Geode's*
> > convenient Function annotation support for both Function implementation
> as
> > well as execution, here [2].
> >
> > NOTE: you will probably notice the link [2] refers to *Spring Data
> > GemFire's* docs.  *Spring Data GemFire* and *Spring Data Geode* are
> > virtually the same with no differences.  Eventually *Spring Data Geode*
> > will have its own home with its own doc locations since it is finally
> > becoming a top-level SD module [3].
> >
> > -j
> >
> > [1]
> > http://geode.apache.org/docs/guide/11/developing/function_<
> redir.aspx?REF=9WC8AmXblm4Q_zDPwbP3MZWeTU5A8b8TcK6A7zAo07x
> PQ7v4gcrUCAFodHRwOi8vc2Nhbm1haWwudHJ1c3R3YXZlLmNvbS8_
> Yz03NDIyJmQ9aXRubjJWd0otbE1xQmNKbU01T3Q1RU1ZSHRrUWgyc1E4QzIx
> bWVsWTd3JnU9aHR0cCUzYSUyZiUyZmdlb2RlJTJlYXBhY2hlJTJlb3JnJTJm
> ZG9jcyUyZmd1aWRlJTJmMTElMmZkZXZlbG9waW5nJTJmZnVuY3Rpb24lNWY.>
> > exec/chapter_overview.html  keRPQ7v4gcrUCAFodHRwOi8vc2Nhbm1haWwudHJ1c3R3YXZlLmNvbS8_Yz03NDIyJmQ9>
> > j7nn2bxZOwcngUOelB5ZmyW7oHpBqfFPnKkha5eWJQ&u=http%3a%2f%
> > 2fgeode%2eapache%2eorg%2fdocs%2fguide%2f11%2fdeveloping%
> > 2ffunction%5fexec%2fchapter%5foverview%2ehtml>
> > [2]
> > http://docs.spring.io/spring-data-gemfire/docs/current/ FeI05aGSUMelSXs5C8eNCYF4w7WOQTrcPuikgbSqvPtPQ7v4gcrUCAFodHRw
> Oi8vc2Nhbm1haWwudHJ1c3R3YXZlLmNvbS8_Yz03NDIyJmQ9aXRubjJWd0otbE1xQm
> NKbU01T3Q1RU1ZSHRrUWgyc1E4SExsbmJ0ZTZnJnU9aHR0cCUzYSUyZiUyZm
> RvY3MlMmVzcHJpbmclMmVpbyUyZnNwcmluZy1kYXRhLWdlbWZpcmUlMmZkb2
> NzJTJmY3VycmVudCUyZg..>
> > reference/html/#function-annotations redir.aspx?REF=yjGtUyX_IqBoLNjffMmotpM0mcz7gdL1cjWOZj
> T4YBFPQ7v4gcrUCAFodHRwOi8vc2Nhbm1haWw.>.
> > trustwave.com/?c=7422&d=j7nn2bxZOwcngUOelB5ZmyW7oHpBqf
>  pZUTH3A6h1PQ7v4gcrUCAFodHRwOi8vc2Nhbm1haWwudHJ1c3R3YXZlLmNvbS8_
> Yz03NDIyJmQ9aXRubjJWd0otbE1xQmNKbU01T3Q1RU1ZSHRrUWgyc1E4SGV6
> bUwwRTd3JnU9aHR0cCUzYSUyZiUyZnRydXN0d2F2ZSUyZWNvbSUyZiUzZmMl
> M2Q3NDIyJTI2YW1wJTNiZCUzZGo3bm4yYnhaT3djbmdVT2VsQjVabXlXN29IcEJxZg..>
> > FPnPsrbJ3BIQ&u=http%3a%2f%2fdocs%2espring%2eio%2fspring-
> > data-gemfire%2fdocs%2fc

Re: Review Request 60718: GEODE-2997: New flow getAll/putAll

[Michael Stolz]

We removed error feedback?
So how is an application programmer supposed to determine what failed now?
Without that information we may have rendered putAll unusable for some
cases.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Wed, Jul 12, 2017 at 1:45 PM, Brian Rowe  wrote:

>
>
> > On July 7, 2017, 11:51 p.m., Udo Kohlmeyer wrote:
> > > geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/operations/PutAllRequestOperationHandler.java
> > > Lines 81 (patched)
> > >  file1771594line81>
> > >
> > > I'm sure that we can use `region.putAll(entries)` for this.
>
> Fixed.  This also resulted in us not being able to determine the state of
> a partially succeeded PutAll, so we updated the PutAllResponse to remove
> the failedKeys field.
>
>
> > On July 7, 2017, 11:51 p.m., Udo Kohlmeyer wrote:
> > > geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufRequestUtilities.java
> > > Lines 76 (patched)
> > >  file1771595line76>
> > >
> > > maybe this variable name should be `getAllRequestBuilder`... not
> advocate of generic `builder`
>
> Fixed
>
>
> > On July 7, 2017, 11:51 p.m., Udo Kohlmeyer wrote:
> > > geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufRequestUtilities.java
> > > Lines 78-80 (patched)
> > >  file1771595line78>
> > >
> > > Could we replace this with `addAllKey(java.lang.Iterable extends org.apache.geode.protocol.protobuf.BasicTypes.EncodedValue>
> values)`
>
> Nice, good catch.
>
>
> > On July 7, 2017, 11:51 p.m., Udo Kohlmeyer wrote:
> > > geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufRequestUtilities.java
> > > Lines 95-97 (patched)
> > >  file1771595line95>
> > >
> > > Could we replace this with `addAllEntry(java.lang.Iterable extends org.apache.geode.protocol.protobuf.BasicTypes.Entry> values)`
>
> Fixed
>
>
> > On July 7, 2017, 11:51 p.m., Udo Kohlmeyer wrote:
> > > geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufResponseUtilities.java
> > > Lines 125-127 (patched)
> > >  file1771596line125>
> > >
> > > Could we replace this with `addAllEntries(java.lang.Iterable extends org.apache.geode.protocol.protobuf.BasicTypes.Entry> values)`
>
> Fixed
>
>
> > On July 7, 2017, 11:51 p.m., Udo Kohlmeyer wrote:
> > > geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufResponseUtilities.java
> > > Lines 141-143 (patched)
> > >  file1771596line141>
> > >
> > > could we possibly use `addAllFailedKeys(java.lang.Iterable extends org.apache.geode.protocol.protobuf.BasicTypes.EncodedValue>
> values)`
>
> This got removed with the failed keys field.
>
>
> - Brian
>
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/60718/#review179967
> ---
>
>
> On July 7, 2017, 9:18 p.m., Brian Rowe wrote:
> >
> > ---
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/60718/
> > ---
> >
> > (Updated July 7, 2017, 9:18 p.m.)
> >
> >
> > Review request for geode, Alexander Murmann, Bruce Schuchardt, Galen
> O'Sullivan, Hitesh Khamesra, and Udo Kohlmeyer.
> >
> >
> > Bugs: GEODE-2997
> > https://issues.apache.org/jira/browse/GEODE-2997
> >
> >
> > Repository: geode
> >
> >
> > Description
> > ---
> >
> > Changed get response to indicate if LookupFailure was a missing key or
> key with null value, added test
> > Added GetAllRequestOperationHandler and unit test
> > Added PutAllRequestOperationHandler and unit test
> > Added an integration test covering the putAll and getAll operations
> >
> >
> > Diffs
> > -
> >
> >   
> > geode-protobuf/src/main/java/org/apache/geode/protocol/protobuf/ProtobufStreamProcessor.java
> ebd5c6a0a
> >   geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/operations/GetAllRequestOperationHandler.java PRE-CREATION
> >   geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/operations/PutAllRequestOperationHandler.java PRE-CREATION
> >   geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufRequestUtilities.java b96f478d1
> >   geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/ProtobufResponseUtilities.java 2114fdbf7
> >   geode-protobuf/src/main/java/org/apache/geode/protocol/
> protobuf/utilities/Pr

Re: getSizeInBytes() return type

[Michael Stolz]

I don't think it is appropriate to store objects of 2GB size in Geode.
I know we have hit networking issues with such large objects in the past.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Sun, Jul 9, 2017 at 2:13 AM, Daniel Farcovich <
daniel.farcov...@amdocs.com> wrote:

> Raising this question again.
>
> Daniel
>
> From: Daniel Farcovich
> Sent: Wednesday, July 05, 2017 1:26 PM
> To: 'dev@geode.apache.org' 
> Subject: getSizeInBytes() return type
>
>
> Hi,
> We implement getSizeInBytes() in from Sizeable interface.
> We have objects with size bigger than MAXINT, bigger than 2GB.
> What is the impact of refactoring the code to return long instead of int?
> I mean except the technical aspect of changing the return types of the
> calling functions etc.
> Which mechanisms / functionalities will be affected from this change, I
> know rebalancing will be affected for example.
>
> Thanks,
> Daniel Farcovich
>
>
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

Re: Geode "minor minor" versions

[Michael Stolz]

@Roi Apelker: There is a big effort going on right now in the community to
get a 1.2 release of Geode out.
You can follow the release notes here:
https://cwiki.apache.org/confluence/display/GEODE/Release+Notes#ReleaseNotes-1.2.0

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Wed, Jul 5, 2017 at 10:58 AM, Roi Apelker  wrote:

> I am sorry if I have been misunderstood (and yes I am new :-) )
>
> I did not mean that GEODE is going to stop, I was referring to the subject
> of minor versions.
>
> Is there any plan to release a 1.1.2, or a 1.2.1, or will this be simply
> an "ad hoc" situation, such that only an important bug fix will trigger
> it... I understand now that this is the case.
>
> -Original Message-
> From: Gregory Chase [mailto:gch...@pivotal.io]
> Sent: Wednesday, July 05, 2017 5:25 PM
> To: dev@geode.apache.org
> Subject: Re: Geode "minor minor" versions
>
> You make it sound like Geode is going to stop. That's definitely not true.
>
> What affects Geode's future are the requests the community makes for new
> features. Check out our Jira to know what's in our future.
>
> Since this is an open source Apache community, all the planning occurs
> here in the open.  I'm trusting you are new to our community?
>
> Updates get triggered as needed whenever there is an important bug to fix
> and/or foundational changes that need to be brought to production, even if
> they don't create new features right away.
>
>
> On Tue, Jul 4, 2017 at 4:56 AM, Roi Apelker 
> wrote:
>
> > Hi,
> >
> > I understood from previous discussions, that apart from version 1.3
> > there are no other foreseeable versions in the future.
> >
> >
> > * What about "minor minor" versions - 1.1.2? 1.2.1? - what is the
> > "trigger" for such versions?
> >
> > * Are fixes being released as well? Or is it just that their code
> > is in github?
> >
> > Thanks,
> > Roi
> > This message and the information contained herein is proprietary and
> > confidential and subject to the Amdocs policy statement,
> >
> > you may review at https://www.amdocs.com/about/email-disclaimer <
> > https://www.amdocs.com/about/email-disclaimer>
> >
>
>
>
> --
> Greg Chase
>
> Product team
> http://www.pivotal.io/big-data
>
> Pivotal Software
> http://www.pivotal.io/
>
> 650-215-0477
> @GregChase
> Blog: http://geekmarketing.biz/
> This message and the information contained herein is proprietary and
> confidential and subject to the Amdocs policy statement,
>
> you may review at https://www.amdocs.com/about/email-disclaimer <
> https://www.amdocs.com/about/email-disclaimer>
>

OQL rewriting

[Michael Stolz]

The old security framework had an authorizeOperation method that had enough
information to be able to inspect and modify an OQL string before it would
be executed. That whole framework is now deprecated, but I feel like it's a
really powerful feature being able to modify OQL in such a way as to
support adding some kind of security column to the where clause so you can
implement row-level security on queries.

My question is, are the new securityManager and the old AccessControl
interface able to both be used together or are they mutually exclusive?

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

Re: [VOTE] Apache Geode release - v1.2.0 RC1

[Michael Stolz]

That shouldn't say incubating should  it?

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jun 17, 2017 12:01 PM, "Anthony Baker"  wrote:

> This is the first release candidate for Apache Geode, version
> 1.2.0-incubating.  Thanks to all the community members for their
> contributions to this release!
>
> *** Please download, test and vote by Wednesday, June 19, 1700 hrs US
> Pacific. ***
>
> It fixes the following issues:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?
> projectId=12318420&version=12339257
>
> Note that we are voting upon the source tags:  rel/v1.2.0.RC1
> https://git-wip-us.apache.org/repos/asf?p=geode.git;a=commit;h=
> b6598890d41dc607ca4897ead04a75dd77f3e852
> https://git-wip-us.apache.org/repos/asf?p=geode-examples.
> git;a=commit;h=7f93d95ad06a6f2afee54312585f48435fff11e8
>
> Commit ID:
> b6598890d41dc607ca4897ead04a75dd77f3e852 (geode)
> 7f93d95ad06a6f2afee54312585f48435fff11e8 (geode-examples)
>
> Source and binary files:
>   https://dist.apache.org/repos/dist/dev/geode/1.2.0.RC1
>
> Maven staging repo:
> https://repository.apache.org/content/repositories/orgapachegeode-1019
>
> Geode's KEYS file containing PGP keys we use to sign the release:
> https://git-wip-us.apache.org/repos/asf?p=geode.git;a=blob_
> plain;f=KEYS;hb=HEAD
>
> pub  4096R/C72CFB64 2015-10-01
> Fingerprint=948E 8234 14BE 693A 7F74  ABBE 19DB CAEE C72C FB64
>
> Anthony
>

Re: [DISCUSS] easy string based partitioning

[Michael Stolz]

Actually it appears that parameterizing the delimeter would be quite
problematic for client side single hop.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jun 7, 2017 1:56 PM, "Galen M O'Sullivan"  wrote:

> I don't think it would be that much harder to add an option for a
> user-specified delimiter than it would to hardcode ':' or '|'. I think the
> utility of that would be much higher, and that it would be a much better
> candidate for inclusion in geode-core. Otherwise, I think it should live in
> an "examples" or "utilities" package or repo.
>
> Galen
>
> On Wed, Jun 7, 2017 at 12:47 PM, Jacob Barrett 
> wrote:
>
> > In regard to sending the configuration state to the clients for the
> > partition resolver maybe the config should be a domain object independent
> > of the business object rather than the serialized form of the partition
> > resolver itself. The rational for this would be that non-Java clients
> could
> > get the config as PDX (or some other language neutral scheme) and
> > initialize their native language version of the partition resolver with
> > this config.
> >
> > -Jake
> >
> >
> > Sent from my iPhone
> >
> > > On Jun 7, 2017, at 11:42 AM, Darrel Schneider 
> > wrote:
> > >
> > > Thanks for all the feedback. Since this is such a simple resolver with
> a
> > > fixed delimiter the following changes were made to the original
> proposal:
> > > 1. The StringPrefixPartitionResolver will be in the
> > > "org.apache.geode.cache.util" package.
> > >This was done to keep it from being viewed as part of the core API.
> It
> > > is just a simple implementation
> > >of one of the core API interfaces that user can use if they find it
> > > helpful.
> > > 2. The fixed delimiter was changed from ":" to "|". The character still
> > has
> > > the visual appearance of
> > >delimiting two fields with the benefit of being used less often than
> > > ":". Since the delimiter can not
> > >be configured that makes it more likely someone can use this
> resolver.
> > >
> > > I think the ideas about more complex resolvers are interesting and the
> > > existence of this simple resolver does not prevent other resolver
> > > implementations from also being added to the util package. For example
> > one
> > > that is regex based or spring expression based.
> > >
> > > One of the things discovered was that if a PartitionResolver has state
> > (for
> > > example a configured delimiter, or regex, or spring expression) then
> that
> > > java clients will not be able to single hop to the server region with
> > that
> > > resolver. Our servers send the single hop java clients just the class
> > name
> > > of the PartitionResolver. The instance of the actual resolver is not
> > > serialized to the client. The java client single hop code attempts to
> > > create an instance of it by invoking a no-arg constructor. So any state
> > > stored in the server's resolver instance will be lost. If the resolver
> > > class does not have a no-arg constructor then the java client will be
> > > unable to load an instance and just disable single hop. But if it had
> two
> > > constructors, say one that has no-arg and configures a default
> delimiter
> > > and another that take a delimiter as an arg, then in that case the
> single
> > > hop client would lose the delimiter configured on the server and revert
> > to
> > > the default one from the no-arg constructor. It seems like this could
> > cause
> > > all kinds of bad things to happen. For example in the case of a
> > > StringPrefixPartitionResolver that allows the delimiter to be
> configured
> > > that one used on this client to route keys to the server would have the
> > > wrong delimiter and complain that the key does not contain the
> > delimiter. I
> > > will file a geode ticket about this issue but wanted to have it on this
> > > thread to help explain why this initial resolver is not configurable.
> > >
> > >
> > >> On Tue, Jun 6, 2017 at 1:20 PM, Jacob Barrett 
> > wrote:
> > >>
> > >> I have to agree. Something this trivial an limiting is better suited
> > for a
> > >> blog post or examples somewhere and not a part of the core codebase.
> > >>
> > >> -Jake
> > >>
> > >> On Mon, Jun 5, 2017 at 1:27 PM Udo Kohlmeyer 
> > >> wrote:
> > >>
> > >>> My concern with this approach is that we provide an implementation
> that
> > >>> might be a white elephant and only used by a 1% user base. In
> addition
> > >>> to that, it does really restrict the user on his keys.
> > >>>
> > >>> Whereas something a little more configurable, like a SPEL
> > >>> implementation, would provide a lot more flexibility. Which means
> that
> > >>> one could easily change the partition resolver expression upon region
> > >>> creation/configuration.
> > >>>
> > >>> --Udo
> > >>>
> > >>>
> >  On 6/5/17 08:46, Jens Deppe wrote:
> >  I like the idea of keeping the configuration 'conventional' and thus
> > >> not
> >  requiring any server config

Re: Need authorization to tweet for Apache Geode

[Michael Stolz]

Hi Greg,

I didn't give you my Twitter handle: @MikeStolz1


--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Wed, Jun 7, 2017 at 1:23 PM, Gregory Chase  wrote:

> Hi Jag,
> Thanks for volunteering to help communicate the goodness of @ApacheGeode!
>
> I'll add you to the authorized Twitter team in TweetDeck!
>
> -Greg
>
> On Wed, Jun 7, 2017 at 1:19 PM, Jagdish Mirani  wrote:
>
> > Hello:
> > I would like to be authorized to tweet for Apache Geode - I would like to
> > Tweet from the project handle on behalf of the project.
> >
> > About me.
> >
> > *Twitter handle: *@jagmirani
> > *Role: *I am the Product Marketing Manager for Pivotal GemFire (which is
> > based on Apache Geode). Also, I work closely with our engineering and
> > Product Management teams on our contributions to the Geode project.
> >
> > Please authorize tweeting privileges for me.
> >
> > Regards
> > Jagdish Mirani
> >
>
>
>
> --
> Greg Chase
>
> Product team
> http://www.pivotal.io/big-data
>
> Pivotal Software
> http://www.pivotal.io/
>
> 650-215-0477
> @GregChase
> Blog: http://geekmarketing.biz/
>

Re: Do you want to Tweet for @ApacheGeode?

[Michael Stolz]

Hey Greg,

I could tweet now and then.

--
Mike Stolz
Principal Engineer, GemFire Product Manager
Mobile: +1-631-835-4771

On Tue, Jun 6, 2017 at 12:28 PM, Greg Chase  wrote:

> Greetings Apache Geode community,
> Tweeting for the Apache Geode project in the past has been a community
> effort.
>
> However, our most prolific tweeters are currently not spending much time
> with the project.
>
> As a result @ApacheGeode has fallen silent.
>
> Per ASF recommendations, we manage access to the @ApacheGeode Twitter
> handle through Tweet Deck. Any contributor is welcome to Tweet on behalf of
> the project.  You simply need to ask for access, such as replying to this
> thread.
>
> Presuming no concerns raised by the PMC, access is granted to your personal
> Twitter credentials to access @ApacheGeode using Tweet Deck.
>
> Administrative rights to @ApacheGeode are maintained by several PMC
> members.
>
> The guideline for Tweeting for @ApacheGeode is to promote the project,
> announce any important releases, events or topics, and to help answer
> questions or direct askers to u...@geode.apache.org.
>
> You are also welcome to engage in technical topics of interest to Apache
> Geode users, such as event driven architectures, caching patterns,
> in-memory computing, etc.
>
> We recommend not engaging in bashing other projects or products, or
> engaging in off-topic discussions.
>
> If you are interested in helping Tweet for @ApacheGeode, I invite you to
> volunteer by replying to this thread.
>
> Regards,
>
> -Greg
>

Re: [DISCUSS] easy string based partitioning

[Michael Stolz]

Yep that will work.

--
Mike Stolz
Principal Engineer - Gemfire Product Manager
Mobile: 631-835-4771

On Jun 5, 2017 12:06 PM, "Darrel Schneider"  wrote:

> I pictured the top level parent region (your customer region) as not having
> the StringPrefixPartitionResolver. Instead it would just use string keys
> that have no prefix and use the default resolver.
> It would be each co-located child region (your order region) that would
> have the StringPrefixPartitionResolver and would format its keys to be
> "parentKey:childKey". Does that make sense? Will it work?
>
>
> On Mon, Jun 5, 2017 at 10:06 AM, Barry Oglesby 
> wrote:
>
> > The top-level region may not have / need a delimiter. If you have a
> > customer region and a colocated orders region, the customer region key
> > could be the customerId, and the orders region key could be the
> > customerId:orderId. The colocation would be on the customerId. In the
> > customer region, it doesn't need a delimiter. Is it ok that this idea
> would
> > require one? Maybe the top-level region shouldn't require a delimiter? If
> > the StringPrefixPartitionResolver is using key.split(":"), the customer
> > region would return the entire key.
> >
> >
> > Thanks,
> > Barry Oglesby
> >
> >
> > On Mon, Jun 5, 2017 at 8:46 AM, Jens Deppe  wrote:
> >
> > > I like the idea of keeping the configuration 'conventional' and thus
> not
> > > requiring any server configuration. As soon as you need to define a
> regex
> > > on the server you might as well be writing PartitionResolver code.
> > >
> > > As an aside I also think that using regexes to parse keys would also
> > > introduce a measurable performance hit.
> > >
> > > --Jens
> > >
> > > On Mon, Jun 5, 2017 at 8:21 AM, Udo Kohlmeyer 
> > > wrote:
> > >
> > > > Whilst I like the idea to make something (and yes,
> > > > DelimitedStringPartitionResolver is the simplest), I feel that we
> > might
> > > > be able to do one better.
> > > >
> > > > */Could/* one possibly have an /*SPEL*/ <
> https://docs.spring.io/spring
> > > > -framework/docs/current/spring-framework-reference/
> > > html/expressions.html>-based
> > > > PartitionResolver for this? At least this way, we don't have to rely
> on
> > > > delimiters or a strong knowledge of REGEX.
> > > >
> > > > IMO, this would provide the greatest bang for buck implementation.
> > > >
> > > > --Udo
> > > >
> > > >
> > > >
> > > >
> > > > On 6/2/17 19:15, Jacob Barrett wrote:
> > > >
> > > >> If you implement as regular expression the user doesn't have to
> > reformat
> > > >> their key to a specific format (akin to making them implement a
> > class).
> > > I
> > > >> would concat the matching groups for generate the routing key.
> > > >>
> > > >> Consider RegEx: .*\bcorrelation=(\d+).*\bmaybe-something-else=(\w)
> > > >> With Keys:
> > > >> A: my,key;with:any-chars;unique=12345;correlation=678/and,maybe
> > > >> -something-else=a
> > > >> B: my,key;unique=876324;correlation=678;and,maybe-
> > something-else=a,foo
> > > >> C: somthing;different=988975;correlation=678;then,maybe-
> > > something-else=ba
> > > >>
> > > >> Keys A and B would have routing key '678a'. Key C would have routing
> > key
> > > >> '678b'.
> > > >>
> > > >> -Jake
> > > >>
> > > >>
> > > >>
> > > >> Consider
> > > >>
> > > >> On Fri, Jun 2, 2017 at 4:02 PM Darrel Schneider <
> > dschnei...@pivotal.io>
> > > >> wrote:
> > > >>
> > > >> Geode partitioned regions usually partition the data based on the
> > key's
> > > >>> hashcode.
> > > >>> You can do your own partitioning by implementing the
> > PartitionResolver
> > > >>> interface and configuring it on the partitioned region.
> > > >>>
> > > >>> In some use cases needing to deploy your class that implements
> > > >>> PartitionResolver can be problematic so we would like to find a way
> > to
> > > >>> offer partitioning based on a portion of the key (instead of the
> > > default
> > > >>> which uses the entire key) that does not require you to implement
> > your
> > > >>> own
> > > >>> PartitionResolver and does not require you to deploy your own code
> to
> > > do
> > > >>> the custom partitioning.
> > > >>>
> > > >>> Another group of users that do not want to implement
> > PartitionResolver
> > > >>> are
> > > >>> non-java clients. So the solution is required to be usable by
> > non-java
> > > >>> geode clients without needing to reimplement the client to support
> a
> > > new
> > > >>> feature.
> > > >>>
> > > >>> Another constraint on the solution is for it to be both easy to use
> > and
> > > >>> easy to implement.
> > > >>>
> > > >>> The proposed solution is to provide a class named:
> > > >>>  org.apache.geode.cache.StringPrefixPartitionResolver
> > > >>> This class will implement PartitionResolver and have a default
> > > >>> constructor.
> > > >>> To use it you need to configure a partitioned region's
> > > PartitionResolver
> > > >>> using the already existing mechanism for this (api, gfsh, or xml).
> > > >>> The StringPrefixPartitionResolver will re