Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Udo Kohlmeyer]

I think at this point I'd be looking at the new V2 Management API's for 
Regions.


I think any new "public" effort that we'd be adding to the product 
should be done through the Management API's for Regions, rather than 
exposing new public API's that in reality should not be made "public".


--Udo

On 12/11/19 3:53 PM, Mark Hanson wrote:

Basically the point is to allow a use to copy a RegionFactory because under 
certain circumstances it is necessary. I found that when migrating from 
AttributesFactory.

Thanks,
Mark


On Dec 11, 2019, at 3:48 PM, Anthony Baker  wrote:

Mark,

Can you share how the API changes will help the user?

Thanks,
Anthony



On Dec 11, 2019, at 2:57 PM, Mark Hanson  wrote:

Hi All,

There was a suggestion that since I am making a couple user visible API changes 
that I might want to ask the dev list.

Basically I was migrating code from AttributesFactory to RegionFactory and hit 
a few snags along the way.

Please see https://github.com/apache/geode/pull/4409 
 specifically Cache.java, 
RegionFactory.java, and for extra credit GemfireCacheImpl.java

I have commented at the relevant changes.

Thanks,
Mark

Re: [VOTE] Release candidate for Apache Geode version 1.11.0.RC3.

[Mark Hanson]

Hi All,

It does not look like we have an assignee for GEODE-7531. Any takers?

Thanks,
Mark

> On Dec 4, 2019, at 2:35 PM, Mark Hanson  wrote:
> 
> So, outstanding issues that I see right now are 
> 
> GEODE-7531
> GEODE-7537
> GEODE-7538
> 
> Thanks,
> Mark
> 
>> On Dec 4, 2019, at 2:11 PM, John Blum  wrote:
>> 
>> This is not a test failure in SDG.  SDG builds fine with Apache Geode 1.11
>> (and all tests pass), as I indicated above in my origin +0 vote.
>> 
>> This is a definitive problem for SBDG when using STDG to mock Apache Geode
>> resources/objects, which is caused by GEODE-7531.
>> 
>> Either way, the design/code changes made in GEODE-6759 were poor and should
>> be fixed regardless which GEODE-7531 describes.
>> 
>> -j
>> 
>> 
>> On Wed, Dec 4, 2019 at 2:04 PM Dan Smith  wrote:
>> 
>>> On Wed, Dec 4, 2019 at 11:16 AM John Blum  wrote:
>>> 
 I am changing my vote to -1!
 
 I have filed GEODE-7531 <
>>> https://issues.apache.org/jira/browse/GEODE-7531>
 [1],
 which is a serious blocking issue for all things *Spring* for Apache
 Geode.  This issue alone is currently preventing me from upgrading
>>> *Spring
 Boot for Apache Geode* (SBDG) to Apache Geode 1.10, which I plan to do in
 SBDG 1.3, which is based on *Spring Data for Apache Geode* (SDG)
 Neumann/2.3, which is currently already pulling in Apache Geode 1.10,
>>> soon
 to be upgraded to 1.11 once this issue is resolved and 1.11 is available.
 
>>> 
>>> 
>>> I commented on the above JIRA. While we definitely don't want to break
>>> spring data geode, it looks like maybe the issue is just with one unit test
>>> in Spring Data Geode using an internal geode API to inject something into a
>>> singleton? If that's the case, I think it would be better to fix that one
>>> test in SDG.
>>> 
>>> -Dan
>>> 
>> 
>> 
>> -- 
>> -John
>> Spring Data Team
> 

Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Mark Hanson]

Yup, fixing that.

> On Dec 11, 2019, at 3:38 PM, Darrel Schneider  wrote:
> 
> Don't expose "InternalCache" on RegionFactory. You probably want it to be
> "Cache".
> 
> On Wed, Dec 11, 2019 at 3:35 PM Mark Hanson  wrote:
> 
>> 
>> In Cache.java
>> 
>> +   RegionFactory createRegionFactory(RegionFactory
>> regionFactory);
>> 
>> In RegionFactory.java
>> + public RegionFactory(InternalCache cache, RegionFactory
>> regionFactory)
>> and
>> + public RegionFactory(RegionFactory regionFactory)
>> 
>> Lastly in GemfireCacheImpl.java
>> + public  RegionFactory createRegionFactory(RegionFactory> V> regionFactory)
>> 
>> Thanks,
>> Mark
>> 
>> 
>>> On Dec 11, 2019, at 3:25 PM, Dan Smith  wrote:
>>> 
>>> I see a lot of PR comments on those PRs. What is the new API you added?
>>> 
>>> -Dan
>>> 
>>> On Wed, Dec 11, 2019 at 2:57 PM Mark Hanson  wrote:
>>> 
 Hi All,
 
 There was a suggestion that since I am making a couple user visible API
 changes that I might want to ask the dev list.
 
 Basically I was migrating code from AttributesFactory to RegionFactory
>> and
 hit a few snags along the way.
 
 Please see https://github.com/apache/geode/pull/4409 <
 https://github.com/apache/geode/pull/4409> specifically Cache.java,
 RegionFactory.java, and for extra credit GemfireCacheImpl.java
 
 I have commented at the relevant changes.
 
 Thanks,
 Mark
>> 
>> 

Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Mark Hanson]

Basically the point is to allow a use to copy a RegionFactory because under 
certain circumstances it is necessary. I found that when migrating from 
AttributesFactory.

Thanks,
Mark

> On Dec 11, 2019, at 3:48 PM, Anthony Baker  wrote:
> 
> Mark,
> 
> Can you share how the API changes will help the user?  
> 
> Thanks,
> Anthony
> 
> 
>> On Dec 11, 2019, at 2:57 PM, Mark Hanson  wrote:
>> 
>> Hi All,
>> 
>> There was a suggestion that since I am making a couple user visible API 
>> changes that I might want to ask the dev list.
>> 
>> Basically I was migrating code from AttributesFactory to RegionFactory and 
>> hit a few snags along the way.
>> 
>> Please see https://github.com/apache/geode/pull/4409 
>>  specifically Cache.java, 
>> RegionFactory.java, and for extra credit GemfireCacheImpl.java
>> 
>> I have commented at the relevant changes.
>> 
>> Thanks,
>> Mark
> 

Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Anthony Baker]

Mark,

Can you share how the API changes will help the user?  

Thanks,
Anthony


> On Dec 11, 2019, at 2:57 PM, Mark Hanson  wrote:
> 
> Hi All,
> 
> There was a suggestion that since I am making a couple user visible API 
> changes that I might want to ask the dev list.
> 
> Basically I was migrating code from AttributesFactory to RegionFactory and 
> hit a few snags along the way.
> 
> Please see https://github.com/apache/geode/pull/4409 
>  specifically Cache.java, 
> RegionFactory.java, and for extra credit GemfireCacheImpl.java
> 
> I have commented at the relevant changes.
> 
> Thanks,
> Mark

Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Darrel Schneider]

Don't expose "InternalCache" on RegionFactory. You probably want it to be
"Cache".

On Wed, Dec 11, 2019 at 3:35 PM Mark Hanson  wrote:

>
> In Cache.java
>
> +   RegionFactory createRegionFactory(RegionFactory
> regionFactory);
>
> In RegionFactory.java
>  + public RegionFactory(InternalCache cache, RegionFactory
> regionFactory)
> and
> + public RegionFactory(RegionFactory regionFactory)
>
> Lastly in GemfireCacheImpl.java
> + public  RegionFactory createRegionFactory(RegionFactory V> regionFactory)
>
> Thanks,
> Mark
>
>
> > On Dec 11, 2019, at 3:25 PM, Dan Smith  wrote:
> >
> > I see a lot of PR comments on those PRs. What is the new API you added?
> >
> > -Dan
> >
> > On Wed, Dec 11, 2019 at 2:57 PM Mark Hanson  wrote:
> >
> >> Hi All,
> >>
> >> There was a suggestion that since I am making a couple user visible API
> >> changes that I might want to ask the dev list.
> >>
> >> Basically I was migrating code from AttributesFactory to RegionFactory
> and
> >> hit a few snags along the way.
> >>
> >> Please see https://github.com/apache/geode/pull/4409 <
> >> https://github.com/apache/geode/pull/4409> specifically Cache.java,
> >> RegionFactory.java, and for extra credit GemfireCacheImpl.java
> >>
> >> I have commented at the relevant changes.
> >>
> >> Thanks,
> >> Mark
>
>

Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Mark Hanson]

In Cache.java

+   RegionFactory createRegionFactory(RegionFactory 
regionFactory);

In RegionFactory.java
 + public RegionFactory(InternalCache cache, RegionFactory regionFactory)
and
+ public RegionFactory(RegionFactory regionFactory)

Lastly in GemfireCacheImpl.java
+ public  RegionFactory createRegionFactory(RegionFactory 
regionFactory) 

Thanks,
Mark


> On Dec 11, 2019, at 3:25 PM, Dan Smith  wrote:
> 
> I see a lot of PR comments on those PRs. What is the new API you added?
> 
> -Dan
> 
> On Wed, Dec 11, 2019 at 2:57 PM Mark Hanson  wrote:
> 
>> Hi All,
>> 
>> There was a suggestion that since I am making a couple user visible API
>> changes that I might want to ask the dev list.
>> 
>> Basically I was migrating code from AttributesFactory to RegionFactory and
>> hit a few snags along the way.
>> 
>> Please see https://github.com/apache/geode/pull/4409 <
>> https://github.com/apache/geode/pull/4409> specifically Cache.java,
>> RegionFactory.java, and for extra credit GemfireCacheImpl.java
>> 
>> I have commented at the relevant changes.
>> 
>> Thanks,
>> Mark

Re: Request GEODE-7537 fix to be cherry-picked into release 1.11

[Owen Nichols]

+1 for bringing this to release/1.11.0

> On Dec 11, 2019, at 2:57 PM, Eric Shu  wrote:
> 
> Hello,
> 
> GEODE-7537 was highlighted as blocking 1.11 release. This issue is being
> addressed now.
> 
> I would like to have this fix to be merged into release 1.11.
> 
> The commit sha is 3a3db82afdf0668e98ca49bcbdfa478e0ac94e10
> 
> Regards,
> Eric

Re: [DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Dan Smith]

I see a lot of PR comments on those PRs. What is the new API you added?

-Dan

On Wed, Dec 11, 2019 at 2:57 PM Mark Hanson  wrote:

> Hi All,
>
> There was a suggestion that since I am making a couple user visible API
> changes that I might want to ask the dev list.
>
> Basically I was migrating code from AttributesFactory to RegionFactory and
> hit a few snags along the way.
>
> Please see https://github.com/apache/geode/pull/4409 <
> https://github.com/apache/geode/pull/4409> specifically Cache.java,
> RegionFactory.java, and for extra credit GemfireCacheImpl.java
>
> I have commented at the relevant changes.
>
> Thanks,
> Mark

[DISCUSS] Adding a couple user APIs dealing with RegionFactory.

[Mark Hanson]

Hi All,

There was a suggestion that since I am making a couple user visible API changes 
that I might want to ask the dev list.

Basically I was migrating code from AttributesFactory to RegionFactory and hit 
a few snags along the way.

Please see https://github.com/apache/geode/pull/4409 
 specifically Cache.java, 
RegionFactory.java, and for extra credit GemfireCacheImpl.java

I have commented at the relevant changes.

Thanks,
Mark

Request GEODE-7537 fix to be cherry-picked into release 1.11

[Eric Shu]

Hello,

GEODE-7537 was highlighted as blocking 1.11 release. This issue is being
addressed now.

I would like to have this fix to be merged into release 1.11.

The commit sha is 3a3db82afdf0668e98ca49bcbdfa478e0ac94e10

Regards,
Eric

Re: Request GEODE-7510/GEODE-7538 be cherry-picked into release 1.11

[Jason Huynh]

I believe we'll have to cherry-pick 1448c83c2a910b2891b4c13f1b4cbed2920252de
across.  Unfortunately it went in as a merge (there was a hiccup on squash
and merge where the try-again didn't do a squash merge :-(.




On Wed, Dec 11, 2019 at 11:30 AM Mark Hanson  wrote:

> Can I get the SHA of the commit?
>
> Thanks,
> Mark
>
> > On Dec 11, 2019, at 11:02 AM, Jason Huynh  wrote:
> >
> > Hello,
> >
> > GEODE-7538 was highlighted as blocking the 1.11 release.  This has now
> been
> > addressed and propose that this gets merged over to release 1.11.
> >
> > This issue solves a few things, most notably: GEODE-7510 shows
> > inconsistency between secondaries and primaries.  GEODE-7538 showed
> > operations not consistently being applied.  The code change is a revert
> > that modified profile calculation.  It may affect other areas that were
> > showing up as flaky as it's timing related.
> >
> > Thanks,
> > -Jason
>
>

Re: Request GEODE-7510/GEODE-7538 be cherry-picked into release 1.11

[Mark Hanson]

Can I get the SHA of the commit?

Thanks,
Mark

> On Dec 11, 2019, at 11:02 AM, Jason Huynh  wrote:
> 
> Hello,
> 
> GEODE-7538 was highlighted as blocking the 1.11 release.  This has now been
> addressed and propose that this gets merged over to release 1.11.
> 
> This issue solves a few things, most notably: GEODE-7510 shows
> inconsistency between secondaries and primaries.  GEODE-7538 showed
> operations not consistently being applied.  The code change is a revert
> that modified profile calculation.  It may affect other areas that were
> showing up as flaky as it's timing related.
> 
> Thanks,
> -Jason

Request GEODE-7510/GEODE-7538 be cherry-picked into release 1.11

[Jason Huynh]

Hello,

GEODE-7538 was highlighted as blocking the 1.11 release.  This has now been
addressed and propose that this gets merged over to release 1.11.

This issue solves a few things, most notably: GEODE-7510 shows
inconsistency between secondaries and primaries.  GEODE-7538 showed
operations not consistently being applied.  The code change is a revert
that modified profile calculation.  It may affect other areas that were
showing up as flaky as it's timing related.

Thanks,
-Jason

Odg: Odg: Odg: Lucene upgrade

[Mario Kevo]

Hi Jason,

This change fix IndexFormatTooNewException, but now we have

 org.apache.geode.cache.lucene.LuceneQueryException: Lucene Index is not 
available, currently indexing

So this means that query doesn't wait until all indexes are created.
In LuceneQueryFunction.java it is set to not wait for repo [execute(context, 
false)]. If we have a bigger queue(like in the test) it will failed as it will 
not wait until indexes are created. I also tried to put just few objects and it 
passed as it had enough time to create indexes.
Do we need to change this part to wait for repo, or put a lower number of 
entries in tests?

BR,
Mario




Šalje: Jason Huynh 
Poslano: 6. prosinca 2019. 20:53
Prima: Mario Kevo 
Kopija: geode 
Predmet: Re: Odg: Odg: Lucene upgrade

Hi Mario,

I made a PR against your branch for some of the changes I had to do to get past 
the Index too new exception.  Summary - repo creation, even if no writes occur, 
appear to create some meta data that the old node attempts to read and blow up 
on.

The pr against your branch just prevents the repo from being constructed until 
all old members are upgraded.
This requires test changes to not try to validate using queries (since we 
prevent draining and repo creation, the query will just wait)

The reason why you probably were seeing unsuccessful dispatches, is because we 
kind of intended for that with the oldMember check.  In-between the server 
rolls, the test was trying to verify, but because not all servers had upgraded, 
the LuceneEventListener wasn't allowing the queue to drain on the new member.

I am not sure if the changes I added are acceptable or not -maybe if this ends 
up working then we can discuss on the dev list.

There will probably be other "gotcha's" along the way...


On Fri, Dec 6, 2019 at 1:12 AM Mario Kevo  wrote:
Hi Jason,

I tried to upgrade from 6.6.2 to 7.1.0 and got the following exception:

org.apache.lucene.index.IndexFormatTooNewException: Format version is not 
supported (resource BufferedChecksumIndexInput(segments_2)): 7 (needs to be 
between 4 and 6)

It looks like the fix is not good.

What I see (from 
RollingUpgradeQueryReturnsCorrectResultsAfterServersRollOverOnPartitionRegion.java)
 is when it doing upgrade of a locator it will shutdown and started on the 
newer version. The problem is that server2 become a lead and cannot read lucene 
index on the newer version(Lucene index format has changed between 6 and 7 
versions).

Another problem is after the rolling upgrade of locator and server1 when 
verifying region size on VMs. For example,

expectedRegionSize += 5;
putSerializableObjectAndVerifyLuceneQueryResult(server1, regionName, 
expectedRegionSize, 5,
15, server2, server3);

First it checks if region has expected size for VMs and it passed(has 15 
entries). The problem is while executing verifyLuceneQueryResults, for 
VM1(server2) it has 13 entries and assertion failed.
>From logs it can be seen that two batches are unsuccessfully dispatched:

[vm0] [warn 2019/12/06 08:31:39.956 CET  tid=0x42] During normal 
processing, unsuccessfully dispatched 1 events (batch #0)

[vm0] [warn 2019/12/06 08:31:40.103 CET  tid=0x46] During normal 
processing, unsuccessfully dispatched 1 events (batch #0)

For VM0(server1) and VM2(server3) it has 14 entries, one is unsuccessfully 
dispatched.

I don't know why some events are successfully dispatched, some not.
Do you have any idea?

BR,
Mario



Šalje: Jason Huynh mailto:jhu...@pivotal.io>>
Poslano: 2. prosinca 2019. 18:32
Prima: geode mailto:dev@geode.apache.org>>
Predmet: Re: Odg: Lucene upgrade

Hi Mario,

Sorry I reread the original email and see that the exception points to a
different problem.. I think your fix addresses an old version seeing an
unknown new lucene format, which looks good.  The following exception looks
like it's the new lucene library not being able to read the older files
(Just a guess from the message)...

Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format
version is not supported (resource
BufferedChecksumIndexInput(segments_1)): 6 (needs to be between 7 and
9). This version of Lucene only supports indexes created with release
6.0 and later.

The upgrade is from 6.6.2 -> 8.x though, so I am not sure if the message is
incorrect (stating needs to be release 6.0 and later) or if it requires an
intermediate upgrade between 6.6.2 -> 7.x -> 8.





On Mon, Dec 2, 2019 at 2:00 AM Mario Kevo  wrote:

>
> I started with implementation of Option-1.
> As I understood the idea is to block all puts(put them in the queue) until
> all members are upgraded. After that it will process all queued events.
>
> I tried with Dan's proposal to check on start of
> LuceneEventListener.process() if all members are upgraded, also changed
> test to verify lucene indexes only after all members are upgraded, but got
> the same error with incompatibilities between lucene versions.
> Changes 

Odg: PRs review

[Mario Kevo]

Thank you @Bruce Schuchardt! 

Šalje: Bruce Schuchardt 
Poslano: 10. prosinca 2019. 23:53
Prima: dev@geode.apache.org 
Predmet: Re: PRs review

Mario, I've merged GEODE-6927.  You can close the JIRA ticket.

On 12/10/19 4:20 AM, Mario Kevo wrote:
> Hi Geode dev,
>
> Need some PR reviewers on the following PRs.
>
> JIRA: https://issues.apache.org/jira/browse/GEODE-6927
> PR: https://github.com/apache/geode/pull/4085
>
>
> JIRA: https://issues.apache.org/jira/browse/GEODE-7561
> PR: https://github.com/apache/geode/pull/4441
>
> BR,
> Mario
>