Re: oak-resilience

[Philipp Suter]

Plus some ideas what Aurora DB from AWS uses for resilience testing:

http://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Aurora.Managing.html#Aurora.Managing.FaultInjectionQueries




On 08/03/16 11:19, "Michael Dürig"  wrote:

>
>+= out of file descriptors
>
>Michael
>
>On 8.3.16 11:17 , Michael Marth wrote:
>> Love it!
>>
>> Ideas in addition what was already mentioned:
>> * network deterioration in Cold Standby setups
>> * OOM in TarMK setups (either on-heap or off-heap)
>> * out of disc space in TarMK setups
>> * out of disc space for persistent cache
>>
>>
>>
>> On 07/03/16 09:30, "Chetan Mehrotra"  wrote:
>>
>>> Cool stuff Tomek! This was something which was discussed in last
>>> Oakathon so great to have a way to do resilience testing
>>> programatically. Would give it a try
>>> Chetan Mehrotra
>>>
>>>
>>> On Mon, Mar 7, 2016 at 1:49 PM, Stefan Egli  wrote:
 Hi Tomek,

 Would also be interesting to see the effect on the leases and thus
 discovery-lite under high memory load and network problems.

 Cheers,
 Stefan

 On 04/03/16 11:13, "Tomek Rekawek"  wrote:

> Hello,
>
> For some time I've worked on a little project called oak-resilience. It
> aims to be a resilience testing framework for the Oak. It uses
> virtualisation to run Java code in a controlled environment, that can be
> spoilt in different ways, by:
>
> * resetting the machine,
> * filling the JVM memory,
> * filling the disk,
> * breaking or deteriorating the network.
>
> I described currently supported features in the README file [1].
>
> Now, once I have a hammer I'm looking for a nail. Could you share your
> thoughts on areas/features in Oak which may benefit from being
> systematically tested for the resilience in the way described above?
>
> Best regards,
> Tomek
>
> [1]
> https://github.com/trekawek/jackrabbit-oak/tree/resilience/oak-resilience
>
> --
> Tomek Rękawek | Adobe Research | www.adobe.com
> reka...@adobe.com
>

Re: OAK-3884 for UDBBroadcaster

[Philipp Suter]

The patch actually fixes the test that was broadcasting to the loopback 
interface :)

Cheers,
Philipp





On 15/01/16 09:30, "ianbos...@gmail.com on behalf of Ian Boston" 
<ianbos...@gmail.com on behalf of i...@tfd.co.uk> wrote:

>Hi,
>Sorry. I misunderstood your question.
>The Serverfault question referenced in the Oak issue was about broadcasting
>to the loopback address, which can't be done.
>The patch looked like it was attempting to do that.
>
>Best Regards
>Ian
>
>
>On 14 January 2016 at 20:48, Philipp Suter <su...@adobe.com> wrote:
>
>> The patch is for unit tests “only”. It executes the unit tests on the
>> first interface address that has the BROADCAST option already configured.
>> Such the loopback interface is not taken into account.
>>
>> The test will fail if there is no interface address that has the BROADCAST
>> option configured.
>>
>> All of that could also be solved differently, e.g. with a virtual network.
>> I am not sure if that exists for unit testing.
>>
>> Cheers,
>>
>> Philipp
>>
>>
>> On 14/01/16 19:41, "ianbos...@gmail.com on behalf of Ian Boston" <
>> ianbos...@gmail.com on behalf of i...@tfd.co.uk> wrote:
>>
>> >Hi,
>> >
>> >Does the patch work ?
>> >
>> >According to the answer in the serverfault article referenced in OAK-3884
>> >it should not
>> >
>> >I tried the pattern referenced on OSX using nc and it doesn't work. The
>> >original poster seems to think it works, but those answering disagree and
>> >the posted wasn't able to tell them which kernel it worked on.
>> >
>> >
>> >"The "solution" you are asking for does not exist (or at least *should not
>> >work*) on any platform unless the loopback interface supports BROADCAST
>> (as
>> >can be determined by the flags field in ifconfig) -- The platform(s) you
>> >are talking about *do not advertise support for broadcasting on the
>> >loopback interface*, therefore you cannot do what you're asking."
>> >
>> >
>> >There are some other less complimentary comments.
>> >
>> >It might be possible, with root access to the test machine, to setup
>> >several tun interfaces, connected to a bridge to create a virtual network
>> >on the same machine. You can do the same with multiple docker hosts on the
>> >same machine but all of that requires some setup that a Java process
>> >isnt going to be able to do.
>> >
>> >
>> >
>> >
>> >
>> >For a non loopback network you should not try and work out the broadcast
>> >address. IIRC you should set the broadcast flag on the UDP packet. (I
>> >assume UDB == UDP ?).
>> >
>> >I assume you are doing something like:
>> >
>> >  socket = new DatagramSocket(, InetAddress.getByName("0.0.0.0"));
>> >  socket.setBroadcast(true);
>> >
>> >
>> >Hosts on the same subnet will have the same network mast, otherwise they
>> >are not on the same subnet. All sorts of things will start to fail. eg if
>> >some are /24 and some are /25 all the broadcasts on the /25 subnet will be
>> >directed at the .127 host on the /24 subnet. (I  haven't tried to see what
>> >a switch does with 2 overlapping and misconfigured subnets.).
>> >
>> >
>> >
>> >HTH
>> >Best Regards
>> >Ian
>> >
>> >
>> >
>> >
>> >
>> >On 14 January 2016 at 17:30, Philipp Suter <su...@adobe.com> wrote:
>> >
>> >> Hi
>> >>
>> >> I added a small patch to https://issues.apache.org/jira/browse/OAK-3884
>> >> that could fix the broadcast unit tests for UDBBroadcaster. It seems the
>> >> loopback interface is not allowing broadcasting on *NIX systems. The
>> >> broadcasting IP has to be found dynamically for a test execution.
>> >>
>> >> Interesting next step: How could this be configured dynamically within a
>> >> clustered set-up? It needs an agreement among all cluster members to use
>> >> the same network mask.
>> >>
>> >> Cheers,
>> >> Philipp
>> >>
>> >>
>> >>
>>

OAK-3884 for UDBBroadcaster

[Philipp Suter]

Hi

I added a small patch to https://issues.apache.org/jira/browse/OAK-3884 that 
could fix the broadcast unit tests for UDBBroadcaster. It seems the loopback 
interface is not allowing broadcasting on *NIX systems. The broadcasting IP has 
to be found dynamically for a test execution.

Interesting next step: How could this be configured dynamically within a 
clustered set-up? It needs an agreement among all cluster members to use the 
same network mask.

Cheers,
Philipp

Re: OAK-3884 for UDBBroadcaster

[Philipp Suter]

The patch is for unit tests “only”. It executes the unit tests on the first 
interface address that has the BROADCAST option already configured. Such the 
loopback interface is not taken into account.

The test will fail if there is no interface address that has the BROADCAST 
option configured. 

All of that could also be solved differently, e.g. with a virtual network. I am 
not sure if that exists for unit testing.

Cheers,

Philipp


On 14/01/16 19:41, "ianbos...@gmail.com on behalf of Ian Boston" 
<ianbos...@gmail.com on behalf of i...@tfd.co.uk> wrote:

>Hi,
>
>Does the patch work ?
>
>According to the answer in the serverfault article referenced in OAK-3884
>it should not
>
>I tried the pattern referenced on OSX using nc and it doesn't work. The
>original poster seems to think it works, but those answering disagree and
>the posted wasn't able to tell them which kernel it worked on.
>
>
>"The "solution" you are asking for does not exist (or at least *should not
>work*) on any platform unless the loopback interface supports BROADCAST (as
>can be determined by the flags field in ifconfig) -- The platform(s) you
>are talking about *do not advertise support for broadcasting on the
>loopback interface*, therefore you cannot do what you're asking."
>
>
>There are some other less complimentary comments.
>
>It might be possible, with root access to the test machine, to setup
>several tun interfaces, connected to a bridge to create a virtual network
>on the same machine. You can do the same with multiple docker hosts on the
>same machine but all of that requires some setup that a Java process
>isnt going to be able to do.
>
>
>
>
>
>For a non loopback network you should not try and work out the broadcast
>address. IIRC you should set the broadcast flag on the UDP packet. (I
>assume UDB == UDP ?).
>
>I assume you are doing something like:
>
>  socket = new DatagramSocket(, InetAddress.getByName("0.0.0.0"));
>  socket.setBroadcast(true);
>
>
>Hosts on the same subnet will have the same network mast, otherwise they
>are not on the same subnet. All sorts of things will start to fail. eg if
>some are /24 and some are /25 all the broadcasts on the /25 subnet will be
>directed at the .127 host on the /24 subnet. (I  haven't tried to see what
>a switch does with 2 overlapping and misconfigured subnets.).
>
>
>
>HTH
>Best Regards
>Ian
>
>
>
>
>
>On 14 January 2016 at 17:30, Philipp Suter <su...@adobe.com> wrote:
>
>> Hi
>>
>> I added a small patch to https://issues.apache.org/jira/browse/OAK-3884
>> that could fix the broadcast unit tests for UDBBroadcaster. It seems the
>> loopback interface is not allowing broadcasting on *NIX systems. The
>> broadcasting IP has to be found dynamically for a test execution.
>>
>> Interesting next step: How could this be configured dynamically within a
>> clustered set-up? It needs an agreement among all cluster members to use
>> the same network mask.
>>
>> Cheers,
>> Philipp
>>
>>
>>

Re: Release dates

[Philipp Suter]

Actual release data would be great to map planing dependencies properly.



On 13/08/15 13:58, Davide Giannella dav...@apache.org wrote:

On 13/08/2015 10:17, Stefan Egli wrote:
 I¹d find it more useful (for us) when it would be the cut-date.

It's always the cut date in the beginning. Shall we change it then to
the actual release date once it's announced or leave it to the cut date? :)

Davide

Re: Modularization

[Philipp Suter]

I took some inspiration from Apache Aries for this hierarchical maven 
structure. It has very fine granular OSGi modules (not that we have to do that 
too) and still keeps everything in one hierarchical Maven project. 

With a hierarchical maven project structure you can build/ test/ install either 
all of the projects in one go or just one of them.

I don’t think that it makes sense to have multiple separated maven projects for 
OAK. There will always be dependencies on each other and with a hierarchical 
structure these are kept in one place. 

And you need some kind of “root” bundle that loads all dependencies within an 
OSGi container. In the OAK case most likely oak-jcr or oak-core.



On 07/08/15 11:31, Thomas Mueller muel...@adobe.com wrote:

Hi,

I have nothing against modularization, I'm just against modularization =
create many many Maven projects. I prefer modularization *within* one
project. Why can't we do that instead?

Ideally you have a ³root² project, e.g.

/oak
  /security
/api
/implementationA
/implementationB
  /core
  /persistence
  /..

That looks like a Java *package* structure to me. The Wikipedia article
you mentioned is not about Maven projects, but about modularity in general.

Regards,
Thomas

Re: Modularization

[Philipp Suter]

On 06/08/15 13:36, Davide Giannella dav...@apache.org wrote:


Grouping two emails together

On 06/08/2015 12:04, Michael Dürig wrote:
 I neither like Sling's constantly moving towards smaller and more
 specific modules as too me this seems to be mostly driven by OSGi as
 a technology and not by modularisation as a way to balance complexity,
 maintainability and reusability. This might be opinionated and
 ignorant due to my overall lack of Sling knowledge though. So let's
 not get sidetracked with a Sling fight here ;-)


  Having many many Maven projects makes

 See my proposal. This is not about having many Maven projects but
 reasonable module boundaries. The definition of which is low cohesion
 between modules and hight cohesion within them.

 To quote from the Wikipedia article linked from the Stackoverflow post
 you cited (emphasis mine): While generally desirable from a coupling
 and cohesion perspective, *overzealous separation* and encapsulation
 of code can bloat call stacks and make navigation through the code for
 maintenance purposes more difficult.

 overzealous separation being key here!
...

 I'd prefer to slowly move towards a more modularised Oak. One such an
approach could be:

 - Move NodeStore implementations into their own modules.

 - Leverage OSGi sub-systems to hide Oak's private parts (i.e. all that
component we export just for ourselves).

 - Fine tune over time, evaluate what worked and what not and follow up
from there.

Once again I totally share Michael's point. It seems you're reading my
mind :)

Davide

I also like the idea of slowly moving towards modularisation. In the beginning 
we could start with “only” subsystems in OSGi and move the stable API parts out 
into single bundles over time. That would allow for an initial set of modules 
that can be re-viewed/ refactored a few releases later without invalidating the 
exposed contract.

Re: Modularization

[Philipp Suter]

Ideally you have a “root” project, e.g. 

/oak
  /security
/api
/implementationA
/implementationB
  /core
  /persistence
  /..

Apache Aries has for example a pretty nicely structured hierarchical Maven 
set-up. You can still build all of it starting on top..

In the end smaller modules help theoretically with

- Risk assessment for deployments
- Test planing
- Rollouts
- Cleaner architecture
- Separation of concern
- Separation of necessary dependencies from unnecessary dependencies 
- Customer specific implementation of API

https://en.wikipedia.org/wiki/Modular_programming#Key_aspects






On 06/08/15 15:28, Thomas Mueller muel...@adobe.com wrote:

Hi,

 we already have 2 modules for access control

That's fine... What I meant is, we shouldn't try to move oak-core
org.apache.jackrabbit.oak.security.** plus
org.apache.jackrabbit.oak.spi.security.** to new Maven projects. Or should
we?

 jcr (top-bread)
 core (meat with all the validation)
persistence


I still don't see a reason why we should have persistence separate Maven
project(s). What would it bring us? We anyway don't plan to combine
persistence 1.2 with core 1.0, because we don't have testing in place.

On the other side, I see many complications. We have datastore and
blobstore, Mongo persistence from the RDB persistence are not completely
independent (and they should not be), we have persisence wrappers
(TimingDocumentStoreWrapper for example), caching (including persistent
cache), discovery lite... If we want to create separate Maven projects for
each of those (plus API projects, plus commons projects), we would
easily have more Maven projects than we have Java packages right now.
Sure, you could say we just create one persistence Maven project... but
what for, really?

Regards,
Thomas

(OAK-2416) Support continuable sessions

[Philipp Suter]

Hi

I had an offline discussion with Francesco regarding the implementation of the 
revision for the continuable session in the OAK API and read through the 
comments in [1].

Some additional comments to what Alex mentioned yesterday:

I like the idea of separating the internal revision from the “exposed 
revision. They might not have the same requirements. Especially an optional 
timeout for the exposed revision similar to the checkpoint timeout could be 
ideal to support efficient garbage collection.

The timeout should have a larger timeframe than for checkpoints. E.g. days 
rather than milliseconds. The timeout should also be optional to have enough 
flexibility for fast changing trees and for more static trees.

For trees with too many “eternal revisions a resetAll function in the 
RevisionFactory could be a solution. It could also help for any yet unknown 
locked-in situations.

Also the creation of a RevisionFactory would be necessary to create exposed 
revisions, load revisions and dispose revisions, etcetc

[1] https://issues.apache.org/jira/browse/OAK-2416