Re: Trunk instability

[Emmanuel Lécharny]

On 1/29/11 10:20 PM, Alex Karasulu wrote:

Hi y'all,

This trunk situation is just horrible. I don't know how I fell into working
this heavily in the trunk. We all usually branch and work isolated. I in
fact, I started refactoring in a branch. I don't know how this shifted into
the trunk. It matters not though, so long as this never happens again. I
sure am stressed over it and the time others are loosing.

It started with some good intentions :
- all the changes were supposed to be automatically impacting studio, 
the API and ADS

- they were not supposed to be so heavy
- it was suppose to be done fast
- merging when many directories change is just a PITA to manage with SVN

That being said, the initial changes leaded to some more changes, then 
some others etc. It was an attempt to catch many birds with one stone.


IMO, and I don't want to blame anyone on that, some really important 
rules about development on a complex project have also been forgotten :

- trunk must build. Always.
- before starting to add new features, the previous one must be done
- when some task takes longer than expected, then usually we won't be 
able to finish it in time before being hit but our human physical limits

- big refactoring should start in a branch...

On the bright side, we've got some great decoupling going on that will make
life much easier, and the codec clearer, especially for use by those who
will have to extend it for their own controls and extended operations. I
prepared some hand written materials to share on these topics but we've been
scrambling, trying to get this done fast. I'd like to prepare these
materials formally after we stabilize.
The decoupling is a major step forward. If we had only focused on it, we 
would have been done earlier.


OTOH, using generics, creating a mechanism to allow codec injection, 
could have been done once the initial decoupling has been terminated. 
Working using 'Sprints', or whatever we name them, is simply better than 
any other options we have when it's not an initial work.

These changes are an important step even if we don't achieve full
pluggability by M1 because it helps us clearly see what MUST be expose to
users of the codec (API) and people extending the codec (SPI). We ain't
gonna get fancy with OSGi either right now. Floating ideas on the ML is cool
but right now we need this M1 out the door. My primary motive was to get the
API roughly demarcated. This way huge shifts in the API between milestones
don't have to happen. We don't want to freak users out. Our milestone
contract gives us some liberty but maybe we should still try to be
conservative with API changes. So that's why NOW is the best time to get
this done fast before the M1. The rest of the milestone series should not
have such massive API changes.

Totally agree.

So let's just not panic about a day or two. (I say this because I am
panicing) I will dedicate my time to get this done ASAP. We even talked
about backing out etc with Emmanuel. But I think it would be an incredible
loss at this point. We're going to have to do this at some point. Why not
now and be done with it? Emmanuel the codec master is also equally involved.
It takes a lot to change when things have settled over the past 6 years in
this shared area. But do realize, we have lots of tests, and most of the
changes are not fundamental to the way the software works. The changes are
high level structural changes. We have not changed the way we encode and
decode, we've simply shifted exposed APIs. The new incongruities introduced
will disappear fast and things will align again. We need to be confident in
that. What worked before will work again.
I would say that : if tomorrow evening we don't get trunk back on rail, 
compiling and tests passing, then I'll suggest to revert to something 
which at least compile. I'm not even sure if we should not revert 
tomorrow morning, create a branch, and finish what we are doing in the 
branch (the cost of branching, reverting, and merging should be around 2 
hours for one person.)


Let's see what tomorrow brings, and then we decide what to do.

Thanks for the heads up, Alex

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Trunk fixing

[Emmanuel Lécharny]

On 1/30/11 4:51 PM, Alex Karasulu wrote:

Emmanuel SVN up. I had fixed all the compilation problems but then the
conflicts broke things again when I SVN up. I merged then fixed again. You
should have all of shared building now.


Can you revert the changes you've made on control tests ? Or fix the tests.

May be it's compiling, but tests are now failing, while I got the code 
compiling *and* tests passing.


Thanks.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Maven 3.0 plugin dependencies

[Emmanuel Lécharny]

On 1/31/11 1:17 PM, Stefan Seelmann wrote:

On Thu, Jan 27, 2011 at 3:40 PM, Emmanuel Lecharny  wrote:

On 1/27/11 3:31 PM, Felix Knecht wrote:

On 01/27/2011 03:17 PM, Emmanuel Lecharny wrote:

Hi again.

I agree that it does not make a lot of sense to move deps into trunks.

We may want to release a new version of the 'project' project with all
the changes done those last days.

Maybe it's worth to wait for the latest Apache pom.xml released soon [1]


[1] http://markmail.org/message/c6ml53rozserwopm




We are still using version 7 :)

What is going to happen if we switch to version 8 first ?

I updated to version 8. The release of version 9 of the Apache POM
seems to take some more time, some new Jiras were opened [2].

So I'm going to prepare a release for org.apache.directory.project:project:19.


Great ! Thanks a lot !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Control interface

[Emmanuel Lécharny]

Comments inline...

On 1/31/11 2:59 PM, Alex Karasulu wrote:

(3) Regardless of what is stored in the value, no application can just use
it as is without some kind of component to decode, parse, or interpret this
binary data. It's useless to API users otherwise. It's opaque according to
LDAP, but there has to be some meaning to make use of this content, and that
always requires some code to convert that binary data into usable
information. A useful representation to an API user is a parameterized
representation making the opaque structure transparent.


I can imagine a use case where someone has 2 LDAP servers, he get some 
response from server A, with a control he does not know anything about, 
and has to send the value stored into the control without having to 
decode it, to send it as a control in a request sent to server B.


Ok, a bit far fetched, but this *can* be a use case.

But let's think about the replication control now. It works in a way we 
get a cooie from the server we are contacting, and we are supposed to 
send this cookie back to the server. What is in the cookie is irrelevant 
to the client. Only the server  knows about it. If the client just have 
to do a getValue() to get the cookie, and a setValue( cookie ) to store 
it in the request control, then we don't need to define a specific 
control : using a basic control with the correct OID and criticality is 
enough.


Assuming we have access to the getValue() setValue() methods of course...


--- o ---

I'll try to make some sense out of my disorderly thoughts now.

(1) The application coder writing the logic to use the control, may not be
the same entity writing/designing the control. The application needs a
transparent parameterized view of the binary opaque data to do anything
sensible with it. Using the data just in pass through situations may occur
but it's the exception and not the rule.

Yep.

(2) Regardless of the binary value encoding (i.e. json, xml, BER), it can be
marshaled into a distinct data representation that makes sense for
application coders to use.  So if a specific Control subtype in addition to
the type (OID) and criticality properties, exposed all properties needed to
act on that control's content, then these properties can be populated. This
way it's actionable data that can directly be used instead of some useless
binary payload.

yep.

(3) It's up to control designers/implementers, to declare what the structure
of their control is. They do this through ASN.1 for BER encoded content for
example. For our codec they should so so by providing a POJI and simple POJO
implementation of their control extending our Control interface which only
has the properties: oid, and criticality.

Yep.

(4) Expose the POJO and POJI to control users. Yet hide the details of
encoding and decoding. This really does not matter to the control user who
just wants to implement some application logic. It matters not to him if the
data is encoded as JSON, XML, or ASN.1 BER. Just just wants to make the
application logic function based on the information inside. Expose this yet
hide the implementation, (encoding) details.

Yep.

(5) Make a pluggable framework which supports #4. Let the control
implementor declare (POJI/POJO), and implement (components for encoding and
decoding content) the control's codec extension. The end result to users is
the same regardless of the implementation details.

Yep.

(6) API's like JNDI left the pass thru door open but did it by exposing the
implementation and potentially allowing the user to shoot themselves in the
foot. Exposing the value is what I am referring to. Really that's fine if
that's all you're going to do but decide on one path. Either expose it and
don't bother representing the opaque data as actionable transparent typed
information. Or don't expose it at all and just let application code deal
with the logical typed information.
I agree. But closing this door also leads to some situation like the one 
I mentionned.


Now, let's be clear : I don't really care, we will cover probably 99.9% 
of all our client usages by not exposing the getValue()/setValue() 
methods. I'm just saying that it may force our users to implement their 
own classes to handle the controls we don't provide, unless we add them 
in the code base.


Is this what we want to do ?


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Trunk is building again : Next steps

[Emmanuel Lécharny]

On 2/1/11 4:56 PM, Alex Karasulu wrote:

On Tue, Feb 1, 2011 at 5:37 PM, Emmanuel Lecharnywrote:


Hi guys,

So we get a building trunk as of today.

We can now move to the next step. There is some cleanup to do in the code
we injected last week, as we were hurrying to get something that works. Here
is a list of tasks to complete :
- Review the CodecService usage


+1



- Remove the CodecService from the DSML grammars


+1 and some other areas:

Where we create new DefaultLdapCodecService instances?


shared-ldap-client-api =>  LdapNetworkConnection

- needed

apacheds-core  =>  DefaultDirectoryService.initialize()

- needed

apacheds-core-api   =>  LdapCoreSessionConnection()

- totally unnecessary: the DirectoryService can be used to get a handle

shared-dsmlv2-parser =>  Dsmlv2Grammar

- unnecessary

shared-dsmlv2-parser =>  Dsmlv2ResponseGrammar

shared-ldap   =>  LdapEncoder

- review


The encoder should not create a CodecService. It's never called out of 
the blue, so the caller can pass it to the encoder.

studio-connection.core=>  CursorStudioNamingEnumeration

- unnecessary

studio-ldapbrowser.core =>  ExportDsmlRunnable

- unnecessary

studio-ldapbrowser.core =>  ImportDsmlRunnable

- unnecessary

Can review again after dust settles to get this organized correctly.





- Rename the *I*xxx interfaces


I don't like this Ixxx interface naming but it did help in many places to my
surprise. It for example narrows searching for an interface to implement a
great deal because of the I in front. I think it should be considered - at
least we need definitive consensus on whether to go with it or not.

I can go either way - I just don't think this should be turned into a "oh no
not again this converstation." Our situation in releasing an API warrants it
one last time at most.

In this case, none of those *I*xxx interfaces are visible from the API.

As we have some inconsistent code atm, I'd suggest we stick to what we 
decided years ago and get back to the standard notation (ie Xxx for 
interfaces and XxxImpl for implementations).

- Review the contol encoding (we need to avoid a double call to the
computeLength() method)


Yep and we need to make sure we're not double wrapping with decorators.

I had trouble with this piece of code.

Right now, to get it working, I moved the control envelop encoding out 
of the ControlDecorator class. The reason was that the encode() methods 
was used to encode the Control's value.


The problem with this approach is that we can't anymore stored the 
control's length when we have computed it in the control, so when we 
encode it, we have to compute it again.


I'd like to add a computeEnvelopLength() and encodeEnvelop() methods in 
the ControlDecorator, and move back the code from the LdapEncoder class 
down to this class.


wdyt ?


- Add the missing Javadocs


+1



- Relome the duplicated fields


+1


I think it's already done



Once done, we will have some refactoring to do :
- move the controls in one single package (right now, they are spread in
any places)


+1

I can push a separate thread on this matter since it might distract from the
high level discussion we have here.
yes, a separate thread could be good, but I don't think there is much to 
discuss here. It's probably more about giving some information.

- check that we have a clear separation between teh API and the SPI (the
extended operation might be a problem here)


+1

What's been done for controls need to be done for extended operations.


Yep.

Last, not least, the PasswordPolicy tests have been ignored, we shoumd move
them to server-integ and make them work.



+1

We need to get all server tests out of core and core-api. There are many of
them.

hmmm, not sure.

We need a list here.

Let's start !

PS : And remember : no trunk breakage, no big refactoring without 
previous discussion on the ML, branching could be the way to go.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: One more thing about the Control hierarchy

[Emmanuel Lécharny]

 Users intuit the OpaqueControl is one whose value cannot be handled.

Agreed.

o The final OpaqueControl class prevents users from accidental extension.

Well, we may allow a user to extend this class. Not sure...

o Code that cannot access the codec, yet has access to the encoded
value for all practical purposes see that control as opaque, if they
need to such code can use this class to encapsulate the value to
interface with code requiring subtypes of the model's Control
interface, the codec can convert it if need be deep below. This is the
only exception to the visibility of the value property and this way it
is OK.

Agreed.

 o The BaseControl is hidden and can only be used by codec
extenders as part of the SPI
Again, I don't see why we should hide the BaseControl in the codec. We 
already have decorators for that. And if this OpaqueControl has a 
getValue() setValue() methods, the getValue() method will return a 
byte[], so it does not need to have a specific encoder/decoder. Although 
we need to create an OpaqueControlDecorator to manage the envelop encoding.



 o When the codec cannot handle a control it creates the OpaqueControl type.

Agreed.

On another note we might need a final OpaqueControlDecorator which
does not need to keep a value member as it's own member but can access
the value via OpaqueControl.getValue().


Absolutely.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Problems trying to get back in sync with ADS

[Emmanuel Lécharny]

On 2/3/11 9:55 PM, Richard Feezel wrote:

Ok, I've done a checkout of trunk and "mvn install -DskipTests" was
successful.
Curiously, after doing "mvn eclipse:clean eclipse:eclipse" I imported the
projects into Eclipse and I get this error:
Access restriction: The type AuthorizationData is not accessible due to
restriction on required library /usr/java/jdk1.6.0_21/jre/lib/rt.jar
KdcReqBody.java
/apacheds-kerberos-codec/src/main/java/org/apache/directory/shared/kerberos/components
line 211Java Problem

Is this reference to a sun-specific class going to be removed in future
work?  If not, how do I address this problem?

Can you svn up and give it a try ?


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Fwd: New Autoexport version to be installed into Confluence

[Emmanuel Lécharny]

Can someone take care of the templates we have for the Directory (and 
MINA) confluence spaces ? We just need to save them somewhere, just in case.


Thanks !

 Original Message 
From:   - Fri Feb 4 07:10:18 2011
X-Account-Key:  account2
X-UIDL: GmailId12def49c6e0a1aba
X-Mozilla-Status:   0001
X-Mozilla-Status2:  
X-Mozilla-Keys: 
Delivered-To:   elecha...@gmail.com
Received: 	by 10.90.34.10 with SMTP id h10cs30712agh; Thu, 3 Feb 2011 
22:09:05 -0800 (PST)
Received: 	by 10.151.83.14 with SMTP id 
k14mr1382ybl.161.1296799745586; Thu, 03 Feb 2011 22:09:05 -0800 (PST)
Return-Path: 

Received: 	from minotaur.apache.org (minotaur.apache.org [140.211.11.9]) 
by mx.google.com with SMTP id w42si3384165ybe.20.2011.02.03.22.09.05; 
Thu, 03 Feb 2011 22:09:05 -0800 (PST)
Received-SPF: 	pass (google.com: best guess record for domain of 
infrastructure-return-56588-apmail-elecharny=apache@apache.org 
designates 140.211.11.9 as permitted sender) client-ip=140.211.11.9;
Authentication-Results: 	mx.google.com; spf=pass (google.com: best guess 
record for domain of 
infrastructure-return-56588-apmail-elecharny=apache@apache.org 
designates 140.211.11.9 as permitted sender) 
smtp.mail=infrastructure-return-56588-apmail-elecharny=apache@apache.org 


Received:   (qmail 38164 invoked by uid 2021); 4 Feb 2011 06:09:01 -
Delivered-To:   elecha...@minotaur.apache.org
Received:   (qmail 37693 invoked from network); 4 Feb 2011 06:08:55 -
Received: 	from hermes.apache.org (HELO mail.apache.org) (140.211.11.3) 
by minotaur.apache.org with SMTP; 4 Feb 2011 06:08:55 -

Received:   (qmail 75030 invoked by uid 500); 4 Feb 2011 06:08:54 -
Delivered-To:   apmail-elecha...@apache.org
Received:   (qmail 74417 invoked by uid 500); 4 Feb 2011 06:08:51 -
Mailing-List:   contact infrastructure-h...@apache.org; run by ezmlm
Precedence: bulk
X-No-Archive:   yes
List-Post:  
List-Help:  
List-Unsubscribe:   
List-Id:"Apache Infrastructure" 
Delivered-To:   mailing list infrastruct...@apache.org
Received:   (qmail 74409 invoked by uid 99); 4 Feb 2011 06:08:50 -
Received: 	from athena.apache.org (HELO athena.apache.org) 
(140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 
2011 06:08:50 +
X-ASF-Spam-Status: 	No, hits=0.7 required=5.0 
tests=RCVD_IN_DNSWL_NONE,SPF_NEUTRAL

X-Spam-Check-By:apache.org
Received-SPF:   neutral (athena.apache.org: local policy)
Received: 	from [61.9.168.152] (HELO nskntmtas06p.mx.bigpond.com) 
(61.9.168.152) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 04 Feb 2011 
06:08:44 +
Received: 	from nskntotgx02p.mx.bigpond.com ([124.177.170.156]) by 
nskntmtas06p.mx.bigpond.com with ESMTP id 
<20110204060821.ours4368.nskntmtas06p.mx.bigpond@nskntotgx02p.mx.bigpond.com> 
for ; Fri, 4 Feb 2011 06:08:21 +
Received: 	from deltaflyer ([124.177.170.156]) by 
nskntotgx02p.mx.bigpond.com with ESMTP id 
<20110204060821.HEPX4790.nskntotgx02p.mx.bigpond.com@deltaflyer> for 
; Fri, 4 Feb 2011 06:08:21 +

From:   Gav... 
To: 
Subject:New Autoexport version to be installed into Confluence
Date:   Fri, 4 Feb 2011 16:08:18 +1000
Message-ID: <003201cbc431$eb0954d0$c11bfe70$@com.au>
MIME-Version:   1.0
Content-Type:   text/plain; charset="us-ascii"
Content-Transfer-Encoding:  7bit
X-Mailer:   Microsoft Office Outlook 12.0
Content-Language:   en-au
thread-index:   AcvEMemt2DasqIOATVeGQEttN57X1g==
x-cr-hashedpuzzle: 	AYBF ArW/ Aum7 BbxP Brpy Bzy7 CVXu D06X EcS9 Emth 
Fbl6 Fem6 IoBU I6hh KQn/ 
KfXK;1;aQBuAGYAcgBhAHMAdAByAHUAYwB0AHUAcgBlAEAAYQBwAGEAYwBoAGUALgBvAHIAZwA=;Sosha1_v1;7;{0FB5917B-FAA3-4EBE-A17C-3349BAC7320B};ZwBhAHYAaQBuAEAAMQA2AGQAZQBnAHIAZQBlAHMALgBjAG8AbQAuAGEAdQA=;Fri, 
04 Feb 2011 06:08:16 
GMT;TgBlAHcAIABBAHUAdABvAGUAeABwAG8AcgB0ACAAdgBlAHIAcwBpAG8AbgAgAHQAbwAgAGIAZQAgAGkAbgBzAHQAYQBsAGwAZQBkACAAaQBuAHQAbwAgAEMAbwBuAGYAbAB1AGUAbgBjAGUA 


x-cr-puzzleid:  {0FB5917B-FAA3-4EBE-A17C-3349BAC7320B}
X-RPD-ScanID: 	Class unknown; VirusThreatLevel unknown, RefID 
str=0001.0A150203.4D4B97D5.009F,ss=1,fgs=0
X-SIH-MSG-ID: 
rhkwENz8TFelgj0khjaxblklnljh0SNytt9NBYd6+kVFXEfBvd/fRM29cqRB34/j0i46dEOVSDRyc7zgTY/XuNm4K75QSrLZ5vk2 





Hi All,

On Sunday 6th February I intend to remove the current AutoExport version we
are
currently using (-dkulp1.0 altered by Uli :) ) and replace it with the new
version now hosted and maintained at Atlassian version [1].

Any project with altered templates might want to consider saving them
elsewhere
before this change occurs.

Gav...


 [1] - https://plugins.atlassian.com/plugin/details/33010

Fwd: Re: [Shared] Refactoring update

[Emmanuel Lécharny]

Sent it to Alex only, but the target was the ML...


 Original Message 





































Alex,

it's juts fine if it's just about creating new modules and spread the
existing packages around. Just be sure to contact pajbam before merging
back into trunk, or even better, check with him what's required OSGi
wise to get M1 working fine, on monday.

Frankly, I'm not scared either worried, I think we are now very close to
the end of this big needed refactoring.

I must also admit I haven't checked a lot of the last commits, as I was
more busy testing some german excellent beers !
Guys, I ate so much food here that I had to drink a lot of beers and
schnapps in order to get hydrated ... Though I wasn't drunk, because the
food was in such a quantity that it diluted the alcohol...

Baseline, with the sun we had, it was an *exzellent* week-end, much
needed ! I also slept a lot, something I forgot was possible :)



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [Shared] Release plan for 1.0-m1

[Emmanuel Lécharny]

On 2/8/11 12:37 PM, Stefan Seelmann wrote:

On Tue, Feb 8, 2011 at 12:20 PM, Emmanuel Lecharny  wrote:

On 2/7/11 10:49 PM, Stefan Seelmann wrote:

On Mon, Feb 7, 2011 at 6:03 PM, Alex Karasulu
  wrote:

Hi all,
Now with this check pointed merge back to trunks which is about to happen
in
15-20 minutes after some super clean, build, test, run of studio etc, we
can
release shared 1.0-m1 if y'all like. Here's a release plan that Pierre
and I
discussed:
(0) confirm a stable build in trunk with merge
(1) commit merge back to trunk
(2) assign release manager
(3) modify pom's for new 1.0-m1 version
(4) confirm stability
(5) tag, kick off vote
(6) if vote passes, release
(7) merge back trunk changes with 1.0-m2-SNAPSHOT version into m1
(8) rename m1 branches to m2
In the meantime right after step #1 we can still continue work on stage
II
in the m1 branch.

Before a shared release we need to release three other artifactes:
- project: with updated plugin versions
- checkstyle-configuration: currently a snapshot is used
- junit-addons: currently a snapshot is used

So let's start with them, then we'll see the progress of the m1 branch.

Correction : I updated the following plugins :
maven-changes-plugin and maven-ear-plugin

Thanks. There is also a new version 2.3.4 of the Felix
maven-bundle-plugin, I'll update that too, test the build, and start
the release.


One more plugin to update :

[INFO] 


[INFO] Building ApacheDS Manuals 1.5.8-SNAPSHOT
[INFO] 


[INFO]
[INFO] --- versions-maven-plugin:1.2:display-plugin-updates 
(default-cli) @ manuals ---

[INFO]
[INFO] All plugins with a version specified are using the latest versions.
[INFO]
[WARNING] The following plugins do not have their version specified:
[WARNING]   com.agilejava.docbkx:docbkx-maven-plugin ... 
(unknown)




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [Shared] Last night's efforts

[Emmanuel Lécharny]

On 2/8/11 1:10 PM, Alex Karasulu wrote:

On Tue, Feb 8, 2011 at 12:26 PM, Emmanuel Lecharny  wrote:


Just wondering : wouldn't it be a good idea to release M1 as is, and
switching to M2 for felix integration ? That would mean I have to inject my
modifications into trunk, but that should not be a big problem.

thoughts ?

I thought this was what we agreed to do all along. Did something
change? I think we made a comment that if we can progress in the m1
branch further we can maybe merge again but that's only if it's
stable.

The felix stuff is not very stable right now. It's very odd in
behavior so I'm not going to be comfortable with a merge for a while.
All these additional cache directories and file locking makes the
concurrent tests all fickle. I don't want Hudson freaking out with
false positives either.


Ok, so we are just golden.

My fixes (warning removals) have been injected into m1, and I don't 
think there is any urgency to have them merged back to trunk.


Let's release trunk as is, that would make the API available. We can 
work in the m1 branch in the mean time, probably rename it to m2...


Thanks Alex !

Regards,
Alex




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Concurrent classes in junit-addons

[Emmanuel Lécharny]

On 2/8/11 1:43 PM, Alex Karasulu wrote:

On Tue, Feb 8, 2011 at 11:40 AM, Emmanuel Lecharny  wrote:

On 2/8/11 10:32 AM, Stefan Seelmann wrote:

On Tue, Feb 8, 2011 at 9:20 AM, Stefan Seelmann
  wrote:

I found something over at stackoverflow to define different compile
versions for main and test code [4], I'll test that.

Obviously that won't work because JDK5 doesn't have a Java6 compiler.

So what I would suggest is to copy the classes from [5] with the
original package names and license/copyright headers to our
junit-addons module. As the classes have the ALv2 license header we
are safe to copy them. An advantage is that once we switch to Java6 we
can drop those classes and use the com.mycila:mycila-junit dependency.

Thoughts?

Go for it.

I'm not sure we can drop Java 5 now, many users are still using it...

One more reason why they should switch to Java 6. Java 5 is way slower.

IMO we should not have to keep building for it. Java 7 is out. We're
supporting 6&  7.


Frankly, I wish we can drop Java 5 completely, but I'm afraid it's still 
widely used in enterprises :

http://www.theserverside.com/discussions/thread.tss?thread_id=61645

Even if this poll is not realistic, it says that 34% of the companies 
are still using Java 5 :/ :

"Question four: At work, what JVM do you target compilation for?

Java 6 won again, with 247 responses (58%); Java 1.4 got 33 (8%), Java 5 
got 146 (34%). One person said they used Retroweaver for clients with 1.4."





--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Concurrent classes in junit-addons

[Emmanuel Lécharny]

On 2/8/11 5:59 PM, Alex Karasulu wrote:

On Tue, Feb 8, 2011 at 6:53 PM, Emmanuel Lécharny  wrote:

On 2/8/11 1:43 PM, Alex Karasulu wrote:

On Tue, Feb 8, 2011 at 11:40 AM, Emmanuel Lecharny
  wrote:

On 2/8/11 10:32 AM, Stefan Seelmann wrote:

On Tue, Feb 8, 2011 at 9:20 AM, Stefan Seelmann
  wrote:

I found something over at stackoverflow to define different compile
versions for main and test code [4], I'll test that.

Obviously that won't work because JDK5 doesn't have a Java6 compiler.

So what I would suggest is to copy the classes from [5] with the
original package names and license/copyright headers to our
junit-addons module. As the classes have the ALv2 license header we
are safe to copy them. An advantage is that once we switch to Java6 we
can drop those classes and use the com.mycila:mycila-junit dependency.

Thoughts?

Go for it.

I'm not sure we can drop Java 5 now, many users are still using it...

One more reason why they should switch to Java 6. Java 5 is way slower.

IMO we should not have to keep building for it. Java 7 is out. We're
supporting 6&7.

Frankly, I wish we can drop Java 5 completely, but I'm afraid it's still
widely used in enterprises :
http://www.theserverside.com/discussions/thread.tss?thread_id=61645

Even if this poll is not realistic, it says that 34% of the companies are
still using Java 5 :/ :
"Question four: At work, what JVM do you target compilation for?

Java 6 won again, with 247 responses (58%); Java 1.4 got 33 (8%), Java 5 got
146 (34%). One person said they used Retroweaver for clients with 1.4."

I hear you. I'd rather focus on the majority. And soon this will
change. By the time we're out the door with a GA it's not going to be
a matter of contention.

When hunting I always used to target slightly ahead of a running
animal. Why? Because I like killing bambi. Well you get my point ...

It's no sweat off my back because I'm just not dealing with the
overhead of this. Thought it might spare others managing it some pain.


All in all, switching to Java 6 is not *that* bad. i'm just wondering if 
the problem we have with the concurrent test tooling (which requires 
Java6) can't be workarounded by requiring the code to be compiled using 
Java 5 and run with Java 6.


If not, then, well, let's go for Java 6

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Concurrent classes in junit-addons

[Emmanuel Lécharny]

On 2/9/11 12:18 AM, Stefan Seelmann wrote:



All in all, switching to Java 6 is not *that* bad. i'm just wondering if the
problem we have with the concurrent test tooling (which requires Java6)
can't be workarounded by requiring the code to be compiled using Java 5 and
run with Java 6.

If not, then, well, let's go for Java 6

I think a test dependency (even when helpful by speeding up the build
and detecting concurrency issues) isn't worth that step.

I agree with Emmanuel that Java 5 or older is still used in companies
(just like in my current project). But I think Java 5 is mainly used
by commercial software like the fat application servers or ERP
software. If a company decides to use ApacheDS we can expect that they
know that there is something beyond Java 5.
I was more specifically thinking about companies using ADS for unit 
tests. I have some clients doing that.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: shared M1 release, or not ?

[Emmanuel Lécharny]

One more thought : if we release a 1.0-m1, it would probably be a good 
idea not to advertize it atm. Ie, no announcement on apache announce.


On the dev mailing list, we should also make it *very* clear that it's 
just a preliminary release, likely to be replaced very soon.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Release JUnit Add-ons 0.1 - take 2

[Emmanuel Lécharny]

On 2/10/11 3:26 AM, Alex Karasulu wrote:

On Thu, Feb 10, 2011 at 12:42 AM, Emmanuel Lecharny  wrote:

On 2/9/11 8:08 PM, Alex Karasulu wrote:

On Wed, Feb 9, 2011 at 8:45 PM, Stefan Seelmann
  wrote:

On Wed, Feb 9, 2011 at 7:36 PM, Stefan Seelmann
  wrote:

On Wed, Feb 9, 2011 at 5:08 PM, Pierre-Arnaud Marcelot
  wrote:

Thinking it twice, yeah, I'm also afraid you're right Alex.
This vote should probably have needed a longer and more formal vote.

I guess the artifacts have already been promoted from Nexus to the
Maven repository now.
What should we do about it?

Yes, I already promoted the artifacts. I'll ask infra if they can stop
it.

It's already in Maven Central, so I guess it's too late:

http://repo1.maven.org/maven2/org/apache/directory/junit/junit-addons/0.1/

The world's not going to come to an end. We've learned a lesson and we
sure will NOT do it again next time. We're also very rushed and things
like this happen when you rush. We need to relax. We're here to have
fun, not to feel like we screwed up.

BTW I just saw your last email with apology:

 "Ok, I'll prepare the shared release. Sorry for the mess."

I must apologize as well I voted +1 when you asked about this in
October. Should have paid more attention then. I thought the
junit-addons like skins (for only our site) and checkstyle (for only
our project) had no code, just configs or css. Then saw the code being
copied into it. Anyways I'm sorry I let up my guard. I'll try to be
harder.

FWIW, you are not personally at fault. Don't beat yourself over the
head. This is a failure of the whole PMC. Let's note what happened,
rectify and move on. No big deal, everyone had good intentions and
apparently we were all one way or another misinformed.

It's funny how anal the ASF gets about this stuff and apparently for
good very reason! See how we slid right into the situation? We were
all relatively unaware of it happening too. Perhaps this TLP POM
release expedition procedure opened the door for it.

Hey, no need to prepare the nails, the whip and the pieces of wood to
crucify someone :)

Who am I crucifying exactly? I think I went out my way to make sure we
all shared the slip up instead of a single individual, which is in
fact the case. These kinds of of comments above do not add any value
to the conversation.
No, you got me wrong, or may be I was not clear enough. I was not 
pointing out anyone here, I was just saying that we should not beat us 
to death in this case. What you said was perfectly right and I just 
wanted to push the same message, with a taint of humor ! Seems like I 
miss the target :-)


Sometime some +1 are just enough...

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Release fo shared

[Emmanuel Lécharny]

Comments on line

Thanks.

I'm not sure when those notices should/must be added.


Let's try to figure out...

It's clear, when distributing a binary distribution (e.g.
ldap-api.zip) where third-party dependencies are included that the
licenses and notices of those third-party dependencies have to be
added.

+1


But is the attribution also required in the JARs (both, binary or
source, there in META-INF/LICENSE and META-INF/NOTICE) that are
distributed via maven?

Depends...


I see the following different cases:
1) In shared-ldap-model we use Antlr to generate Java files. So I
think in the distributed shared-ldap-model-X.Y.Z.jar the Antlr
attribution is required.

+1

2) The common case that a 3rd-party libary is used/linked in main code
(e.g. dom4j or slf4j). Our distributed JAR only contains our
.java/.class files. The third-party jar is not redistributed. The
dom4j and slf4j licenses say that attribution is required in case the
software is 'used'. Does 'use' already include the case that their
classes are linked? But in that case we
As soon as we distribute something which makes necessary to include a 
thrid party jar, I think we should also include the 3rd party licenses.


Remember that we release *sources*, not binaries. Binaries are just 
generated for convenience. But in any case, we release in order for 
users to be able to get our packages, and use them in their own 
products. Somehow, we have to make them safe when doing so, that means 
include the mandatory licenses and notice to spare the the burden to do so.


At least, this is how I understand the way we should do things at the ASF...


3) Similar like 2, but the 3rd-party is only used as test dependency
(like junit). Here the code is not distributed at all.
Still, we distribute sources, which means tests, and users should be 
able to build the project by downloading our sources. That include 
tests. Of course, we don't distribute the associated jars (I was 
thinking about findbugs), so in this case, we are not forced to inject 
the associated license. Tests are supposed to be run using Maven, 
pointing to external dependencies we *don't* provide. However, I still 
think it's safe to add a reference to the used libs in the NOTICE.



4) 3rd-party source code is included (e.g. in apacheds/jdbm or in
junit-addons). Here it is clear that attribution is required.

+1

Note that this is my perception of the way we should handle those 
license/notice thingy. I may be wrong...


Hope it helps...

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [Shared] [LDAP API] Additional classes and method arguments for GSSAPI binds

[Emmanuel Lécharny]

On 2/14/11 6:44 PM, Pierre-Arnaud Marcelot wrote:

On lundi 14 février 2011 at 17:29, Alex Karasulu wrote:

On Mon, Feb 14, 2011 at 12:15 PM, Emmanuel Lecharny  wrote:

On 2/14/11 11:05 AM, Stefan Seelmann wrote:

  On Mon, Feb 14, 2011 at 9:05 AM, Pierre-Arnaud Marcelot  
wrote:

  One more question.
  Should we push this into Shared-1.0.0-M1 or wait for the next iteration?
  Too late, vote already started. But maybe we need to rollback ;-)

  No, let's push all the move to the M2 branch (which has to be renamed, btw, 
it's M1 atm)

  We will have lots of modifications in the API in M2 anyway, so don't worry 
too much. M1 != API freeze !

How about naming this branch 1.0 and that way we don't have to keep changing 
the name with every RC or milestone release we have?

Yeah, that sounds a good "stable" naming for this branch.
+1


+1 too.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: SchemaManager usage

[Emmanuel Lécharny]

On 2/15/11 12:41 AM, Stefan Seelmann wrote:

On Tue, Feb 15, 2011 at 12:10 AM, Emmanuel Lecharny  wrote:

Hi guys,

today, I continued to document the Dn class. I did it on the confluence
wiki, as I have a tool to export a confluence docuemnt to a docBook format,
which is way more convenient than typing text in XML...

Which tool?


Wikbook. It takes a wiki format and generates docbook formt. Still have 
to test it, but I know the guy working on it.


http://code.google.com/p/wikbook/

Here is the current doco about Dn :
https://cwiki.apache.org/confluence/display/DIRAPI/DN+API

Feel free to review, and fix it.

Great, thanks.

One first comment wrt to the code samples. Today I had the idea to
create some code samples on how to use the API in the
documentation/samples directory (I didn't start yet). One idea I had
that it would be cool if the samples project could be added to the
"trunks" aggregator so that changes in trunk are directly reflected in
the  samples. The 2nd idea was that it would be cooler if the examples
in the doco could be directly extracted from the samples code. The big
problem with code snippets in doco is that it isn't updated when
refactorings in the main code are done. Maybe Xinclude
(http://www.w3.org/TR/xinclude/) helps here.


I have created a project where I put all the code samples I have added 
in the wiki (just to be sure they are not wrong).


I think we can add the code in the apacheds-manual project.

So, yes, good idea !

Ok, that being said, this is a very interesting exercise, as it forces you
to use the API and to think like a user. And what we have is not really
perfect :/

I'm currently trying to leverage the SchemaManager in Dn, which somehow
forced me to create a DefaultSchemaManager INSTANCE static field, which
loads the default schemas (those that are enabled by default). Now, creating
a schema aware DN is just a matter of calling :

Dn dn = new Dn( DefaultSchemaManager.INSTANCE, "dc=example, dc=com");

Works great. Sadly, I had to move the Dn tests out of the shared-ldap-model
otherwise there would be no way to use the SchemaManager instance, as it's
declared in another module. So I moved it to shared-integ (all in all, I
think it's the right place for tests anyway).

I'm now going a step further and move some other tests that requires a
SchemaManager to shared-integ.

I agree that if classes from different modules are used in test then
the right location is the shared-integ module.

Otherwise, if the test should be kept in the model module a
SchemaManager mock is the way to go.
We can discuss that. Frankly, right now, I'm more into checking the Dn 
API and writing the doco than thinking about the right place for tests, 
so I took the path of least resistance. But, basically, you are 100% 
right : we may have to define a MockSchemaManager.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/16/11 9:02 PM, Alex Karasulu wrote:

On Wed, Feb 16, 2011 at 4:50 PM, Emmanuel Lecharny  wrote:

Hi,

we have had some convo about the Dn methods last night. Here are some of the 
things we discussed and came with :

o the getPrefix(N)/getSuffix(N) are cumbersome, and not easy to manipulate. The 
main issue is that they depend on the RDN order, which is the opposite as what 
people are used to manipulate. Everytime you have to get a prefix from a Dn, 
you always wonder what the position should be, and if it's 0 based or 1 based...

We propose to replace those methods by getParent(Dn) and getDescendant(Dn). Let 
me give you an example :

// A DN
Dn dn = new Dn( "cn=test,ou=server,ou=directory,dc=apache,dc=org" );

// Get the right part (equivalent to getprefix( 2 ) )
Dn parent = dn.getParent( "cn=test,ou=server,ou=directory" ); // returns 
"dc=apache,dc=org"

// Get the left part (equivalent to getSuffix( 3 ))
Dn descendant = dn.getDescendant( "ou=directory,dc=apache,dc=org" ); // returns 
"cn=test,ou=server"

o The Add method is a bit annoying to remove, because first, the JNDI Name 
interface has such a method, and people are used to it, second removing it 
means we have to add some more constructors line Dn( Dn, Rdn... ). I agree that 
someone doing something like :

Dn dn = new Dn( "dc=apache,dc=org" );
dn.add( "ou=directory" );

will expect that the dn is now "ou=directory,dc=apache,dc=org", when it's 
unchanged.

This is really troublesome... What about rename it concatenate() ?

Thoughts ?

Sounds good. But how about this:

// not showing full Rdn but an index value representing the actual
rdns in the dn for pos clarity
Dn dn = new Dn( “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” );

dn.getAncestorDn( “9, 8, 7, 6” );

=>  “5, 4, 3, 2, 1, 0”


That's ok, but why not getParent() ? getAncestorDn() is lng. I'd 
rather use getAncestor(), but I find it more user friendly to use 
getParent()

dn.getAncestorDn( 6 );

=>  “5, 4, 3, 2, 1, 0”

Does not work. What if the initial dn is "9, 8, 7, 6, 5, 4, 6, 3, 2, 2, 0" ?

dn.getDnToAncestor( “1, 0” );


Again, the name is frankly difficult to understand, and also confusing 
(getDnToAncestor ? getAncestorDn ? Which one does what ?).


dn.getDescendant() is easier to understand, even if not perfect.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/16/11 9:29 PM, Alex Karasulu wrote:

On Wed, Feb 16, 2011 at 10:02 PM, Alex Karasulu  wrote:

On Wed, Feb 16, 2011 at 4:50 PM, Emmanuel Lecharny  wrote:

Hi,

we have had some convo about the Dn methods last night. Here are some of the 
things we discussed and came with :

o the getPrefix(N)/getSuffix(N) are cumbersome, and not easy to manipulate. The 
main issue is that they depend on the RDN order, which is the opposite as what 
people are used to manipulate. Everytime you have to get a prefix from a Dn, 
you always wonder what the position should be, and if it's 0 based or 1 based...

We propose to replace those methods by getParent(Dn) and getDescendant(Dn). Let 
me give you an example :

// A DN
Dn dn = new Dn( "cn=test,ou=server,ou=directory,dc=apache,dc=org" );

// Get the right part (equivalent to getprefix( 2 ) )
Dn parent = dn.getParent( "cn=test,ou=server,ou=directory" ); // returns 
"dc=apache,dc=org"

// Get the left part (equivalent to getSuffix( 3 ))
Dn descendant = dn.getDescendant( "ou=directory,dc=apache,dc=org" ); // returns 
"cn=test,ou=server"

o The Add method is a bit annoying to remove, because first, the JNDI Name 
interface has such a method, and people are used to it, second removing it 
means we have to add some more constructors line Dn( Dn, Rdn... ). I agree that 
someone doing something like :

Dn dn = new Dn( "dc=apache,dc=org" );
dn.add( "ou=directory" );

will expect that the dn is now "ou=directory,dc=apache,dc=org", when it's 
unchanged.

This is really troublesome... What about rename it concatenate() ?

Thoughts ?

Sounds good. But how about this:

// not showing full Rdn but an index value representing the actual
rdns in the dn for pos clarity
Dn dn = new Dn( “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” );

dn.getAncestorDn( “9, 8, 7, 6” );

=>  “5, 4, 3, 2, 1, 0”

dn.getAncestorDn( 6 );

=>  “5, 4, 3, 2, 1, 0”

dn.getDnToAncestor( “1, 0” );

=>  “9, 8, 7, 6, 5, 4, 3, 2”

dn.getDnToAncestor( 2 );

=>  “9, 8, 7, 6, 5, 4, 3, 2”

Dn namingContext = “1, 0”;

// all same below
Dn dnAfterNamingContext = dn.getDnToAncestor( "1, 0" );
Dn dnAfterNamingContext = dn.getDnToAncestor( namingContext );
Dn dnAfterNamingContext = dn.getDnToAncestor( namingContext.size() );
Dn dnAfterNamingContext = dn.getDnToAncestor( namingContext.toString() );

dnAfterNamingContext ===>  “9, 8, 7, 6, 5, 4, 3, 2”

Overloads should be provided for Dn, String and int.


Maybe some convenience methods like:

dn.isAncestor( namingContext );
namingContext.isDescendant( dn );
We may add such methods, sure. Again, I'd rather use isParent( Dn ). 
IsDescendant( Dn ) fits me.



dn.isRelated( namingContext ); =>  i.e. two DNs in different
namingContexts will not be related
dn.hasParent( Dn ) sounds again simpler. Every time you peek a different 
name for the same element (ancestor, related, ...), you confuse the user.

namingContext.isParent( dn );
dn.isChild( namingContext );
isChild is not good, as the child is supposed to be the RDN only. 
isDescendant() and hasDescendant() is ok IMO, unless we decide that 
child fits better (dn.getChild( Dn ), dn.isChild( dn ), dn.hasChild( dn ))


I'd like to get the API as simple as possible, avoiding using different 
terminology to describe the same cocept, plus to keep it simple to 
remember (and here, xxxdescendant() is problematic, because it's not 
easy to type without doing a mistake. I'd rather use 'child', if we 
don't mind about the Rdn relation)


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/17/11 12:03 AM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 12:55 AM, Emmanuel Lécharny
  wrote:

On 2/16/11 9:02 PM, Alex Karasulu wrote:

On Wed, Feb 16, 2011 at 4:50 PM, Emmanuel Lecharny
  wrote:

Hi,

we have had some convo about the Dn methods last night. Here are some of
the things we discussed and came with :

o the getPrefix(N)/getSuffix(N) are cumbersome, and not easy to
manipulate. The main issue is that they depend on the RDN order, which is
the opposite as what people are used to manipulate. Everytime you have to
get a prefix from a Dn, you always wonder what the position should be, and
if it's 0 based or 1 based...

We propose to replace those methods by getParent(Dn) and
getDescendant(Dn). Let me give you an example :

// A DN
Dn dn = new Dn( "cn=test,ou=server,ou=directory,dc=apache,dc=org" );

// Get the right part (equivalent to getprefix( 2 ) )
Dn parent = dn.getParent( "cn=test,ou=server,ou=directory" ); // returns
"dc=apache,dc=org"

// Get the left part (equivalent to getSuffix( 3 ))
Dn descendant = dn.getDescendant( "ou=directory,dc=apache,dc=org" ); //
returns "cn=test,ou=server"

o The Add method is a bit annoying to remove, because first, the JNDI
Name interface has such a method, and people are used to it, second removing
it means we have to add some more constructors line Dn( Dn, Rdn... ). I
agree that someone doing something like :

Dn dn = new Dn( "dc=apache,dc=org" );
dn.add( "ou=directory" );

will expect that the dn is now "ou=directory,dc=apache,dc=org", when it's
unchanged.

This is really troublesome... What about rename it concatenate() ?

Thoughts ?

Sounds good. But how about this:

// not showing full Rdn but an index value representing the actual
rdns in the dn for pos clarity
Dn dn = new Dn( “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” );

dn.getAncestorDn( “9, 8, 7, 6” );

=>“5, 4, 3, 2, 1, 0”

That's ok, but why not getParent() ?

Because the result is not necessarily the parent of 'dn'. It may be if
it's immediately superior. You're mixing together the meanings I
think.


Sorry, I don't get it. You have a DN, and you want the parent of it's 
left part, how possible the result couldn't be the parent ? If the 
parameter is not the left part, then you get an exception, but that's in 
the contract.


Did I missed something ?

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/17/11 12:43 AM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 1:20 AM, Emmanuel Lécharny  wrote:

On 2/17/11 12:03 AM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 12:55 AM, Emmanuel Lécharny
wrote:

On 2/16/11 9:02 PM, Alex Karasulu wrote:

On Wed, Feb 16, 2011 at 4:50 PM, Emmanuel Lecharny
  wrote:

Hi,

we have had some convo about the Dn methods last night. Here are some
of
the things we discussed and came with :

o the getPrefix(N)/getSuffix(N) are cumbersome, and not easy to
manipulate. The main issue is that they depend on the RDN order, which
is
the opposite as what people are used to manipulate. Everytime you have
to
get a prefix from a Dn, you always wonder what the position should be,
and
if it's 0 based or 1 based...

We propose to replace those methods by getParent(Dn) and
getDescendant(Dn). Let me give you an example :

// A DN
Dn dn = new Dn( "cn=test,ou=server,ou=directory,dc=apache,dc=org" );

// Get the right part (equivalent to getprefix( 2 ) )
Dn parent = dn.getParent( "cn=test,ou=server,ou=directory" ); //
returns
"dc=apache,dc=org"

// Get the left part (equivalent to getSuffix( 3 ))
Dn descendant = dn.getDescendant( "ou=directory,dc=apache,dc=org" ); //
returns "cn=test,ou=server"

o The Add method is a bit annoying to remove, because first, the JNDI
Name interface has such a method, and people are used to it, second
removing
it means we have to add some more constructors line Dn( Dn, Rdn... ). I
agree that someone doing something like :

Dn dn = new Dn( "dc=apache,dc=org" );
dn.add( "ou=directory" );

will expect that the dn is now "ou=directory,dc=apache,dc=org", when
it's
unchanged.

This is really troublesome... What about rename it concatenate() ?

Thoughts ?

Sounds good. But how about this:

// not showing full Rdn but an index value representing the actual
rdns in the dn for pos clarity
Dn dn = new Dn( “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” );

dn.getAncestorDn( “9, 8, 7, 6” );

=>  “5, 4, 3, 2, 1, 0”

That's ok, but why not getParent() ?

Because the result is not necessarily the parent of 'dn'. It may be if
it's immediately superior. You're mixing together the meanings I
think.

Sorry, I don't get it. You have a DN, and you want the parent of it's left
part, how possible the result couldn't be the parent ? If the parameter is
not the left part, then you get an exception, but that's in the contract.

Did I missed something ?

The dn is “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” you're getting one of it's
ancestors not it's parent.

The parent would be “8, 7, 6, 5, 4, 3, 2, 1, 0”, the ancestor in the
example is “5, 4, 3, 2, 1, 0”.

You see what I mean?

yes, but in the sample I gave it was :

“9, 8, 7, 6, 5, 4, 3, 2, 1, 0”.getParent( "9, 8, 7, 6" ) which is "5, 4, 3, 2, 
1, 0”


The idea is that the parameter is the left part of the DN, and the 
getParent returns the right part.


OTOH,

“9, 8, 7, 6, 5, 4, 3, 2, 1, 0”.getParent() would return  "8, 7, 6, 5, 4, 3, 2, 
1, 0”

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/17/11 12:43 AM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 1:20 AM, Emmanuel Lécharny  wrote:

On 2/17/11 12:03 AM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 12:55 AM, Emmanuel Lécharny
wrote:

On 2/16/11 9:02 PM, Alex Karasulu wrote:

On Wed, Feb 16, 2011 at 4:50 PM, Emmanuel Lecharny
  wrote:

Hi,

we have had some convo about the Dn methods last night. Here are some
of
the things we discussed and came with :

o the getPrefix(N)/getSuffix(N) are cumbersome, and not easy to
manipulate. The main issue is that they depend on the RDN order, which
is
the opposite as what people are used to manipulate. Everytime you have
to
get a prefix from a Dn, you always wonder what the position should be,
and
if it's 0 based or 1 based...

We propose to replace those methods by getParent(Dn) and
getDescendant(Dn). Let me give you an example :

// A DN
Dn dn = new Dn( "cn=test,ou=server,ou=directory,dc=apache,dc=org" );

// Get the right part (equivalent to getprefix( 2 ) )
Dn parent = dn.getParent( "cn=test,ou=server,ou=directory" ); //
returns
"dc=apache,dc=org"

// Get the left part (equivalent to getSuffix( 3 ))
Dn descendant = dn.getDescendant( "ou=directory,dc=apache,dc=org" ); //
returns "cn=test,ou=server"

o The Add method is a bit annoying to remove, because first, the JNDI
Name interface has such a method, and people are used to it, second
removing
it means we have to add some more constructors line Dn( Dn, Rdn... ). I
agree that someone doing something like :

Dn dn = new Dn( "dc=apache,dc=org" );
dn.add( "ou=directory" );

will expect that the dn is now "ou=directory,dc=apache,dc=org", when
it's
unchanged.

This is really troublesome... What about rename it concatenate() ?

Thoughts ?

Sounds good. But how about this:

// not showing full Rdn but an index value representing the actual
rdns in the dn for pos clarity
Dn dn = new Dn( “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” );

dn.getAncestorDn( “9, 8, 7, 6” );

=>  “5, 4, 3, 2, 1, 0”

That's ok, but why not getParent() ?

Because the result is not necessarily the parent of 'dn'. It may be if
it's immediately superior. You're mixing together the meanings I
think.

Sorry, I don't get it. You have a DN, and you want the parent of it's left
part, how possible the result couldn't be the parent ? If the parameter is
not the left part, then you get an exception, but that's in the contract.

Did I missed something ?

The dn is “9, 8, 7, 6, 5, 4, 3, 2, 1, 0” you're getting one of it's
ancestors not it's parent.

The parent would be “8, 7, 6, 5, 4, 3, 2, 1, 0”, the ancestor in the
example is “5, 4, 3, 2, 1, 0”.

You see what I mean?
Alex

Bottom line, it does not really matter... What is important is that we 
agree on a name that express what the operation does.


So, considering that a DN can be expressed either as [RDN][DNp] or 
[DNd][DNa] with DNp = parent, DNa = ancestor, DNd = descendant (DNp and 
DNa are equivalent, it's just that a DNa may have a DNd, when a DNp can 
only ave a RDN) :

* Dn.getParent() -> returns the DNp in [RDN][DNp]
* Dn.getAncestor( DNd ) -> returns the DNa in [DNd][DNa]
* Dn.getDescendant( DNa ) -> returns the DNd in [DNd][DNa]

Is that ok ? Should we use something more clear, like 
getAncestorOf/getDescendantOf ? Or use getAscendant/getDescendant ?


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/17/11 12:07 PM, Alex Karasulu wrote:


getAncestorOf/getDescendantOf ? Or use getAscendant/getDescendant ?
Yeah getAncestorOf/getDescendantOf sounds like it flows better and
clarifies that we're taking this from the dn the operation is applied
to.

I like it better too. Will rename to use those names.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/17/11 1:27 PM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 2:19 PM, Emmanuel Lécharny  wrote:

On 2/17/11 12:07 PM, Alex Karasulu wrote:

getAncestorOf/getDescendantOf ? Or use getAscendant/getDescendant ?
Yeah getAncestorOf/getDescendantOf sounds like it flows better and
clarifies that we're taking this from the dn the operation is applied
to.

I like it better too. Will rename to use those names.

Well hold on a second with the getDescendantOf operation. I think we
have some outstanding issues with it. Please see my previous post.


I'm not closing the door here. Here is what I suggest : I'll move the 
API to use those name *for the moment*, in order to get rid of the old 
getPrefix/getSuffix names, but we can discuss those names further until 
the RC1. If we find a better name, I'll change them, it's a 5 mins task. 
I'm *not* considering that as  final move, again.


What is important is really to reach a point we all more or less agree, 
which is not easy, as it's not a technical matter, but much more a mix 
of many concerns :

- semantic
- ease of use
- language (correct and accurate english)

We should not expect to get the API defintion to be a breeze, I'm not 
really surprised that such things raise discussions like the one we are 
having. Frankly, if we were in a room, all of us, it would be exactly 
the same thing with people thinking this, other thinking that, etc, but 
nobody would hear about those disagreements outside the room. We are 
doing that in the open, and even if we don't agree on everything, at 
then end the result will be the same : we will reach consensus, and we 
will get something better than any decision made by one single person.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

Heads up. Here are some suggested changes on the Dn API. The existing methods 
have a (e) prefix. The static methods have a (s) prefix. The added methods have 
a (+) prefix. The deleted methods have a (-) prefix. The renamed methods have a 
(r) prefix.

Constructors :

(e)Dn()

(e)Dn(SchemaManager)

(e)Dn(Rdn...)

(e)Dn(SchemaManager, Rdn...)

(-)Dn(String) not needed, the Dn(String...) can be used instead

(-)Dn(SchemaManager, String)

(e)Dn(String...)

(e)Dn(SchemaManager, String...)

(a)Dn(SchemaManager, Dn)

(a)Dn(Dn, Rdn) ->  to create a new DN with a new RDN

(-)add(String)

(-)add(String…)

(-)add(Rdn)

(-)add(int, Rdn)

(-)add(int, String)

(-)addAll(Dn)

(-)addAll(int, Dn)

(-)addAllNormalized(int, Dn)

(all replaced by constructor calls)

(es)   isNullOrEmpty(Dn)

(es)   isValid(String)

(er)   isChildOf(Dn) ->  isDescendantOf(Dn)

(er)   isChildOf(String) ->  isDescendantOf(String)

(e)isEmpty()

(-)isNormalized() ->  delete

(er)   isParentOf(Dn) ->  isAncestorOf(Dn)

(er)   isParentOf(String) ->  isAncestorOf(String)

(e)isRootDSE()

(?s)   getBytes(Dn) ->  package protected/delete

(?s)   getNbBytes(Dn) ->  package protected/delete

(-)get(int) ->  delete

(e)getNormName()

(e)getParent()

(er)   getParent(Dn) ->  getAncestorOf(Dn)

(er)   getParent(String) ->  getAncestorOf(String)

(e)getRdn()

(-)getRdn(int)

(e)getRdns()

(er)   getSuffix(int) ->  getDescendantOf(Dn)/getDescendantOf(String)

(+)getSchemaManager() ->  added

(-)normalize(Dn, Map) ->  delete

(-)normalize(SchemaManager, String) ->  delete

(-)normalize(SchemaManager) ->  delete

(-)hasSuffix(Dn) ->  delete

(-)remove(int) ->  delete

(e)size()

(e)equals(Object)

(e)hashCode()

(e)iterator()

(e)toString()


I won't change any of those methods unless we agreed on the operations, 
because it impacts the server and studio deeply, so I want to be sure we 
don't have to do that many times. Although as soon as we agree on some 
of them, I'll apply the agreed on modification in code.


Thanks !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some more thoughts about the Dn class

[Emmanuel Lécharny]

On 2/17/11 6:53 PM, Alex Karasulu wrote:

On Thu, Feb 17, 2011 at 5:10 PM, Emmanuel Lécharny  wrote:

Heads up. Here are some suggested changes on the Dn API. The existing
methods have a (e) prefix. The static methods have a (s) prefix. The added
methods have a (+) prefix. The deleted methods have a (-) prefix. The
renamed methods have a (r) prefix.

Constructors :

(e)Dn()

(e)Dn(SchemaManager)

(e)Dn(Rdn...)

(e)Dn(SchemaManager, Rdn...)

(-)Dn(String) not needed, the Dn(String...) can be used instead

(-)Dn(SchemaManager, String)

(e)Dn(String...)

(e)Dn(SchemaManager, String...)

(a)Dn(SchemaManager, Dn)

(a)Dn(Dn, Rdn) ->to create a new DN with a new RDN

(-)add(String)

(-)add(String…)

(-)add(Rdn)

(-)add(int, Rdn)

(-)add(int, String)

(-)addAll(Dn)

(-)addAll(int, Dn)

(-)addAllNormalized(int, Dn)

(all replaced by constructor calls)

(es)   isNullOrEmpty(Dn)

(es)   isValid(String)

(er)   isChildOf(Dn) ->isDescendantOf(Dn)

(er)   isChildOf(String) ->isDescendantOf(String)

(e)isEmpty()

(-)isNormalized() ->delete

(er)   isParentOf(Dn) ->isAncestorOf(Dn)

(er)   isParentOf(String) ->isAncestorOf(String)

(e)isRootDSE()

(?s)   getBytes(Dn) ->package protected/delete

(?s)   getNbBytes(Dn) ->package protected/delete

(-)get(int) ->delete

(e)getNormName()

(e)getParent()

(er)   getParent(Dn) ->getAncestorOf(Dn)

(er)   getParent(String) ->getAncestorOf(String)

(e)getRdn()

(-)getRdn(int)

(e)getRdns()

(er)   getSuffix(int) ->getDescendantOf(Dn)/getDescendantOf(String)

(+)getSchemaManager() ->added

(-)normalize(Dn, Map) ->delete

(-)normalize(SchemaManager, String) ->delete

(-)normalize(SchemaManager) ->delete

(-)hasSuffix(Dn) ->delete

(-)remove(int) ->delete

(e)size()

(e)equals(Object)

(e)hashCode()

(e)iterator()

(e)toString()


I won't change any of those methods unless we agreed on the operations,
because it impacts the server and studio deeply, so I want to be sure we
don't have to do that many times. Although as soon as we agree on some of
them, I'll apply the agreed on modification in code.

OK go ahead.


Note that some modifications are *very expensive* : for instance, 
removing the Dn.normalize( SchemaManager ) has a huge impact on the server.


Atm, i'm trying to see where inside the server we use not normalized Dn, 
and to remove them.


If any of you have some issue with any of the actions to be applied to 
the Dn methods, feel free to speak your mind : I won't apply the 
modifications you don't like, until we reach a consensus.


Thanks !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: DN serialization issues

[Emmanuel Lécharny]

On 2/19/11 12:24 AM, Alex Karasulu wrote:

On Fri, Feb 18, 2011 at 8:17 PM, Emmanuel Lecharnywrote:


Hi guys,

once upon a time, in order to improve the server permformance, the DN was
serialized with its normalized value. This was to avoid a costly
normalization when we pull back the DN from the backend. But now that we
don't store anymore the DN into the entry, it does not make any sense to
store the normalized DN.

We should get rid of that.



+1




Another problem is that the LogManager also serialize entries with a full
DN on the disk, and when it reads them back, it should be able to inject the
schemaManager into the DN, which is not done.

All in all, the DnSerializer has nothing to do in the Ldap-API. It's a
Server thingy, as is the EntrySerializer. Moreover, the DefaultEntry
implements the Externalizable interface, but does also provides
serialize()/deserialize() methods, which export the RDN, not the DN, simply
because it's used only by the server.

This is a mess...


Separating out the implementations might help decouple these server side
ONLY concerns. But I don't know how I feel about having to have a ServerDn
verses a regular Dn.


This is what we had once upon a time, and it was, well, awful.
What I'm trying to do right now is to make all the components schema 
aware (ie, DN RDN, AVA) and just store the minimum information when 
serializing them (ie, just serialize the UP types and values), assuming 
that the normalized values can be deduced using the SM, if it's injected.


That should help on the server side, because the SM will always be 
available on the server. Also the burden (and cost) of normalizing a 
serialized value on the server (something we have to deal with in three 
places only : the backend, the session and the ChangeLog) will not be 
that high, assuming that we will have an entry cache later, that  the 
changeLog can be desactivated and that we anyway have to do the job on 
the session side.


This should work, but it's quite painful as it implies a *lot* of 
modifications in the AVA and Rdn classes.


We also have to deal with the different serializer we have, the static 
ones we use in the server, which must take a SM as a parameter.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Problem when running server-integ tests

[Emmanuel Lécharny]

On 2/21/11 11:37 AM, Alex Karasulu wrote:

On Mon, Feb 21, 2011 at 11:24 AM, Emmanuel Lecharnywrote:


Hi guys,

this week-end, I experienced some bad issues with the server-integ tests,
while trying to use the same LdapDecoder for both client and server. After 3
or 4 tests, I get blocked for 30 seconds, and then move to the next test,
get blocked again, etc.

Of course, at the end, I got test failures, as expected.

I did some deep analysis on what was going on, added some traces, and that
led me to a point where I get blocked while trying to access to any of the
LdapContainer method.

Here is an exemple of the plcae I get blocked in :

public void decode( ByteBuffer stream, Asn1Container container ) throws
DecoderException
{
boolean hasRemaining = stream.hasRemaining();
System.out.println( "  : " + hasRemaining + "/" +
stream.remaining() );

// Increment the PDU size counter.
int remaining = stream.remaining();
System.out.println( "- 0" );
container.incrementDecodeBytes( remaining );

System.out.println( "- 1" );

and in the console, the "- 0" get printed, not the "- 1". The
incrementDecodeBytes is a very simple method :

public void incrementDecodeBytes( int nb )
{
System.out.println( "---in incrementDecodedBytes" );
decodeBytes += nb;
}

(note that I don't get either the "---in incrementDecodedBytes" message).
It's not synchronized, not the decodeBytes is an AtomicInteger.

I did a threadDump, which is available here : http://pastebin.com/wPNPL1FY

It shows that 15 threads are blocked on monitors :

pool-24-thread-1Task5   513269765680140288
  waiting on condition [15290]
pool-22-thread-1Task5   506122245684236288
  waiting on condition [152ce8000]
pool-14-thread-4Task5   568627205715816448
  waiting on condition [154b06000]
net.sf.ehcache.CacheManager@6f7cf6b6Daemon  5   27879424
  5657616384 in Object.wait() [151385000]
mainTask5   503336964300214272  waiting on
condition[1004fe000]
kdcReplayCache.data Daemon  5   507740165658677248
  waiting on condition[151488000]
groupCache.data Daemon  5   507781125660798976
  waiting on condition [15168e000]
changePwdReplayCache.data   Daemon  5   278814725659738112
  waiting on condition[15158b000]
SynchWorkerThread   Task5   352583685652594688  in
Object.wait() [150ebb000]
Surefire fork timeout timer Task5   269885445647831040
  in Object.wait()[150a3]
Reference Handler   Daemon  10  503623685634641920  in
Object.wait() [14fd9c000]
Finalizer   Daemon  8   503644165635702784  in
Object.wait()[14fe9f000]
FelixStartLevel Daemon  5   352931845670658048  in
Object.wait() [151ff5000]
FelixPackageAdmin   Daemon  5   335175685655359488  in
Object.wait() [15115e000]
FelixDispatchQueue  Task5   286187525661859840  in
Object.wait() [151791000]


I have *no idea* why the test get blocked, I see no rational reason for it
to be blocked. I'll add the thread-id n the logs to see which thread it is
running on...

Can felix be an issue here ?


OK let's just list what we know:

(1) These problems were not present until we experimented with changes to
the codec.
(2) A small part of codec code may be running inside Felix, but only if
we're hitting some of these extra controls. The codec core is not inside
Felix.

It's probably unlikely that Felix is the culprit but who knows: are there
specific tests that are showing this behavior or is it all of them? This
behavior is weird so we should suspect everything. Can you post the stack
traces from the thread dump?


I have some reproductible scenario, I'll commit it n a branch. I'm not 
sure at all that Felix is the culprit, but even if it's innocent, as the 
new kid in the block, throwing stones to it seems pretty natural to me :)


This is *very* weird.

It seems that there is some contention on the container instance, which 
is simply strange.


More to come after lunch.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Problem when running server-integ tests

[Emmanuel Lécharny]

Ok, Felix is not the culrpit, definitively...

I don't know why, but I got some illumination. If the following code :

   int remaining = stream.remaining();
   System.out.println( "- 0" );
   container.incrementDecodeBytes( remaining );

   System.out.println( "- 1" );

does not print the "- 1" message, it's either because we are blocked into 
the container.incrementDecodeBytes() method. That was my first assumption, but after 
having add some print in this method, I saw that we don't enter into it.

I tried a few other modifications, wasted 2 hours around this, and then, flash 
in my brain, what if 'container' is null ? You'd expect a NPE to be reported, 
yes? Yes. And we *have* a NPE, but it's silently swallowed by the clien 
LdapNetworkConnection.execptionCaught() method :

public void exceptionCaught( IoSession session, Throwable cause ) throws 
Exception
{
LOG.warn( cause.getMessage(), cause );

if ( cause instanceof ProtocolEncoderException )
{
Throwable realCause = ( ( ProtocolEncoderException ) cause 
).getCause();

if ( realCause instanceof MessageEncoderException )
{
int messageId = ( ( MessageEncoderException ) realCause 
).getMessageId();

ResponseFuture  response = futureMap.get( messageId );
response.cancel( true );
response.setCause( realCause );
}
}
}

So now, it's totally rational, it's not a Felix bug, it's not even a JVM bug, 
it's a bad bug in *our* code.

Next step : why is this container null ???

/me digging...



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Rdn and Ava cleanup

[Emmanuel Lécharny]

On 2/24/11 3:01 AM, Alex Karasulu wrote:

On Wed, Feb 23, 2011 at 12:14 PM, Emmanuel Lecharny  wrote:

Hi guys,

those last 2 days I did some cleanup in the Rdn and Ava classes, in th
espirit of Dn cleanup. Here is a summary of what has been done, roughly :

- injection of the SchemaManager in both classes' constructors
- removing of the compareTo method
- made some methods private
- made the classes final
- removed the Comparable interface from the equation

One important modification is the last one : it makes no sense to have a Rdn
be comparable. First, how do we compare a cn and a jpegPhoto ? Second, how
do we compare two RDN which attributeType does not have an equality matching
rule ?

Of course, this has an impact in the way the backend works, as the Rdn index
needs to be able to do ordered comparison between Rdn, as this index is a
BTree. What I did is that I replaced the Rdn.compareTo(Rdn) method by a
direct String comparison in Jdbm between the Rdn's normalized name. That
does work.

It made me think that maybe using a hashed index for Rdn is probably a
better idea, because then we won't need this comparison to be done (the
equals method would be enough) and also because it would be faster (finding
an element in a Hash table is an O(1) operation - at least, theorically -
when looking in a BTree is an O(log2(N)))

thoughts on this last point?

I just replied to Stefan's post first but I thought about this after
my reply. Here's what came to mind:

(1) Let's presume we back the one and sublevel indices with the RDN index.
(2) Do we need to sort either of these three indices when they are
coupled this way or all independent?

We don't need to really sort them, we just need to be able to walk the
index producing candidates for one level and sublevel relationships.
We could probably still do this and get away without using a sorted
B+Tree.

The thing is we cannot apply certain assertions any longer with a hash
like using the>=.<=, and substring assertions.
DN syntax and the ATs that use it does not provide a Substring or 
Ordering MR. Does not make a lot of sense, IMO (comparing DN for 
equality is ok, trying to order them is just a vain thing)



But this is not
something performed on these indices anyway. You do these with respect
to an attribute type which that might be indexed and do not use these
operators on one level, or sub level inquiries.

So yeah I think this can be done and may have big performance
benefits. But it's a big shift.
This is an interesting side effect of my question :) I haven't expected 
such a wide modification proposal when I asked about using Hash instead 
of BTree, but this is were it's start to become interesting ! 
Definitively to be investigated, and hopefully Stefan has already 
started to dig in this direction with his Hashbase implementation.

Fun project too.

Sure is !


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Serialization/deserialization questions

[Emmanuel Lécharny]

On 2/25/11 2:57 PM, Alex Karasulu wrote:

On Fri, Feb 25, 2011 at 2:22 PM, Emmanuel Lecharny  wrote:

Hi guys,

I'm reworking all the serializations for the base objects we are
manipulating. I have a few questions and I'd like to get you opinion.

The base idea is to get rid of all the Externalizable/Serializable
interfaces, and to provide our own helper classes to serialize the objects
we have to write down the disk. The rationnal is that we avoid the cost of
writing the useless java class informations, keeping the written objects as
small as possible, and the process as fast as possible.

Could you provide some of the problems that you encountered due to the
use of the Externalizable/Serializable intefaces in some of these
classes?


The main issue is the injection of the SchemaManager when we create the 
base object. readExternal() does not take any parameter, when using a 
helper class to do the same thing allows you to do so.

Q1 : Do we have to store a version number into the written data ?

+1


Q2 : I'm going to inject the SchemaManager in all the deserialized data (it
can be null, if the SchemaManager has not been created). Do you think it's a
problem ?

We can have separate serialization services that have a handle on the
SchemaManager. These services can use schema information to handle
their task of [de]serializing. However there are some entities that
need access to schema information in order to conduct some operations
like Entry.add( attribute ). How we handle access to schema
information while they perform schema checks is another matter which
we need to discuss.

Perhaps a schema aware Entry wrapper can access this information
leaving the simple Entry free to do as it likes while it intercepts
calls and conducts schema live checks before delegating the call.
Things like this can also be handled automatically with byte code
manipulation. We can discuss this separately which would be involved.
What we need here is mainly to store the schemaManager, not to process 
the read data against the schemamanager. Doing so is a costly operation, 
and if we can avoid it, this is a net gain.


Also keep in mind that the serializations are mainly used inside the 
server, not anywhere else.

Q3 : In order to be able to create the object as it was when it was
serialized, I'll create package visible only constructors, with all the
needed parameters. Does it sounds the right approach ?

This will require the [de]-serializer to be in the same package. That
could present an issue with split packages if we want to make these
entities into services down the line. I'd go ahead and make the
constructors public for now.  Off the top of my head I cannot think of
a detriment to exposing these constructors but perhaps you have some
in mind?
Well, making those constructors public make the user think he can use 
them, when it's just intended to be used by the deserialization process. 
I don't think, OTOH, that having the deserializers along the class it 
works on is a big problem.


One other option would be to use a static method in the class to do the 
serialization/deserialization, and make the constructors private.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [Shared] [SCM] Thinking about some hierarchy in shared.

[Emmanuel Lécharny]

On 2/28/11 1:02 PM, Alex Karasulu wrote:

Hi Emmanuel,

On Mon, Feb 28, 2011 at 1:27 PM, Emmanuel Lecharny  wrote:

 ldap/
 codec/
 model/
 schema/
 schema-converter/

wouldn't it be better to make it a sub-module ?

Yes these are all sub modules.

I mean, schema-convertor as a sub-model under schema/

 client-api/
 codec-standalone/
 all/

Shouldn't it be a separated module ?

Hmm I don't quite understand you here. These are all separate modules
broken down for better hierarchical organization. Could you be a bit
more specific, I want to make sure I understand you fully.

The 'all' module shoudl'nt it be moved at the top level ?

What is this module about ?



The archetype module? It contains some Maven archetypes for rapidly
setting up new extension oriented maven projects. Like for example an
archetype to start a new control project or a new ext req/resp pair.

Ahh, ok. Call it maven-archetype, then, it would be more explicit.



Seems ok to me, more or less, but I think we should reorganize after M2.

I don't see a benefit before or after M2. Just curious though why you
want to do this after M2? If you have a preference we can cater to
that.
Not a big deal, trying to connect the two mails you sent, was thinking 
you wanted to get M2 out now.


Sorry for the short answers, I'm a bit busy dealing with real life atm.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Serializer heads-up

[Emmanuel Lécharny]

On 3/2/11 4:21 PM, Alex Karasulu wrote:

Just a wild thought. Really isn't the codec doing serialization too?

No. The codec processes Messages, not base elements as DNs, RDNs or AVAs.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Serializers : done

[Emmanuel Lécharny]

On 3/5/11 11:00 AM, Stefan Seelmann wrote:

If anyone wants to launch a vote for 1.0.0-M2, please do so.

Do you ping me?

I'll first check why builds on Hudon and Continuum fail.

It seems like a problem with Jenkins itself, not with the build.

I'll be around today, if you need me.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [Shared] [Model] What was the reason for removing the schema entity interfaces like AttributeType?

[Emmanuel Lécharny]

On 3/8/11 12:38 AM, Alex Karasulu wrote:

On Tue, Mar 8, 2011 at 1:22 AM, Emmanuel Lecharny  wrote:

On 3/8/11 12:09 AM, Alex Karasulu wrote:

I'm sitting here starring at this nest and wondering what
the heck I should do first.

Wait for M2 to be out ... :)

Why?

First I am in a personal branch and second I merged that last time to
prep for M2 so nothing will be merged to impact the M2. This work is
for M3. We already check pointed M2.


keep in mind that the SchemaManager and all the classes were modified in
order to allow a better schema handling.

OK I'm not going to bother here. I'll just fix the problems I see.

Now, we are facing some new issues

because we want to allow schema objects to be loadable, which is a different
target, and for sure needs some heavy refactoring.

OK.


I suggest we close M2 now, and then discuss about what to do with the
SchemaManager.

Again this has no bearing. Please see above.
Np at all. I thought you where working on trunks. No reason to stop 
anything if you are in a branch.


For the record, yes, we have to refactoring this code now that we are exposing 
the Schema.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [Shared] [Model] What was the reason for removing the schema entity interfaces like AttributeType?

[Emmanuel Lécharny]

On 3/9/11 2:29 PM, Alex Karasulu wrote:

On Tue, Mar 8, 2011 at 6:12 PM, Alex Karasulu  wrote:


Let's be exact here. Are you saying that you were crucified?


I'd like a clear answer to this question.

You seem to be saying this too frequently and on public mailing lists for me
to ignore. The same insinuation last occurring over some IP issue I
discovered with a release. Again crucification was the word used. So either
stop the passive aggressive insinuations or just say what you're thinking
directly: I'm more than willing to work with you to be as gentle as you want
me to be with you.


Alex, please *stop* being paranoid.

This was a general sentence targeting no one, as your initial mail 
contained a similar one :


"...I'm not blaming anyone here so please don't get the wrong idea..."

At some point, you have to realize that you are working in a community 
where most of us are not native english speakers, and that reflects in 
the sentence we use and the word we pick to express an idea.


If I have to call someone specifically names, I have some way to express 
that feeling. Look at my first sentence, for instance.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Shade plugin errors

[Emmanuel Lécharny]

On 3/18/11 8:47 AM, Stefan Seelmann wrote:

Which goals and parameters did you use? Simple "mvn clean install"?


mvn clean install -Dintegration

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Shade plugin errors

[Emmanuel Lécharny]

On 3/18/11 10:58 AM, Alex Karasulu wrote:

On Fri, Mar 18, 2011 at 2:53 AM, Emmanuel Lecharnywrote:


Hi guys,

I have no idea why I get some erros on shared-all :

...
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-shade-plugin:1.4:shade (default) on project
shared-all: Error creating shaded jar: error in opening zip file
/Users/elecharny/apacheds/trunks/shared/asn1/api/target/classes ->  [Help 1]


Odd it seems as though it's trying to unzip a directory as if it were a jar
file.

Do you have a -X run you can post?


No, not anymore.

I rm -rf shared, doomed my repo, and rebuilt everything.
It works now.

Really really strange. Clearly a shaded-maven bug, but I'll have to dog 
the source to get a clue about it...



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Dn, RDn, Ava : Final classes ?

[Emmanuel Lécharny]

On 3/21/11 5:43 PM, Alex Karasulu wrote:

On Mon, Mar 21, 2011 at 6:39 PM, Emmanuel Lecharnywrote:


Hi guys,

do you think it's a good idea to make the Dn, Rdn and Ava classes final ?



I'd avoid this because it would limit extension. Are we gaining a lot by
making them final?
Not that much, but as the classes are immutable, I was thinking that 
it's not really necessary to have them not final. Also it might be 
faster (inline methods, etc)


Just wanted to hear other's opinion here.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Dn, RDn, Ava : Final classes ?

[Emmanuel Lécharny]

I reverted the change made on those classes a while back (they where all 
final) We can make them final if needed later, but I don't think it was 
a good idea to have made them final at first (most certainly my fault).


We can continue the discussion further.

On 3/22/11 12:31 PM, Alex Karasulu wrote:

On Mon, Mar 21, 2011 at 7:21 PM, Emmanuel Lécharnywrote:


On 3/21/11 5:43 PM, Alex Karasulu wrote:


On Mon, Mar 21, 2011 at 6:39 PM, Emmanuel Lecharny
wrote:

  Hi guys,

do you think it's a good idea to make the Dn, Rdn and Ava classes final ?


  I'd avoid this because it would limit extension. Are we gaining a lot by

making them final?


Not that much, but as the classes are immutable, I was thinking that it's
not really necessary to have them not final.


Immutability and extension are orthogonal concerns. You can still have
immutability in the base class, extend the class and add more fields that
are needed for some aspect of handling like we did for example with these
decorators in the codec for the req/resp classes.

Hence for the sake of the core properties they're still immutable but
extension may be an option left for users allowing the use of the class in
various scenarios we cannot think of in advance.

Also it might be faster (inline methods, etc)



Property accessor/mutators can be declared final but optimizations should
occur anyway, of course this is JVM dependent but I'd pass on small
optimization gains for now.



Just wanted to hear other's opinion here.



I appreciate that.

Regards,
Alex




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Some few classes to move out of model ?

[Emmanuel Lécharny]

On 3/22/11 11:15 PM, Stefan Seelmann wrote:



For instance, the DnPlaser class (well, it's not even a class, it's an enum,
which sound very strange to me)

I just followed Joshua Bloch's "Effective Java(TM) Second Edition":
Item 3: Enforce the singleton property with (...) an enum type.


Good idea.

But
now that the parsers are only used from the Dn/Rdn class that may not
make sense anymore.
Well, I think it does make sense. I kept the enum around, I just changed 
the enum to be package protected.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Go for a Apache Directory Studio 1.5.4 Bugfix Release

[Emmanuel Lécharny]

On 3/23/11 9:17 PM, Stefan Seelmann wrote:

On Wed, Mar 23, 2011 at 4:45 PM, Emmanuel Lecharny  wrote:

On 3/23/11 4:37 PM, Stefan Seelmann wrote:

Hi devs,

it's almost one year since the last Apache Directory Studio release
(1.5.3).
I think the 2.0.0 release will take some more time.

So I think we should release a 1.5.4 bugfix release in the next future.

I browsed the issue list in Jira and there are already some issues marked
as
fixed for 1.5.4. There are also many bugs reported by users within the
last
year we should fix. Last not least I also found some issues with MS AD and
ADAM I'd like to fix.

yes, sure. Or we may also want to eat the Milestone road instead. Like
releasing a 2.0.0-M1 instead of a 1.5.4.

IMO that are distinct cases.

Milestone releases to a 2.0.0 would be good, +1.

But the current trunk contains lot of changes. Not all people want to
use a milestone to work on productive directory servers. I think a
bugfix release of the current stable version makes sense.


The thing is that it will depend on a milestone itself (ie, LDAP API 
1.0.0-M2, or even M3). This is why I suggested to release a 2.0.0-M1 
instead of a 1.5.4.


However, it's not a really big deal, IMO. Whatever fits you is ok.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: ERR_732 Cannot process a Request while binding

[Emmanuel Lécharny]

On 3/28/11 7:56 PM, Amila Jayasekara wrote:

Hi Emmanuel,

Thanks for the reply.
As far as i know my application does not do any binding operations.
I don't know what you do with your application, but without binding, you 
are very unlikely to inject new data into the base.


How do you communicate with the server ? Which API are you using ?

But i will debug more, and see whether any binding requests are
originating. Thus the error message says "Cannot process a Request
while binding". What exactly meant by this error message ? In other
words, what is it referring to as "Request" ?
It means that some BindRequest is pending, and it's only generated when 
a SASL bind is being procedeed.

Does this error occurs if we try to send multiple bind requests at the
same time ?
Unlikely. Again, this error message is only meet when processing a SASL 
bind. There is no other place in the code where the 732 error is generated.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Filter parsing and values

[Emmanuel Lécharny]

On 3/30/11 3:01 PM, Alex Karasulu wrote:

On Wed, Mar 30, 2011 at 1:35 AM, Emmanuel Lecharnywrote:


Hi guys,

I'm facing an interesting issue. When we parse a filter, we build a Value
which is a BinaryValue, even if the attribute is H/R. The problem is that we
have to provide a getString() method which translates this byte[] when we
want to process the Value.

I tried to get rid of the getString() method in the StringValue class and
getBytes() in BinaryValue, but I can't because of the above issue.

Now, I'm wondering if it wouldn't be better to store a StringValue() when
we parse a Filter, as a default value, excpet if the Filter is schema aware
(which is not currently the case).



Most of the time ad hoc filters are string based. However we have seen
situations where they will be binary as well. As long as we're
interchangeable there should be no problem and since the binary case is the
exception then it makes sense using string values.
It's a bit more complex than that, in fact. May be I wasn't exposing the 
whole issue...


You may use binary values, using \XY for non ascii values. Something like :

(userCertificate=\00\AB\3F*)

is valid. Now, we can't store that in a StringValue, because the escaped 
hex are converted to bytes in the filter parser.


I was wonderig if it would not be a better idea to store the value as a 
byte[], unless we have a schema aware Filter instance, and then use 
either a StringValue or a BinaryValue, accordingly to the AttributeType 
used.


I also have more concerns :
1) do we have to deal with a schema aware filter on the client side ?

IMO, it *may* be usefull, as it allows the user to check the filter 
before sending it to the server.


2) do we have to expose all the XXXNode classes to the user ? I'm not 
sure it's a good idea.


wdyt ?


How we do this however is important since this impacts virtually everything
in the server.
yes, but it's not a big deal, as we correctly construct schema aware 
filters high enough in the interceptor chain (namely in the 
normalization filter).


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Continuum failures ...

[Emmanuel Lécharny]

On 4/6/11 11:05 PM, Stefan Seelmann wrote:

Hi Emmanuel,

On Wed, Apr 6, 2011 at 9:28 AM, Emmanuel Lecharny  wrote:

Hi guys,

it's now days Continuum is producing notifications about weird failures.
Those so-called failures have nothing to do with the code we pushed, it's
most certainly some problem in continuum itself.

I have removed the notification atm, but we have to question ourselves if we
should continue to use Continuum, when Je,kins is providing correct
information about the code state.

thoughts ?

Yes, that's annoying. The build that fails and sends notifiactions is
"ApacheDS 1.5 Build With Dependencies" which is the old
"apacheds/trunk-with-dependencies". May that be the reason?

Anyway, atm we have the following builds on Continuum:
   Apache Directory Complete Build With All Projects
   Apache Directory Shared
   Apache Directory Studio
   ApacheDS 1.5 Build With Dependencies
   ApacheDS Archetype Web Application
   Directory Skins

IMO it doesn't make sense to keep them all because the "Apache
Directory Complete Build With All Projects" (the new "trunks"
aggregator) contains them all. If there is no objection I'll delete
the oher builds.

Go ahead !


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: API documentation heads up

[Emmanuel Lécharny]

On 4/12/11 12:42 AM, Alex Karasulu wrote:

On Sat, Apr 9, 2011 at 10:26 AM, Emmanuel Lecharny  wrote:

Hi guys,

just to inform you that the Name and Entry packages classes have been
completely reviewed, and documentation for the classes is now completed.

I'm now moving to the Schema elements classes and documentation, starting
with AttributeType.


It's always good to have doco on this but I think there may be some
changes in the next milestone. You might want to put this off a tiny
bit until we get M4 out the door.


So I suggest we cut a 1.0.0-M3 right away, now that Entry and Name 
packages are clean.


I haven't spent too much time on schema the past few days, having had to 
deal with MINA release.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: API documentation heads up

[Emmanuel Lécharny]

On 4/12/11 1:50 AM, Alex Karasulu wrote:

I haven't spent too much time on schema the past few days, having had to
deal with MINA release.

Np was just saying your effort on documenting the schema code might
require changes after M4 when we refactor the schema model to reduce
dependencies. I did not want you to needlessly waste your time there.


I understand, but don't worry too much. What I was doing focus more on 
generalities than details, regarding the Schema documentation.


My main concern, as we are documenting the API, is how to use the 
SchemaManager, not how to create new Schema elements, because this is 
not something people will do frequently.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [VOTE] MINA 2.0.3, take 2

[Emmanuel Lécharny]

On 4/12/11 10:47 AM, Niklas Gustavsson wrote:

On Tue, Apr 12, 2011 at 10:39 AM, Emmanuel Lécharny
  wrote:

Although, we don't distribute a binary package, containing all the jars :
it's not useful, as nobody includes all the jars in its application (usually
only mina-core plus an extra module, but not often).

Not sure I'm getting this. We have distributed binary release packages
for the previous releases as far as I know. Here's the one for 2.0.2:
http://mina.apache.org/dyn/closer.cgi/mina/2.0.2/apache-mina-2.0.2-bin.tar.gz

And they are listed on the downloads page:
http://mina.apache.org/downloads.html


Hmmm, right...

I *think* I removed the distrib from the maven repository, so now the 
big tar ball is not anymore pushed there.


Ok, I'll copy them in people.a.o/~elecharny.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [VOTE] Release of Shared 1.0.0-M3

[Emmanuel Lécharny]

On 4/19/11 10:00 PM, Stefan Seelmann wrote:

Hi Emmanuel,

thanks for preparing the release. Most things look good to me.

But the checksums and signatures of the distribution packages are not
there. They also were missing at the last release. Aren't they
generated automatically when building the release? If not, could you
please create the checksums and signatures and upload them? That
should be fixed for the next release.


Ahhh, it's because I haven't signed them.
I wil do that. The wiki didn't mention this extra step, I will add it.

Another minor issue: the source distributions contain
"pom.xml.releaseBackup" and "release.properties" files. I wonder how
that happended? The source tar.gz and zip should be created from the
created tag.

I have no idea. This is something we have to investigate.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [VOTE] Release of Shared 1.0.0-M3

[Emmanuel Lécharny]

On 4/19/11 10:00 PM, Stefan Seelmann wrote:

Hi Emmanuel,

thanks for preparing the release. Most things look good to me.

But the checksums and signatures of the distribution packages are not
there. They also were missing at the last release. Aren't they
generated automatically when building the release? If not, could you
please create the checksums and signatures and upload them? That
should be fixed for the next release.


Ok, the distribution files have been signed.

The files are available on

http://people.apache.org/~elecharny/shared-1.0.0-M3/


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [VOTE] Release of Shared 1.0.0-M3

[Emmanuel Lécharny]

On 4/19/11 10:00 PM, Stefan Seelmann wrote:

Hi Emmanuel,

Another minor issue: the source distributions contain
"pom.xml.releaseBackup" and "release.properties" files. I wonder how
that happended? The source tar.gz and zip should be created from the
created tag.


I have added those exclude in th src.xml file :

pom.xml.releaseBackup
release.properties

Not sure if it's enough, and it also sounds like a bad hack.

We might fix that in a better way, but my maven skills are a bit too weak...

Also it's hard to determinate what's going on, because the release 
process is not a 2 mins task, and it's not a real pleasure to have to 
rollback everything :/


Anyway, thanks for the checks, Stefan !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [VOTE] Release of Shared 1.0.0-M3

[Emmanuel Lécharny]

On 4/20/11 6:10 PM, Stefan Seelmann wrote:

On Tue, Apr 19, 2011 at 10:59 PM, Emmanuel Lécharny
  wrote:


Ok, the distribution files have been signed.

The files are available on

http://people.apache.org/~elecharny/shared-1.0.0-M3/

Thanks for the update, but I only see the MD5 and SHA checksums, the
PGP/GPG signatures are still missing. Could you please add them?


What a dumb @ss I am ...

I signed the files using 4096R/7C6B7034 as the Key ID, and I just 
realized that the Key ID is just 7C6B7034, thus the .asc weren't 
produced. I didn't checked that. My bad...


Files (and sig) copied on 
http://people.apache.org/~elecharny/shared-1.0.0-M3/


Sorry for the latency...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: [VOTE] Release of Shared 1.0.0-M3

[Emmanuel Lécharny]

On 4/20/11 6:10 PM, Stefan Seelmann wrote:

On Tue, Apr 19, 2011 at 10:59 PM, Emmanuel Lécharny
  wrote:


Ok, the distribution files have been signed.

The files are available on

http://people.apache.org/~elecharny/shared-1.0.0-M3/

Thanks for the update, but I only see the MD5 and SHA checksums, the
PGP/GPG signatures are still missing. Could you please add them?


What a dumb @ss I am ...

I signed the files using 4096R/XXX as the Key ID, and I just 
realized that the Key ID is just XX, thus the .asc weren't produced. 
I didn't checked that. My bad...


Files (and sig) copied on 
http://people.apache.org/~elecharny/shared-1.0.0-M3/


Sorry for the latency...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Getting Exception while writting the DHCP response message to the Session

[Emmanuel Lécharny]

On 4/21/11 8:07 AM, Prasanna Kumar G wrote:

java.net.SocketException: Permission denied

Are you on windows ? Have you a firewall set ?

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Getting Exception while writting the DHCP response message to the Session

[Emmanuel Lécharny]

On 4/21/11 8:50 AM, Prasanna Kumar G wrote:

I run my own java program to handle Datagram packet as same as dhcp server
using mina and it's working fine in the same system.


Are you using the same port ?


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Getting Exception while writting the DHCP response message to the Session

[Emmanuel Lécharny]

On 4/21/11 9:17 AM, Prasanna Kumar G wrote:

Yes i am Running ApacheDS ladap Dhcp Server on 67 port And run my Java
Datagram packet handling program on port 67 only. Ldap dhcp server getting
exception and my program is working fine.


Strange...

Although the Apache DS DHCP server is not actively developed, so it may 
be buggy. I'm afraid that you are on your own here, but if you find some 
fix, we would be happy to apply it on trunk.




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Controls handling in the API

[Emmanuel Lécharny]

On 4/24/11 11:42 AM, Alex Karasulu wrote:

Hi Emmanuel,

On Sun, Apr 24, 2011 at 12:24 PM, Emmanuel Lecharnywrote:


Hi guys,

currently, we only implement a few controls, those that we support. In the
API, if one wants to use another control to send it to another LDAP server,
he has to use an OpaqueControl. So far, so good, except that it's really
painful as the value has to be encoded and decoded by the user.



Absolutely and this is why we groked the pain of having a plugin
architecture using OSGi to load bundles that add additional controls and
extended operations. Users can add explicit (none opaque) handling of
controls and extended operations by extending the codec.



I would suggest we create a shared-ldap-extra-control module, where we will
b able to add the Controls other LDAP server are using. We can have a
package for each LDAP server, assuming they use some special controls not
shared with any other. For shared controls, we can put them in the base
model.

Thoughts ?


We already have something like this: the shared-ldap-extras-codec module
contains a set of non-standard extra controls and extended operations that
we (Apache Directory) would provide our users who can optionally use it with
the API. So if we decide to implement a control and provide it we can
package it into this project.

We chose to group these extra controls and extended operations together into
a single bundle for now so all the extended operations and controls are
registered with the codec in one shot. We could have separated them so we
had 2 projects: a project module for extra controls and a project module for
extra extended operations. However with the limited number of extra
(optional) controls and extended operations we have it was easier to manage
as a group.

We can add more optional controls and extended operations to this module
until we notice a congestion problem and then can break it apart but for now
it perfectly suites our needs.

Is this what you were thinking about?


More or less. The 'extra-codec' name is not easy to deal with for our 
users. At some point, for an extra effort (ie splitting this module in 
two with explicit names) would probably help.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Cascade control : is it still usdeful ?

[Emmanuel Lécharny]

On 4/24/11 11:48 AM, Alex Karasulu wrote:

Hi Emmanuel,

On Sun, Apr 24, 2011 at 12:29 PM, Emmanuel Lecharnywrote:


Hi,

while checking the CascadeControl, I'm wondering f we use it at all in the
server. I'm not sure either about its semantic.

Alex, some insight maybe ?



Wow this one is really old. From what hazy memory I have of it, this control
is a marker control. It has no payload from what I remember.

The idea that drove creating it was to be able to issue subtree deletes. I
don't remember if there is an RFC or like minded control to do the same.
Have not checked. Don't remember if we even looked.

At some point PAM and I had some conversations about how to be able to
manage some schema operations and cascade deletes had come up but I don't
remember them now.

I don't think the control is being used, if even referenced it might have
just had some scaffolding setup for it without actual use.


AFAICT, it's only propagated to the schema handling part, and even 
there, it's not used.


If it has to be the treeDelete control, then I suggest we rename it and 
modify the way we handle it in the server. Many LDAP server implement 
this treeDelete control (a draft issued by M$ peeps back in 2000), may 
be we should have it...




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Controls handling in the API

[Emmanuel Lécharny]

On 4/24/11 12:17 PM, Alex Karasulu wrote:


At some point, for an extra effort (ie splitting this module in two with

explicit names) would probably help.


Yeah that would have been good but there's one extra little PITA problem
here. By splitting it you're going to have to have 4 modules. Right now
there are 2. One for the API holding the non-opaque control and extop
interfaces with specific accessor/mutator pairs to their payloads and the
actual codec extension (implementation) which does the work of
[de]marshalling to and fro.

If we break this up into a extras-controls and an extras-extended-ops we'll
have created two more new modules not just one for a total of 4 modules to
handle them.

Something more to think about.


I wasn't thinking about having 2 distinguished modules, but something 
like extra-extensions could be better. I don't really know, except that 
extra-codec does not fit.


At least, I wanted to ope a discussion around this.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: API announcement, V2

[Emmanuel Lécharny]

On 4/27/11 11:04 AM, Stefan Seelmann wrote:

Looks good.

Just some issues with the left-hand navigation of the web site:

- The link to "Generated Reports (e.g. JavaDocs)" was broken. I copied
the reports from /www/directory.apache.org/shared/gen-docs/1.0.0-M3 to
/www/directory.apache.org/api/gen-docs. I also added a symbolic link
"latest" there. I also updated the pom.xml of shared, with the next
release the site should be copied to "api".
Thank. I thought I copied the generated docs to www.apache.org, but I my 
have missed some steps.

- What about the "Tutorials"? Is it still up-to-date? Should that link
and page be removed?
I was thinking about removing it or replacing it with a LDAP API in 5 
minutes, because it's really too short to b a tutorial.


wdyt ?

PS : i'm going to work on the API doco in the next couple of days.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Apacheds package collision (OSGi)

[Emmanuel Lécharny]

On 4/27/11 12:54 PM, Alex Karasulu wrote:

Sorry I don't get it, can you list which two artifacts contain the same
package instead of this listing?


Sorry, I inverted the listing.

The packages come first, then for each duplication, you have the modules 
in which they are declared.


Let me reformat that :

package /org/apache/directory/server/core/ is declared in modules 
core-api and core
package /org/apache/directory/server/core/changelog/ is declared in 
modules core-api and core
package /org/apache/directory/server/core/event/ is declared in modules 
core-api and core
package /org/apache/directory/server/core/journal/ is declared in 
modules core-api and core
package /org/apache/directory/server/core/partition/ is declared in 
modules core-api and core
package /org/apache/directory/server/core/replication/ is declared in 
modules core-api and core
package /org/apache/directory/server/core/schema/ is declared in modules 
core-api and core


package /org/apache/directory/server/ is declared in modules 
http-directory-bridge, service, wrapper
package /org/apache/directory/server/config/ is declared in modules 
server-config and service-builder
package /org/apache/directory/server/core/integ/ is declared in modules 
core-integ and test-framework
package /org/apache/directory/server/kerberos/protocol/ is declared in 
modules kerberos-codec and protocol-kerberos
package /org/apache/directory/server/ldap/handlers/extended/ is declared 
in modules protocol-ldap and server-tools


We just have 12 duplications, most of them being between core and 
core-api (which makes sense. Renaming the core packages by adding an 
impl can solve this issue.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com





--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Search result in LDAP API

[Emmanuel Lécharny]

On 4/28/11 4:32 PM, Stefan Seelmann wrote:

b) Convenience methods:  higher-level objects (Dn, Rdn, Strings) are
passed and returned. Examples
void modify(String dn, Modification... modifications)
void rename (Dn entryDn, Rdn newRdn)
Cursorsearch( Dn baseDn, String filter, SearchScope scope,
String... attributes )
   ->those methods throw an LdapException in case of an error that
contains the LDAP error code and message.

We already have those convenient methods.

Yes, but they return a XyzResponse (except for the search) and the
user needs to check the response object to see if the operation was
successful or not (please correct me if I'm wrong). I think that is
not what 99% of the users expect.
Ahhh, right I see your point. Throwing an exception in case we had an 
error is way better in these case, sure.

So my idea of the "high-level" methods" is
- they take high-level objects as arguments
- the return type is void (for bind, unbind, add, modify, moddn,
delete, abandon), boolean (for compare), Cursor  (for search),
not sure about extended
- they throw an LdapException (or an appropriate sub-type) in case the
result code is not 0

Makes perfect sense.


- we should add support for referral handling and search continuation
(skip, chase, throw exception), not only for search, but also for the
write methods, but that could be done later

Yeah, it can be done later.

And the "low-level" methods:
- take XyzRequest objects as single argument
- they return an XyzResponse object (for bind, unbind, add, modify,
moddn, delete, abandon, compare, extended) or a SearchCursor (for
search)
- the respose objects must be inspected to find out if the operation
was sucessful or not, or if there was an referral/search continuation

So for the existing search() method that would mean:

SearchCursor search(SearchRequest searchRequest)
->  remains as is

Cursor  search(Dn baseDn, String filter, SearchScope scope,
String... attributes)
Cursor  search(String baseDn, String filter, SearchScope scope,
String... attributes)
->  return value changed, additional overloads to specify referral
handling may be needed



I just think a high-level and low-level API is required and try to
find a consistent rule.


Ok, fine. If everybody agree, we can start refactoring the existing API 
to provide such methods for 1.0.0-M4. Should not be complex.




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Search result in LDAP API

[Emmanuel Lécharny]

On 4/28/11 8:21 PM, Stefan Seelmann wrote:

On Thu, Apr 28, 2011 at 8:08 PM, Emmanuel Lecharny  wrote:

On 4/28/11 6:28 PM, Stefan Seelmann wrote:

This is not an easy choice.

The problme is that the user might be puzzled to see that a
LdapConnection.search( SearchRequest ) returns a cursor
  but
that the LdapConnection.search( base, filter, scope, attrs... ) returns a
cursor.

Using send( Request ) instead first makes the API lighter (we don't have
to
create one send() method per type of request, it can be hidden in the
LdapConnection class) and it's also for advanced users.

Another idea: We have two interfaces: LdapConnection and
LdapAsyncConnection. What about a third interface (e.g.
AdvancedLdapConnection, need to find a better name) which extends
LdapConnections. Then we can move all the methods for advanced users
to this new interface. Then the LdapConnection interface is much
smaller. We still can have a single LdapNetworkConnection class that
implements all three interfaces.

Thoughts?

Hmmm, that would be shitty... I tried to apply this modification on the
existing code, but sadly, we have two different implementations,
LdapCoreSessionConnection and LdapNetworkConnection. The first implements
LdapConnection, the second one implements LdapAsyncConnection.

Changing this would create a lot of problems...

Maybe we should keep it simple : search( SearchRequest ) returning a
SearchCursor for advanced users, and search( base, filter, scope, attrs...)
returning a Cursor  for normal users.

wdyt ?

Yes, I agree.


Ok, here is what I'm going to do :
- keep it as simple as possible, by just changing the return type for 
the moment


we can discuss further for other options, but at least, we will have 
something simple enough to start with.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Search result in LDAP API

[Emmanuel Lécharny]

On 4/28/11 8:21 PM, Stefan Seelmann wrote:

On Thu, Apr 28, 2011 at 8:08 PM, Emmanuel Lecharny  wrote:

On 4/28/11 6:28 PM, Stefan Seelmann wrote:

This is not an easy choice.

The problme is that the user might be puzzled to see that a
LdapConnection.search( SearchRequest ) returns a cursor
  but
that the LdapConnection.search( base, filter, scope, attrs... ) returns a
cursor.

Using send( Request ) instead first makes the API lighter (we don't have
to
create one send() method per type of request, it can be hidden in the
LdapConnection class) and it's also for advanced users.

Another idea: We have two interfaces: LdapConnection and
LdapAsyncConnection. What about a third interface (e.g.
AdvancedLdapConnection, need to find a better name) which extends
LdapConnections. Then we can move all the methods for advanced users
to this new interface. Then the LdapConnection interface is much
smaller. We still can have a single LdapNetworkConnection class that
implements all three interfaces.

Thoughts?

Hmmm, that would be shitty... I tried to apply this modification on the
existing code, but sadly, we have two different implementations,
LdapCoreSessionConnection and LdapNetworkConnection. The first implements
LdapConnection, the second one implements LdapAsyncConnection.

Changing this would create a lot of problems...

Maybe we should keep it simple : search( SearchRequest ) returning a
SearchCursor for advanced users, and search( base, filter, scope, attrs...)
returning a Cursor  for normal users.

wdyt ?

Yes, I agree.


Oops, I forgot to add one bullet to my list of simple modifications :
- I will make all the methods not using a request returning void or 
boolean for compare


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Fwd: Re: ApacheDS status and future

[Emmanuel Lécharny]

Forwarded to the mailing list

Emmanuel

Thanks for this, sounds very promising. I'll do some proof-of-concept work with 
1.5 for now and look forward to 2.0.  One more question for the future - you 
mention using OSGi, but our app runs in a web container (tomcat for example) 
and does not use OSGi internally.  Will it be possible to embed ApacheDS 
without OSGi?

Richard

(I subscribed to the list too late to get your reply, so this is a 
copy-and-paste job).

On 5/9/11 1:39 PM, Richard Evans wrote:

 Hi


Hi,

 We're looking at embedding ApacheDS into our java application, but I'm a bit 
concerned as to the status of the project.

Valid point :) Let's try to see what's going on.

 The last release was over a year ago and whilst this seems very functional, I 
get the impression that major changes are in the pipeline - discarding XML 
based configuration, for example.

Right. We decided to go with OSGi, because it opens many doors, for
instance. We also decided that Spring was not the right solution, and
that a LDIF based configuration was easier to manage, including when you
want to modify the server config without having to restart it.

Nevertheless, the LDAP part is still offering the very same features
that it was before. We are just polishing some others and also add some
new important ones:
- pluggable elements
- multi-master replication
- disaster recovery system


 So is this the right time to start embedding?  Is a new release due shortly?

We expect to go for quicker releases soon. The 1.0/1.5 etc release
schema was probably not a good idea, and we now are moving toward a
2.0.0-M1 release in the next 2 months. The idea is to be able to go for
very fast release (like once a month) with each release offering some
new feature (a bit à la eclipse).

 Will the online documentation get improved?

Well, hmmm, this is an open source project, and it's difficult to get
the doco written by coders... :/

 What is the status of the triplesec stuff - there is a link to this on the 
front page but I can't see it anywhere in svn.

It has been moved to a dormant status atm. We simply don't have time to
kiss it and wake him up atm. But as soon as the underlying pieces are in
a decent shape (ie 2.0.0-RC1), then TSec will probably be revived.

 Don't get me wrong, this is a great project, I just need to know what to do 
with it.


It's really up to you.  We can't make any promise, we ca just say we are
working hard to get a release out this year.

Also keep in mind that ApacheDS is not only about the server : we also
have released a LDAP API last week, and we have to get it polished at
the same time (this API is internally used by the server). We also have
heavy developements going on on Studio.

Lots of work for a bright future hopefully !


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: OSGi startup and shutdown problems

[Emmanuel Lécharny]

On 5/11/11 3:42 PM, Alex Karasulu wrote:

Usually, we are just hiding it if the user does not have its own OSGi
container.



Please clarify.


The user does not know we embed and start Felix. For him, it's 'hidden'.

Btw, the pb I was trying to solve (ie, the Felix EventDispatchQueue 
thread was still running even if we quit the application) has a nice 
workaround. The pb was that the felix thread was a User thread, not a 
Daemon thread, thus needed to be stopped explicitly. What would have 
been good is to be able to make this thread a Daemon thread, but as it's 
created internally of Felix, there was no way to do that.


Hopefully, Karl and Richard mentionned that if you start a thread from a 
daemon thread, then the inner thread will inheritate it's parent status. 
So I started felix inside a Daemon thread, and it did the trick :


Thread felixThread = new Thread(new Runnable() {
public void run()
{
try
{
felix.start();
}
catch ( BundleException e )
{
String message = "Failed to start embedded felix 
instance: " + e.getMessage();

LOG.error( message, e );
throw new RuntimeException( message, e );
}
}
}, "FelixThread");

felixThread.setDaemon( true );
felixThread.start();

instead of :

felix.start();


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: OSGi startup and shutdown problems

[Emmanuel Lécharny]

On 5/11/11 4:08 PM, Alex Karasulu wrote:

On Wed, May 11, 2011 at 4:56 PM, Emmanuel Lécharnywrote:


On 5/11/11 3:42 PM, Alex Karasulu wrote:


Usually, we are just hiding it if the user does not have its own OSGi

container.


  Please clarify.

The user does not know we embed and start Felix. For him, it's 'hidden'.



Well it's not immediately apparent since we don't have documentation for
this but eventually we will expose that this is what is actually taking
place in order to present this feature of being able to use codec extensions
for user defined controls and extended operations.

I know understand why you mean hidden. However this is not our agenda,
meaning to hide this implementation detail since those extending will have
to create their own OSGi bundles for extension points.


It's not accessible for the user. It's hidden deep into our code. Even 
if you document it, there is nothing the user can do to deal with Felix 
at this point.


In this context, hidden == buried. Not to be understand as 'hidden agenda'.

Btw, the pb I was trying to solve (ie, the Felix EventDispatchQueue thread
was still running even if we quit the application) has a nice workaround.
The pb was that the felix thread was a User thread, not a Daemon thread,
thus needed to be stopped explicitly. What would have been good is to be
able to make this thread a Daemon thread, but as it's created internally of
Felix, there was no way to do that.

Hopefully, Karl and Richard mentionned that if you start a thread from a
daemon thread, then the inner thread will inheritate it's parent status. So
I started felix inside a Daemon thread, and it did the trick :

Thread felixThread = new Thread(new Runnable() {
public void run()
{
try
{
felix.start();
}
catch ( BundleException e )
{
String message = "Failed to start embedded felix
instance: " + e.getMessage();
LOG.error( message, e );
throw new RuntimeException( message, e );
}
}
}, "FelixThread");

felixThread.setDaemon( true );
felixThread.start();

instead of :

felix.start();


Awesome!  That solves the thread kill issue. Did you commit this?


Not yet, I still have some issues with this trick. Working on it. (more 
or less, I have to be sure that Felix is started before continuing)



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: OSGi startup and shutdown problems

[Emmanuel Lécharny]

On 5/11/11 4:16 PM, Guillaume Nodet wrote:

Indeed, that's a nice trick.
If I understand the code correctly, you need to make sure the
exception is reported back to the current thread, else you the user
won't even notice any problem (well, in the log of course but no
exception).


Yes. It's just a trick at the moment, and some more love is needed to 
get it usable.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: API evolution

[Emmanuel Lécharny]

On 5/11/11 6:40 PM, Alex Karasulu wrote:

On Wed, May 11, 2011 at 7:25 PM, Emmanuel Lecharnywrote:


Again, using a connection.search( SearchRequest ), you get back a
Cursor   as a result.


This looks a lot cleaner. Great job Emmanuel.


Note that we have more to do, like handling referrals chasing. But this 
can be done later.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Creating ACIs in trunk code

[Emmanuel Lécharny]

On 5/13/11 11:47 AM, Richard Evans wrote:

Hi Emmanuel

But I'm running a new build from a recent (yesterday) checkout of 
trunk-with-dependencies.  Judging from the commit log for the fix for 1524, I 
have the fix in the source I have checked out and built.  The log shows that 
AciAuthorizationInterceptor is being initialised and reading the tuple cache.

My previous mail hasn't been sent to the ML.

I tested the code, and the entry is correctly added into the server, and 
found back the entry from it after a restart.


However, I found a bug in the TupleCache class : as we are looking for 
all the entries with an accessControlSubentry ObjectClass, and for the 
prescriptiveACI AT, we found none. The reason is that the search filter 
is getting back all the entry's attributes, except the operational ones 
(the PrescriptiveACI AT is operational), thus the warning.


I'm applying a patch and running tests atm. Will commit as soon as the 
tests correctly pass.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Creating ACIs in trunk code

[Emmanuel Lécharny]

I have applied a fix to trunk :

http://svn.apache.org/viewvc?rev=1102672&view=rev

Can you give it a try ?



On 5/13/11 11:47 AM, Richard Evans wrote:

Hi Emmanuel

But I'm running a new build from a recent (yesterday) checkout of 
trunk-with-dependencies.  Judging from the commit log for the fix for 1524, I 
have the fix in the source I have checked out and built.  The log shows that 
AciAuthorizationInterceptor is being initialised and reading the tuple cache.

I'd like to put together an Aci entry for the RootDSE to allow anonymous 
access.  Is there an example (or documentation) anywhere?

I do the test searches using ldapsearch or the javax.naming APIs.  With access 
control enabled but no Aci loaded I don't get the error, just an empty search.

Richard

-Original Message-
From: Emmanuel Lecharny [mailto:elecha...@gmail.com]
Sent: 13 May 2011 10:43
To: Apache Directory Developers List
Subject: Re: Creating ACIs in trunk code

Hi Richard,

On 5/12/11 6:17 PM, Richard Evans wrote:

I'm running a 1.5.8 snapshot freshly built from trunk-with-dependencies.  I'm 
set up access controls and am trying to define the 'enable search for all 
users' ACI entry as explained in the online docs.




This looks a bit like DIRSERVER-1524.  Have I created the Aci entry correctly 
for trunk code?

This is exactly the problem. It has been fixed in trunk. Otherwise, your
entries are perfectly fine.

A couple of other related queries:

1. With access controls enabled, I can't connect anonymously to the RootDSE - 
do I need another Aci entry for this?

Yes. RootDSE is also controlled by access controls rules.

2. Attempting a search as a test user, I would expect searches to fail with an 
error, instead I just get no results.  Is this intentional?

Depends on the LDAP API you are using. In any case, if you check the
ResultCode, you should have a *insufficientAccessRights* error (or
something similar)





--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: How the Dn of Child entries are updating when Parent Moved

[Emmanuel Lécharny]

On 5/17/11 12:30 PM, Prasanna Kumar G wrote:

Hi,

  I am trying to understand ApacheDS code. I am not understanding the
Moving of Entry(Renaming the Dn). When Renaming Dn of an Entry, where are
you updating the child entries.

We don't. We store RDNs, not DNs.

If not updating the child entries, will it cause any Problem?

No.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: How the Dn of Child entries are updating when Parent Moved

[Emmanuel Lécharny]

On 5/17/11 1:56 PM, Alex Karasulu wrote:

On Tue, May 17, 2011 at 2:08 PM, Emmanuel Lecharnywrote:


On 5/17/11 1:01 PM, Richard Evans wrote:


Related to this, is there any mechanism for automatically updating the
member value of a group (eg groupOfNames) if the DN of the member changes?


No, not yet. We need to add a mechanism allowing such an update.


  The best way for use to handle this is not to create a specific mechanism
for this but to use triggers and stored procedures to manage referential
integrity in general.


This was my first idea, but I think in this specific case, it would be a 
bit complex to manage : a memberof attributeType has references to many 
potential DNs, and as SP are only activated when a modification occurs, 
it would have to check all the places in the server which potentially 
holds a memberOf attribute and control if it contains one of the moved 
entry's DN.


Instead of what having a computed memberOf AttributeType would easily 
solve the problem: you keep a reference to the entry, and rebuild the DN 
on the fly.


Note that with a SP, we could do the exact same thing except that we 
don't execute a SP when processing search requests...


The best solution would of course to have a SP which is associated with 
the entry containing the memberOf, and does the lookup (a computed 
memberOf attribute, sort of)


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: How the Dn of Child entries are updating when Parent Moved

[Emmanuel Lécharny]

On 5/17/11 2:18 PM, Rajesh Akkineni wrote:
On Tue, 17 May 2011 16:07:24 +0530, Emmanuel Lécharny 
 wrote:



On 5/17/11 12:30 PM, Prasanna Kumar G wrote:

Hi,

  I am trying to understand ApacheDS code. I am not 
understanding the
Moving of Entry(Renaming the Dn). When Renaming Dn of an Entry, 
where are

you updating the child entries.

We don't. We store RDNs, not DNs.


But that RDN also not being used as the DN which is constructed in 
ServerEntrySerializer is not valid, and it isbeing overwritten in 
AbstractStore.lookup


ServerEntrySerializer :

public Object deserialize( byte[] bytes ) throws IOException
{
ObjectInputStream in = new ObjectInputStream( new 
ByteArrayInputStream( bytes ) );


try
{
Entry entry = new DefaultEntry( schemaManager );

// Read the Dn, if any
byte hasDn = in.readByte();

if ( hasDn == 1 )
{
Rdn rdn = new Rdn( schemaManager );
rdn.readExternal( in ); <== Here we read 
the RDN.



The AbstractStore.lookup method just rebuild the full DN, using the 
serialized RDN and the position in the tree of this entry.




So why to save the RDN at all?
Because it *is* the entry's only name. If we don't save it, we have no 
way to reconstruct the entry.

Why I am asking is, is there any other purpose for saving RDN?

No.

I suggest you step through the server code in debug mode, you'll see how 
it works. Keep in mind it's a huge piece of code, so don't expect to 
understand it inside out in a few days...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: How the Dn of Child entries are updating when Parent Moved

[Emmanuel Lécharny]

On 5/17/11 2:38 PM, Rajesh Akkineni wrote:
On Tue, 17 May 2011 17:59:58 +0530, Emmanuel Lécharny 
 wrote:



On 5/17/11 2:18 PM, Rajesh Akkineni wrote:
On Tue, 17 May 2011 16:07:24 +0530, Emmanuel Lécharny 
 wrote:



On 5/17/11 12:30 PM, Prasanna Kumar G wrote:

Hi,

  I am trying to understand ApacheDS code. I am not 
understanding the
Moving of Entry(Renaming the Dn). When Renaming Dn of an Entry, 
where are

you updating the child entries.

We don't. We store RDNs, not DNs.


But that RDN also not being used as the DN which is constructed in 
ServerEntrySerializer is not valid, and it isbeing overwritten in 
AbstractStore.lookup


ServerEntrySerializer :

 public Object deserialize( byte[] bytes ) throws IOException
 {
 ObjectInputStream in = new ObjectInputStream( new 
ByteArrayInputStream( bytes ) );


 try
 {
 Entry entry = new DefaultEntry( schemaManager );

 // Read the Dn, if any
 byte hasDn = in.readByte();

 if ( hasDn == 1 )
 {
 Rdn rdn = new Rdn( schemaManager );
 rdn.readExternal( in ); <== Here we read 
the RDN.



The AbstractStore.lookup method just rebuild the full DN, using the 
serialized RDN and the position in the tree of this entry.


I am sorry(for being persistently stupid), I have seen the above code. 
But here is code from AbstractStore


public Entry lookup(ID id) throws Exception {
Entry entry = master.get(id);

if (entry != null) {
Dn dn = buildEntryDn(id);
entry.setDn(dn);
return entry;
}

return null;
}
I see that the Dn is being replaced

s/replaced/reconstructed/

The serialization/deserialization occurs a layer down. In the 
AbstractStore, we already have a deserialized entry, containing the RDN.



. And Rdn or Entry is not being used to get the Dn. Using ID and 
rdnIndex we are getting the Dn, and rdnIndex does not depend on 
anything we save in masterTable.
There might be a slight improvement here, as the DN is constructed using 
the RDN table, right. Please fill a JIRA for that.




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: How the Dn of Child entries are updating when Parent Moved

[Emmanuel Lécharny]

On 5/17/11 3:16 PM, Alex Karasulu wrote:

On Tue, May 17, 2011 at 3:49 PM, Emmanuel Lécharnywrote:



  . And Rdn or Entry is not being used to get the Dn. Using ID and rdnIndex

we are getting the Dn, and rdnIndex does not depend on anything we save in
masterTable.


There might be a slight improvement here, as the DN is constructed using
the RDN table, right. Please fill a JIRA for that.


Sorry but it was not clear to me what sort of improvement you were
envisaging. Can you elaborate on that?
If the DN is reconstructed using the RDN table, then either we don't 
need to serialize the RDN in the entry at all, or we can stop building 
the DN when we reach the RDN, saving one extra step.


Let's discuss that on DIRSERVER_1618, Rajesh might have a valid point here.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Getting the feeling that we may need a ApacheDS release soon?

[Emmanuel Lécharny]

On 5/23/11 11:55 AM, Alex Karasulu wrote:

On Mon, May 23, 2011 at 12:52 PM, Emmanuel Lecharnywrote:


On 5/23/11 11:29 AM, Alex Karasulu wrote:


Hi all,

With the various posts and bugs people have to work around should we do
another release of ApacheDS under the current versioning scheme just to
let
our users breath easier.

I was thinking once we start moving to OSGi we can begin the 2.0-Mx
releases
but this is taking longer than I had thought. Always seeming to be the
case
these days for me at least.

Regardless of what release label we use I think we need a bug fix release
soon. Anyways WDYT?


I totally agree with the need of a release asap. We are trying to cut a API
1.0.0-M4 this week, but then, we *must* come back to ADS.

The Milestone system seems to fits remarkably well for teh API, I think we
should switch to milestone for ADS too. There are too many differences
between 1.5.7 and the current trunk (mainly the way we manage the
configuration) for our users to deserve an incremented version, IMO.

If possible, we could jump to a 2.0.0-M1 at the very beginning of june (may
be and of may) and then start incrementally as fast as needed with new
milestones, like a M2 for OSGi, a M3 for replication, a M4 for
Administrativepoints, etc.

Thoughts ?


+1 to all points made.


Let's start to fix issues that are listed for 2.0.0-M1 then. We have 129 
of them, I guess many will be postponed to 2.0.0-M2.


This is an exciting time again :)

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: DN & Attribute values issue : heads up

[Emmanuel Lécharny]

On 6/7/11 8:22 AM, Stefan Seelmann wrote:

On Mon, Jun 6, 2011 at 9:58 PM, Emmanuel Lecharny  wrote:

Hi guys,

following the two mails I sent today, I'd like to give a bit more heads up
about the pb we are facing

Currently, if we want to get the Dn/Rdn as a String, we have two methods to
do so :
- getName() which returns the User provided Dn's name
- getNormName() which returns the normalized Dn

The toString() method calls getName() internally.

So far, so good, except that it's not enough. There is a confusion between a
'normalized' DN and a 'Printable' Dn. Let's see with an example. The
following Dn :

"cn= Exa\+mple  one"

once parsed, will return the following results :
getName() : "cn= Exa\+mple  one "
getNormName() : "cn=Exa\+mple  one"

If the Dn is schema aware, the result will be slightly different :
getName() : "cn= Exa\+mple  one "
getNormName() : "2.5.4.3=exa\\+mple one"

Are there really two backslashes in the normalized DN? That is wrong.


No, it's just a copy from a Java String (so the '\' is duplicated). My bad.

Now, the problem is that the normalized value is only useful when
manipulating a Dn or a Rdn as a String. There are cases where we want to
compare the Rdn with some AttributeTypeAndValue. For instance, if we create
an entry like :

dn: cn= Exa\+mple  one
Objectclass: top
ObjectClass: person
sn: example
cn: exa+mple one

then the server will blow chunks because the CN attribute cannot be compared
to the Rdn ("exa\+mple one" is not equal to "exa+mple one").

I agree that the DN/RDN's name can't be used. Instead the RDN's AVA
type and value must be compared with the attribute's type and value.
This is what we do internally, but there are some other problems (see 
the mail I sent this morning)



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Values normalization and special characters

[Emmanuel Lécharny]

On 6/7/11 8:26 AM, Stefan Seelmann wrote:

On Tue, Jun 7, 2011 at 8:09 AM, Emmanuel Lecharny  wrote:

Hi guys,

I'm a bit deeper into the DN/RDN problem. Now, the issue I have is that I
don't know how we should handle shuch case :
- an AT like "cn: \ user" having an escaped space at the beginning.
- an entry which DN is "cn=\ user,ou=system"

The check is done internally comparing the RDN ("cn=\ user") and the
Attribute value. Sadly, when we parse the DN, we remove the '\' (as it's not
significant), keeping the leading space. But then, we normalize the RDN
applying the CN normalizer to it, which removes the leading space.

Here we should not compare the RDN and the attribute value. Instead
the RDN's AVA value need to be used for comparison. The RDN AVA value
should not contain the '\'.
It does not contain the '\', but the space is also stripped. So "cn: \ 
User" is internally transformed into an AVA containing "user" as a value 
(the \ and the space are removed after the value normalization...



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Values normalization and special characters

[Emmanuel Lécharny]

On 6/7/11 8:56 AM, Stefan Seelmann wrote:

On Tue, Jun 7, 2011 at 8:49 AM, Emmanuel Lécharny  wrote:

On 6/7/11 8:26 AM, Stefan Seelmann wrote:

On Tue, Jun 7, 2011 at 8:09 AM, Emmanuel Lecharny
  wrote:

Hi guys,

I'm a bit deeper into the DN/RDN problem. Now, the issue I have is that I
don't know how we should handle shuch case :
- an AT like "cn: \ user" having an escaped space at the beginning.
- an entry which DN is "cn=\ user,ou=system"

The check is done internally comparing the RDN ("cn=\ user") and the
Attribute value. Sadly, when we parse the DN, we remove the '\' (as it's
not
significant), keeping the leading space. But then, we normalize the RDN
applying the CN normalizer to it, which removes the leading space.

Here we should not compare the RDN and the attribute value. Instead
the RDN's AVA value need to be used for comparison. The RDN AVA value
should not contain the '\'.

It does not contain the '\', but the space is also stripped. So "cn: \ User"
is internally transformed into an AVA containing "user" as a value (the \
and the space are removed after the value normalization...

Yes, that sounds ok. But the space should also be removed from the
attribute value during normalization, so the normalized attribute
value should also be "user".

Is the space removed from both, the normalized value and user provided
value? It should only be removed from the normalized value.

The Up value remains the same.

What bugs me is that there is no reason to escape a space if it will be 
removed by the normalization...


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Alias cycle detection

[Emmanuel Lécharny]

On 6/13/11 10:12 PM, Howard Chu wrote:

Ludovic Poitou wrote:

Sun Directory Server, OpenDS, OpenDJ, Port 389 (formerly known as Fedora
Directory) at least, do not support Alias dereferencing because the 
complexity
out-weights the benefits, and very few client applications make use 
of it anyway.


My 2 cents.


We had no alias support in back-bdb originally; I added it back in 2003.

Indeed, you only need to track all of the alias entries that you hit, 
and that's typically a small number. Each new entry defines a new 
search scope. Any time a dereference leads to an entry that resides 
anywhere within your already-known search scopes that deref can be 
considered fully resolved, because the result was already part of the 
scope.

This is teh algorithm I foresee too.


It's actually quite simple and quite fast. Using the objectclass index 
it's trivial to obtain the list of all alias entries within the 
database, so from the outset you already know the maximum size of what 
you're dealing with.
We already have a cache that is constructed at startup, gathering all 
the aliases from the backend, using the OC index. This cache is of 
course updated on the fly, if one alias is added or removed.


I don't think it should take more than one day to fix this issue.

Thanks !


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Alias cycle detection

[Emmanuel Lécharny]

On 6/13/11 11:19 PM, Stefan Seelmann wrote:

It's actually quite simple and quite fast. Using the objectclass index
it's trivial to obtain the list of all alias entries within the database, so
from the outset you already know the maximum size of what you're dealing
with.

We already have a cache that is constructed at startup, gathering all the
aliases from the backend, using the OC index. This cache is of course
updated on the fly, if one alias is added or removed.

I don't think it should take more than one day to fix this issue.

In that case we can also get rid of all the alias indices (aliasIdx,
oneAliasIdx, subAliasIdx).


Yes absolutely.

There are a few steps we also have to fulfill :
- create an Alias cache ( I thought we had one, but in fact, we have the 
opposite : a notAliasCache in the ExceptionInterceptor)
- create an AliasInterceptor to manage the Add and Delete operations 
done on alias entries (and also move, rename and combined ops)

- modify the Search to handle a set of met aliases.

I'll proceed by creating the alias interceptor first, and Ill remove the 
part that handle Aliases in ExceptionInterceptor


The Alias index removal will be done at the end.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Alias cycle detection

[Emmanuel Lécharny]

On 6/14/11 3:26 PM, Stefan Seelmann wrote:

There are a few steps we also have to fulfill :
- create an Alias cache ( I thought we had one, but in fact, we have the
opposite : a notAliasCache in the ExceptionInterceptor)

I wonder why we need an alias cache? For fast lookup of the search
base in case the "find" bit is set?
Good point... As soon as we fetch an alias, we just have to check its OC 
to see if it's an alias, or not.

- create an AliasInterceptor to manage the Add and Delete operations done on
alias entries (and also move, rename and combined ops)

You mean that interceptor is used to update the cache, right?

Right. But if we don't need a cache, we don't either need this interceptor.

I will keep it atm to insolate the search filter in charge of detecting 
the cycle.

The Alias index removal will be done at the end.

Ok.


Two other issues I see with the new algorighm:
- It is efficient if there are only few aliases. But if a user adds
million of alias entries we may get a memory problem.
If a user adds millions of Alias entries, he will have issues with any 
LDAP server, I guess. And it's very unlikely to happen...


What we can do is to decide that we don't check for cycle once we have 
more than, say, 10 aliases in the search alias cache.



  I just want to
mention that to make clear that such an issue may occur. I don't think
it makes sense to create so many alias entries, but I saw an example
where group membership was implemented using aliases...

Make sense.


- It is possible that duplicates occur, for example if an alias
enlarges the initial search scope by pointing to a parent of the
initial search base. I think duplicates can be avoided by tracking
each search base and filter result enties within already processed
search bases.
Duplicate should not occur if we store the root DN  in the search alias 
cache : we will meet it at some point if the alias point to a parent.


Thanks for your feedback, it's very valuable.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Alias cycle detection

[Emmanuel Lécharny]

On 6/15/11 2:25 AM, Alex Karasulu wrote:

On Mon, Jun 13, 2011 at 5:39 PM, Emmanuel Lecharny  wrote:

Alias cycle detection
-

There is an unsolved question about how we should detect Alias cycles. Right
now, we check for cycles *before* they are created. The alternative would be
to stop any search that could lead to an infinite loop.

That would slow down reads. The best is to stop this from happening
with write operations: meaning doing the computation to detect and
prevent the cycle then and there instead of exhausting the search
process to deal with such wicked constructs.


If we do that, then we won't be able to pass VSLDAP tests. And the 
slowdown will be *very* minimal : we just need an additional filter 
which checks if the read entry is an alias (if not, we just do nothing), 
and if it's an alias, we put its DN in a Set of DNs in the 
SearchRequestContect.



The problem with the first approach is that we can't anymore pass the VSLDAP
tests. It's a major burden. Also most of the current servers support this
feature.

Is there a VSLDAP test that allows for alias cycle creation? If so we
should be able to bring this up with the Open Group. This is
definitely a gray area in the protocol but it makes little sense to
create alias cycles. Alias chaining on the other hand is a different
story.
Alias creation is not the server business. The user (or admin) is 
responsible for such mistakes. Sadly, it *will* happen, and we must 
forbid this. Right now, we detect a cycle the hard way, using indexes, 
and I'm pretty sure that in complex cases where the cycle is far away 
from the base search, it might take a while to detect it.


True, this is a gray area, because nowhere in the RFCs it is said that 
Alias cycle are forbidden, and we must protect the server from such 
behavior.

So let me ask once again since I know little about the VSLDAP tests:
do they allow alias chaining or alias loops? The two would be
different.


Alias chaining is definitively accepted.

Now, we also have some JIRAs that are a direct consequence of what we 
have coded in the server atm :

https://issues.apache.org/jira/browse/DIRSERVER-803
And this is a VSLDAP Standard compliance test.

I will continue my experiments on the Filter approach, it's almost done 
(I was surprised how easy it was to implement a cycle detection this 
way), then we can compare this algorithm with another where we forbid 
the cycle creation when adding a new alias.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: svn commit: r1135472 - /directory/apacheds/trunk/xdbm-partition/src/main/java/org/apache/directory/server/xdbm/AbstractStore.java

[Emmanuel Lécharny]

On 6/15/11 11:17 AM, Alex Karasulu wrote:

On Wed, Jun 15, 2011 at 8:52 AM, Emmanuel Lecharny  wrote:

On 6/15/11 2:45 AM, Alex Karasulu wrote:

-1

Please revert these changes. Sufficient consideration has not been
given to these matters which have been around for years. We need more
time to discuss these matters.

This check is just not good enough, even if we want to detect cycles when
adding aliases : it does not check anything but a cycle from an alias and a
direct ascendant.

I removed it because of its deficiencies, not because I wanted to remove the
check. If we decide to implement a check when creating an alias, we will add
a correct detection mechanism, trust me on that.

OK understood. I thought we were going to change this entire policy
regarding how we deal with aliases. Then this is no problem at all.
Adding a better check to prevent these cycles will be good. If we
don't do this we might seriously screw up search and the problems
generated will be very interesting if aliasDereferencing is enabled.

I just don't want to take such a step without guaranteeing we're not
going break search handling.


As I said to Alex on IM, I haven't closed the JIRA that relates the pb. 
I don't consider the problem fixed by the check removal. It's an 
intermediary step.



--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Alias cycle detection

[Emmanuel Lécharny]

On 6/15/11 11:21 AM, Howard Chu wrote:

Alex Karasulu wrote:
On Mon, Jun 13, 2011 at 5:39 PM, Emmanuel 
Lecharny  wrote:

Alias cycle detection
-

There is an unsolved question about how we should detect Alias 
cycles. Right
now, we check for cycles *before* they are created. The alternative 
would be

to stop any search that could lead to an infinite loop.


That would slow down reads. The best is to stop this from happening
with write operations: meaning doing the computation to detect and
prevent the cycle then and there instead of exhausting the search
process to deal with such wicked constructs.


You may be being over-paranoid here. First a client has to explicitly 
request alias dereferencing and most of them don't by default, 
Actually, they do. The default for people using JNDI is to Deref 
Always.(http://download.oracle.com/javase/jndi/tutorial/ldap/misc/aliases.html) 
("When you use Sun's LDAP service provider, you can control how aliases 
are dereferenced in one of four ways, by using the 
"java.naming.ldap.derefAliases" environment property, as shown in the 
following table. If this environment property is not set, then the 
default is "always".")
so in general reads will be unaffected. Also the DB operations 
required to detect a cycle at write time are the kinds of things you 
would already be performing efficiently in a search. Doing them at 
search time is far better from a concurrency perspective because 
you're only doing read operations inside a reader transaction, and 
nothing touched inside the DB needs to stay locked for long. If you're 
doing these searches during a write operation then you're going to 
accumulate huge numbers of locks that must be held until the write 
transaction commits.


And as I said in a previous mail, detecting cycle during search is 
probably totally unnoticeable, wrt performances.Of course, with millions 
of aliases, that would be a different problem, but in this case, I would 
likely tell the user to switch to another LDAP server :)




--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Is this intended, or should we consider that is a bug ?

[Emmanuel Lécharny]

On 6/15/11 2:01 PM, Alex Karasulu wrote:

On Wed, Jun 15, 2011 at 2:53 PM, Emmanuel Lecharny  wrote:

Hi,

I'm wondering if the following code snippet is just a bug, or if this was
intended :

public PresenceEvaluator( PresenceNode node, Store  db,
SchemaManager schemaManager )
throws Exception
{
this.db = db;
this.node = node;
this.schemaManager = schemaManager;
this.attributeType = node.getAttributeType();

if ( db.hasUserIndexOn( attributeType ) )
{
idx = db.getPresenceIndex();
}
else
{
idx = null;
}
}

In my case, the AttributeType is ObjectClass, which is a System index. I
find it extremely doubtful that we want to set idx to null in this very
case.

This is valid code for the general case when you don't know what the
attributeType is. If the attributeType is not indexed then we cannot
use the presence index for =* evaluation. In the case of objectClass
this does seem moot but this code operates on all cases so it's valid.

Does this explanation make sense?
Yes, but the ObjectClass AT *is* indexed, so why don't we set the idx to 
point on the ObjectClass index ? This is what I don't understand.


I was expecting such code :



   if ( db.hasUserIndexOn( attributeType ) || ( db.hasSystemIndexOn( 
attributeType ) )
   {
   idx = db.getPresenceIndex();
   }
   else
   {
   idx = null;
   }


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Is this intended, or should we consider that is a bug ?

[Emmanuel Lécharny]

On 6/15/11 2:09 PM, Alex Karasulu wrote:

On Wed, Jun 15, 2011 at 3:04 PM, Emmanuel Lécharny  wrote:

On 6/15/11 2:01 PM, Alex Karasulu wrote:

On Wed, Jun 15, 2011 at 2:53 PM, Emmanuel Lecharny
  wrote:

Hi,

I'm wondering if the following code snippet is just a bug, or if this was
intended :

public PresenceEvaluator( PresenceNode node, Storedb,
SchemaManager schemaManager )
throws Exception
{
this.db = db;
this.node = node;
this.schemaManager = schemaManager;
this.attributeType = node.getAttributeType();

if ( db.hasUserIndexOn( attributeType ) )
{
idx = db.getPresenceIndex();
}
else
{
idx = null;
}
}

In my case, the AttributeType is ObjectClass, which is a System index. I
find it extremely doubtful that we want to set idx to null in this very
case.

This is valid code for the general case when you don't know what the
attributeType is. If the attributeType is not indexed then we cannot
use the presence index for =* evaluation. In the case of objectClass
this does seem moot but this code operates on all cases so it's valid.

Does this explanation make sense?

Yes, but the ObjectClass AT *is* indexed, so why don't we set the idx to
point on the ObjectClass index ? This is what I don't understand.

I was expecting such code :



   if ( db.hasUserIndexOn( attributeType ) || ( db.hasSystemIndexOn(
attributeType ) )
   {
   idx = db.getPresenceIndex();
   }
   else
   {
   idx = null;
   }

OK I see what you mean. ObjectClass is a system index. Yes you are
right here. We changed this and did not update this code. I did not
understand this situation in my last post - my apologies.


Interesting debugging session :)

public boolean evaluateEntry( Entry entry ) throws Exception
{
if ( db.hasSystemIndexOn( attributeType ) )
{
// we don't maintain a presence index for objectClass, 
entryUUID, and entryCSN
// however as every entry has such an attribute this 
evaluator always evaluates to true

return true;
}


Seems like the idx can be null in this very case, as it's handled 
elsewhere in the code (here, attributeType is ObjectClass).


So my initial mail is now answered : it's intended.

Thanks guys !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Is this intended, or should we consider that is a bug ?

[Emmanuel Lécharny]

On 6/15/11 2:07 PM, Stefan Seelmann wrote:

On Wed, Jun 15, 2011 at 1:53 PM, Emmanuel Lecharny  wrote:

Hi,

I'm wondering if the following code snippet is just a bug, or if this was
intended :

public PresenceEvaluator( PresenceNode node, Store  db,
SchemaManager schemaManager )
throws Exception
{
this.db = db;
this.node = node;
this.schemaManager = schemaManager;
this.attributeType = node.getAttributeType();

if ( db.hasUserIndexOn( attributeType ) )
{
idx = db.getPresenceIndex();
}
else
{
idx = null;
}
}

In my case, the AttributeType is ObjectClass, which is a System index. I
find it extremely doubtful that we want to set idx to null in this very
case.

AFAIK we only add user attributes to the precence index. System
attributes (objectClass, entryUUID, entryCSN) are not added to the
precense index because they are always present in an entry so it would
be a waste of space to add them to the presence index. Instead in
another part of the PresenceEvaluator we check if the index is a
system index and return true.


Stefan, you are absolutely right. The following code demonstrate it :

public boolean evaluateEntry( Entry entry ) throws Exception
{
if ( db.hasSystemIndexOn( attributeType ) )
{
// we don't maintain a presence index for objectClass, 
entryUUID, and entryCSN
// however as every entry has such an attribute this 
evaluator always evaluates to true

return true;
}

Thanks a lot !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Alias handling issues

[Emmanuel Lécharny]

On 6/15/11 7:11 PM, Stefan Seelmann wrote:

On Wed, Jun 15, 2011 at 7:01 PM, Emmanuel Lecharny  wrote:

On 6/15/11 6:17 PM, Pierre-Arnaud Marcelot wrote:

On 15 juin 2011, at 18:01, Emmanuel Lecharny wrote:


On 6/15/11 5:19 PM, Emmanuel Lecharny wrote:

Hi,

sadly, things are not as rosy as I exepected...

I have written a few tests to check if we correctly detect cycles while
processing a search, and at some point, I found that cycle are not an issue
at all. But wait, it's not a good news :/

Let's say you have the following entries :

cn=test
  cn=foo,cn=test
cn=barAlias,cn=foo,cn=test ->cn=bar,cn=test
  cn=bar,cn=test
cn=dohAlias,cn=foo,cn=test ->cn=doh,cn=test
  cn=doh,cn=test

Logically, doing a SUBTREE search on cn=foo,cn=test, you should get the
following entries :
cn=foo,cn=test
cn=bar,cn=test
cn=doh,cn=test

This is *not* what we get. Currently, you'll have :
cn=foo,cn=test
cn=bar,cn=test

The second alias dereferencing is never done.

Obviously, it helps when it come sto avoid cycle, but this is certainly
not the expected behavior.

Now, if we want to do alias chasing on the server, we will have to
modify the way we handle alias : each one of them will issue a new search,
wth a new cursor.

Hopefully, stacking the aliases to be processed works well with the fact
that we have to stack the aliases for cycle detection. One possible solution
would be to process this stack alias after alias, and if we get back an
alias, we add it in the stack if it's not already present (otherwise, that
means we have a cycle).

In our sample, we will have the following stack :
() : empty stack
(cn=barAlias,cn=foo,cn=test) : first alias met
->here, we have processed all the entries for the initial search
  [cn=foo,cn=test]
  [cn=bar,cn=test] (the dereferenced alias)
->now, we get the leftmost element in the stack, and launch a new
search
() : this alias is being processed, thus
the<>
(, cn=dohAlias,cn=foo,cn=test) : second
alias met
->again, all the entries have been processed, we take the next alias
in the stack
  [cn=doh,cn=test]
(,) : second
alias met
->there are no further entries. We are done

Seems to work... Did I miss something ?

Yes, I missed something (thanks Pierre-Arnaud for pointing this out) :

if the alias get dereferenced to an entry below one of the DN already
stored in the stack, or to the descendant of one of the stored DNs, then we
don't need to proceed the search for this alias, as the entries have already
been provided. That will avoid sending duplicate entries to the user.

Yeah, a basic example would be to execute the same search as above but one
level higher, from "cn=test".

In that case, the first search will already give us the three resulting
entries:
- cn=foo,cn=test
- cn=bar,cn=test
- cn=doh,cn=test

And then, in the other searches, triggered when following aliases, some of
these entries would match again. For instance:
- cn=bar,cn=test
- cn=doh,cn=test

We have to make sure that we don't send them multiple times to the user.
Maybe by storing a list of all the DNs we already returned to the user...
But that might be a little bit overkill and suck too much memory.

By storing the root DN of each consecutive search, we will avoid such
problem.

In this case, the initial search is done with cn=foo,cn=test, and this is
the stored DN.
The first search will just returns the cn=foo,cn=test entry and the
cn=barAlias,cn=foo,cn=test entry, which will be dereferenced.

This dereferemced operation consists on doing a search on the aliasedObject
stored in the alias. As it's a new search, we store the base DN
(cn=bar,cn=test), and we launch the new search. Etc...

Should work pretty well without having to store all the entries' DNs

Yes, that stored DNs help for both, (1) avoiding duplicates and (2)
avoiding cycles. But remember, for millions of aliases you have to
store millions of DNs ;-)

Too bad for the millions of aliases :)

Those using too many aliases may :
- either have a very badly designed database
- or managing the CIA employees...

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Is the AndCursor useful ?

[Emmanuel Lécharny]

On 6/16/11 2:02 PM, Kiran Ayyagari wrote:

On Thu, Jun 16, 2011 at 5:00 PM, Emmanuel Lecharny  wrote:

On 6/16/11 1:18 PM, Kiran Ayyagari wrote:

On Thu, Jun 16, 2011 at 3:29 PM, Emmanuel Lecharny
  wrote:

Hi guys,

just wondering if we need to keep the intermediate AndCursor class ? All
in
all, when building the cursors, if we have a conjonction, we will pick
the
smallest cursor to store it as wrapped into the AndCursor, so I wonder if
it's not a good idea to simply remove this cursor from the equation ?


I don't think so, we do need evaluators also

What for ? Can't we use the underlying cursor's evaluator ?


yeap, agree, this indeed speeds up evaluation

OTOH, I prefer to keep a hint about conjunction in the returned cursor
or we can also change AndCursor to not use evaluators
but just as a dumb wrapper (i.e new AndCursor(IndexCursor) )


Hmmm, seems that the AndCursor also needs to evaluate candidate against 
the non selected indexes. Removing it will simply retruns too many entries.


PS : Sorry if those seem stupid questions, I'm trying to get a key about 
this part of the code I don't know atm. Thanks for the responses.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Changes in 'project' pom.xml and release of version 22

[Emmanuel Lécharny]

Hi all,

I'd like to make the following changes in project pom.xml:
- Surefire Plugin versions have been updated

When done I'd like to release version 22 and use that for the
upcoming shared and server release.

Kind regards
Emmanuel

[VOTE] Release of Apache Directory Server 2.0.0-M1

[Emmanuel Lécharny]

Hi,

This is a vote for the first milestone of the 2.0 Apache Directory
Server. While it's not considered as feature complete, it's a first
step toward the 2.0 version. Many issues have been fixed since the
last release (1.5.7) with more than 200 JIRA's issues closed.

The SVN tag:
http://svn.apache.org/repos/asf/directory/shared/tags/1.0.0-M4

The source and binary distribution packages:
http://people.apache.org/~elecharny/

The staging repository:
https://repository.apache.org/content/repositories/orgapachedirectory-038/

The generated site:
http://directory.apache.org/apacheds/gen-docs/2.0.0-M1/

The list of fixed issues :
https://issues.apache.org/jira/secure/IssueNavigator.jspa?mode=hide&requestId=12315879

Please cast your votes:
[ ] +1 Release Apache Directory Server 2.0.0-M1
[ ]  0 abstain
[ ] -1 Do not release Apache Directory Server 2.0.0-M1


--

Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Index reverse tables : are they useful ?

[Emmanuel Lécharny]

On 6/24/11 9:40 AM, Alex Karasulu wrote:

On Thu, Jun 23, 2011 at 7:20 PM, Emmanuel Lecharny  wrote:

Hi guys,

as I was reviewing the Table interface, and the Index hierarchy, I had a bit
of time to try to understand this part of the server I was not comfortable
with. And I saw that each index is using two tables : a forward and a
reverse table.

The forward table is obviously mandatory. It is used to retrieve the list of
entry's ID from an AT value. Fine. But what about the reverse table ?

Alex told me that it's useful when dealing with such filters :
(&(cn=A)(sn=B)).

The optimization engine will get the index that provides the smallest set of
candidate (say, CN here). We then grab from the CN forward table all the
entry's ID associated with the 'A' value. Let's call it the CN-IDs set.

Now, if we were to grab all the entries from the MasterTable to compare
their SN attribute with the 'B' value, t would be quite costly. We use the
SN's reverse table. For each entry's ID, we check if the associated value in
the SN's reverse table exists or not. If it exists, we have a valid
candidate, otherwise, we simply discard the candidate.

As we can see, we never fetch the entry unless we have a valid candidate (of
course, if the SN and CN AT are indexed).

However, I do think this reverse table is spurious. We can get the same
result by using the SN's forward table : for each entry's ID, check in the
SVn's forward table if the set of values (SN-IDs) contains the ID we pulled
from the CN-IDs set.

We can even do an SN-IDs ^ CN-IDs ( '^'<=>  intersection) to get the list of
valid candidates. No need to do N fetches in the reverse table (where N is
the number of ID in CN-IDs).

I think we can remove those reverse table, and I expect that it can speed up
the search a bit, but mainly speed up greatly any modification operation
(add, delete, modify) and even decrease the disk consumption.

I think this is possible. It would be interesting to test these two
different systems using different kinds of data. The number of
duplicate keys will obviously impact the performance: it will be a
trade off.

To test an Evaluator (i.e. sn=foo) on a candidate entry say id 37, we
need to perform a reverse lookup on the sn index. The btree key is 37
and we check if it contains a tuple with a value of 'foo'. If it does
then the Evaluator returns true if not it returns false. Now since we
know we're looking for tuple ('foo', 37) and we only have the forward
table we can lookup the 'foo' entries, and the values from duplicate
keys would be sorted. We would just need to check if 37 were in these
values.

The reverse index has no duplicate keys. The only way to get a
duplicate key in the reverse index is if the same entry (i.e. 37)
contained the same value ('foo') for the same (sn) attribute. And this
we know is not possible. So the lookups against the reverse table will
be faster.


I was thinking about something a bit different : as soon as you have 
grabbed the list of entry's ID from the first index, looking into the 
other indexes will also return a list of Entry's ID. Checking if those 
IDs are valid candidate can then be done in one shot : do the 
intersection of the two sets (they are ordered, so it's a O(n) 
operation) and just get the matching entries.


Compared to the current processing (ie, accessing the reverse index for 
*each* candidate), this will be way faster, IMO.


Note that it's true for the AND connector, but even for the OR 
connector, we don't have this kind of issue.

May be I'm missing something though. I have created a branch
(http://svn.apache.org/viewvc/directory/apacheds/branches/apacheds-no-reverse-index/)
to play with this idea.

As long as we're careful and don't just push this into the main branch
I'm very happy to experiment with this. Let's see really what the
costs are. Because I can sit around and theorize all day but some
benchmarks really show us what reality is about.
The main issue so far is with the RDN, oneLevel and Subtree indexes, 
which are using entry's ID, and are representing the tree hierarchy.


It impacts the way we delete entries, too, as we currently use the 
reverse index to quickly retrieve the values to remove from the forward 
index. However, we can just iterate on the entry's attribute to do the 
same thing, invoking the drop( K, Value) method instead of invoking the 
drop( ID ) method.


Very interesting experiment !

I'll commit what I'm coming with today.

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Index reverse tables : are they useful ?

[Emmanuel Lécharny]

On 6/24/11 9:51 AM, Alex Karasulu wrote:



Note that it's true for the AND connector, but even for the OR connector, we
don't have this kind of issue.

Sorry don't understand connector? You probably mean the cursors?


Sorry, from the mathematic semantic stand point, OR and AND are 
connectors (AFAIU). And /Or are "connecting" two terms together to form 
a new result.


I should probably have said operators.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Index reverse tables : are they useful ?

[Emmanuel Lécharny]

On 6/24/11 9:51 AM, Alex Karasulu wrote:



The reverse index has no duplicate keys. The only way to get a
duplicate key in the reverse index is if the same entry (i.e. 37)
contained the same value ('foo') for the same (sn) attribute. And this
we know is not possible. So the lookups against the reverse table will
be faster.

I was thinking about something a bit different : as soon as you have grabbed
the list of entry's ID from the first index, looking into the other indexes
will also return a list of Entry's ID. Checking if those IDs are valid
candidate can then be done in one shot : do the intersection of the two sets
(they are ordered, so it's a O(n) operation) and just get the matching
entries.

Compared to the current processing (ie, accessing the reverse index for
*each* candidate), this will be way faster, IMO.

This is a VERY interesting idea. Maybe we should create a separate
thread for this and drive deeper into it. You got something I think
here.

have a look at 
https://cwiki.apache.org/confluence/display/DIRxSRVx11/Index+and+IndexEntry, 
where I added some paragraphs explaining this idea. We can comment on 
this page.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

Re: Index reverse tables : are they useful ?

[Emmanuel Lécharny]

On 6/24/11 10:04 AM, Alex Karasulu wrote:

On Fri, Jun 24, 2011 at 11:00 AM, Emmanuel Lécharny
  wrote:

On 6/24/11 9:51 AM, Alex Karasulu wrote:

The reverse index has no duplicate keys. The only way to get a
duplicate key in the reverse index is if the same entry (i.e. 37)
contained the same value ('foo') for the same (sn) attribute. And this
we know is not possible. So the lookups against the reverse table will
be faster.

I was thinking about something a bit different : as soon as you have
grabbed
the list of entry's ID from the first index, looking into the other
indexes
will also return a list of Entry's ID. Checking if those IDs are valid
candidate can then be done in one shot : do the intersection of the two
sets
(they are ordered, so it's a O(n) operation) and just get the matching
entries.

Compared to the current processing (ie, accessing the reverse index for
*each* candidate), this will be way faster, IMO.

This is a VERY interesting idea. Maybe we should create a separate
thread for this and drive deeper into it. You got something I think
here.


have a look at
https://cwiki.apache.org/confluence/display/DIRxSRVx11/Index+and+IndexEntry,
where I added some paragraphs explaining this idea. We can comment on this
page.

Nice pictures - what did you use for that? Reading further ...

YeD

Also if you're doing this in a branch, hence we're not yet committed
on the approach, can you please do this on a separate page so you
don't alter the existing documentation?
Right now, I just updated the existing doco with some extended 
explanation on the existing code. I can move suggestions to another 
page, sure.


--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com

[VOTE] [Result] release of Shared 1.0.0-M5

[Emmanuel Lécharny]

The vote was successful with 5 +1 ( Alex, Felix, Kiran, Pierre-Arnaud

and me). No -1.

I'll release the binaries and update the site.

Thanks !

--
Regards,
Cordialement,
Emmanuel Lécharny
www.iktek.com