I can confirm what Brett says on ADSI, and for that matter .NET. Anything those "toolsets" are doing is through the standard client interfaces exposed by AD/Windows through the LDAP, DS[1] and NET[2] APIs. The NET and DS calls all come through the RPC interface. Most of .NET thunks[3] down into ADSI which then thunks down to LDAP or NET.
For anyone interested, there is a new Dr. Dobbs out now with an article on DB replication in general and it talks about some open source DB and implementing replication there. It isn't the greatest written article but it does talk about several of the issues involved with keeping databases synced. I think the author spent almost a whole page of I don't know how many paragraphs explaining the need for a unique identifier for every object that flows between replicated instances which just kept shouting GUID GUID GUID to me. Maybe some folks would prefer to understand why the GUID is handy for this and that helps there as it states many of the problems and why you need that info. I disagreed with the described implementation and several of the ergo's stated but software development is a lot of opinions. :) joe [1] The DS API is stuff that admins tend to see through NLTEST or the brilliantly written repadmin. ;o) See the publicly published parts of that API here --> http://msdn.microsoft.com/library/default.asp?url=/library/en-us/ad/ad/activ e_directory_functions.asp [2] Not to be mistaken with DOT NET APIs. These are calls that have been around a long time in the NT world. Such as NetUserGetInfo and others at http://msdn.microsoft.com/library/default.asp?url=/library/en-us/netmgmt/net mgmt/network_management_functions.asp [3] This isn't imply any datasize changes, i.e. 16 to 32 bit or what not, but instead the generic thunk that is you are mapping from one convention to another. NET and ADSI are supposed to make things look consistent and they do that by trying to adopt one convention and handling under the covers for you, any other conventions that are needed to accomplish your goal on the different systems. -- O'Reilly Active Directory Third Edition - http://www.joeware.net/win/ad3e.htm -----Original Message----- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED] On Behalf Of Brett Shirley Sent: Monday, May 15, 2006 3:37 AM To: ActiveDir@mail.activedir.org Subject: RE: [ActiveDir] User Accounts I started, it will take a long time to do a proper diagram that doesn't take too many liberties w/ the actual implementation ... But I found something that is approximately accurate (but with too many liberties IMO): http://www.microsoft.com/technet/itsolutions/cits/interopmigration/unix/usec dirw/03wsdsu.mspx Skip down to see diagram 3.3 (Security sub-system's interaction), and especially 3.4 (Directory Service proper) ... they're ok, but I wouldn't read / trust the text too closely. One thing that is not made very clear in fig 3.4 is that everything except ESE and (most of) SAM is in ntdsa.dll. Also parts of LDAP and MAPI may use helper libraries to expose thier network heads (such as an ASN.1 [de|en]coding library + TCP / sockets stuff, and RPC respectively). I honestly don't know too much about ADSI, but if there is something ADSI can do that actually can't be done through LDAP, then I would suspect it is cheating and skipping around and using the SAM RPC head (what the "net apis" eventually trickle down to). The first diagram here is even further refined on the replication side (though has taken some liberties, though a scant less): http://technet2.microsoft.com/WindowsServer/en/Library/1465d773-b763-45ec-b9 71-c23cdc27400e1033.mspx?mfr=true When you saying the DB is different on each server ... what I think your trying to describe is that AD replication is what I would call "object logical". - object logical - meaning that two objects can be shown to be logially equivalent on separate servers, even if the actual datatable data, link_table data, etc are different. Though I might say it isn't pure, as some data on the objects may be different, when not replicated, such as USNs, instanceTypes, etc. If it was truly object logical, you wouldn't be able to view anything non-replicated/different from the object interface (LDAP). - Another option would be "database logical", meaning the ESE DBs could be described as logically equivalent (i.e. the same object's row, would have the same DNTs, etc) ... i think SQL offers something like this with at least one form of SQL replication (SQL Merge Replication is springing to mind?)? Also an offline defragged ESE database would be database logically the same as the original DB. - One last common option is physical replication, where the databases are equivalent data at the same byte offsets into the databases. Often done with transaction log shipping (although not the only option), which SQL supports, and Exch/ESE will support with E12 (well it's mostly physically equivalent). Very difficult (err impossible) to do multi-master replication with such a mechanism. Good point, yes, you could reset the DNTs by "defragging" the original datatable table to a new table, and thus compressing the number space to only those DNTs used, and keeping a translation table while you do it to fix up the references afterwards. As you point out, it would take a bit of time. This would be an O(n) operation with the number of objects in the DB, and thus "not in the spirit of IFM". It could probably be made somewhat fast, but still wouldn't compare to the raw file copy, IFM does today. Cheers, BrettSh On Mon, 15 May 2006, Ulf B. Simon-Weidner wrote: > Nice - poking with the finger works - give it to me babe ;-) > > I wasn't aware that ADSI is 100% LDAP, I thought it's just 9x% + some > special stuff (AFAIK setting pwds directly with LDAP doesn't work), so > I thought there's some stuff which supports it server side. > > Seems like you guys have a pretty good definition of the layers, would > be great if you get the time to create a diagram or just dump thoughts > to us and we'll handle visio. Having a diagram of the layers (even if > not 100% > correct) would make some things easier to explain. E.g. the > replication - it's pretty hard for many to understand that it's not > handled in the DB - they just think AD and don't get that the DB is different on each server. > > Resetting DNTs: OK - if DNT is a auto-incrementing primary key > (compared with SQL) there's a third option: reading the backup db and > writing it into the real, while keeping a dnt-translation table during > the process. However would slow down dcpromo /IFM (OK - not correct - > you know what I mean) and really doesn't make any sense since it would > be way easier to have larger values. And there would be other options > in the future, but mentioning those would make me look like and > alcoholic (and it's actually way to early here to handle thinking like that). > > Gruesse - Sincerely, > > Ulf B. Simon-Weidner > > Profile & Publications: > http://mvp.support.microsoft.com/profile=35E388DE-4885-4308-B489-F2F1214C811 > D > Weblog: http://msmvps.org/UlfBSimonWeidner > Website: http://www.windowsserverfaq.org > > > > > >-----Original Message----- > >From: [EMAIL PROTECTED] > >[mailto:[EMAIL PROTECTED] On Behalf Of Brett > >Shirley > >Sent: Monday, May 15, 2006 7:57 AM > >To: ActiveDir@mail.activedir.org > >Subject: RE: [ActiveDir] User Accounts > > > >Hmmm, you've actually combined too many layers in my opinion ... ADSI > >is client side, and based entirely on LDAP, and there is an LDAP > >marshalling component both on the client and LDAP server. Having an > >arch diagram where you don't clearly differentiate where the network > >interfaces is, seems confusing. The replication logic is actually > >split fairly evenly between the Directory and DBLAYER. USNs are in > >the dblayer for instance, while things like instanceType are handled > >in the Directory layer. > > > >With the current ESE level schema defined for the ntds.dit by AD you > >could not reuse DNTs, even after IFM. This is because AD creates the > >DNT column with the JET_bitColumnAutoincrement, so the > >auto-increment-ness is done in the ESE layer. I don't believe > >(though not 93% sure on this) that ESE provides a way to explicit set > >an auto-increment column, so you're stuck losing those DNT values. > >You would either have to add the ability to reuse orphaned auto-inc's > >in ESE, or make AD define the column as a regular integer, and manage > >the auto-inc'ness and reuse itself. Neither of those options is > >probably as good as making AD just have 64-bit DNTs. > > > >I'll try to write up a more explicit arch diagram, that is a little > >more accurate if it doesn't take me too long ... > > > >Cheers, > >BrettSh [msft] > > > > > >On Sun, 14 May 2006, Ulf B. Simon-Weidner wrote: > > > >> Agreed - very good thread. Let's extend the model a bit: > >> > >> ----------------------- > >> | ... | > >> | LDAP/NETLOGON/ADSI | <- Services using the > >Dir/providing interfaces > >> | ... | > >> ----------------------- > >> | | The Directory provider itself > >> | Directory | <- Replication works in here, so > >everything below > >> is local to the DC > >> | | Version numbers, USN,.. are > >managed here > >> ----------------------- > >> | | > >> | DBLAYER | <- Gluepart between Directory and DB > >> | | (P)DNTs, Links, SIS-SDs,.. are > >managed here > >> ----------------------- > >> | | > >> | DB | <- Just the ESE with it's > >features, such as > >> defrag > >> | | > >> ----------------------- > >> > >> I also believe that the "not reused DNTs" on IFM is by design, IMHO > >> there would be a possibility to reset DNTs programmatically > >after IFM, > >> however this would need additional code and time after > >reading the DB > >> and rebooting the DC for the first time. > >> > >> Gruesse - Sincerely, > >> > >> Ulf B. Simon-Weidner > >> > >> Profile & Publications: > >> > >http://mvp.support.microsoft.com/profile=35E388DE-4885-4308-B48 > >9-F2F1214C811 > >> D > >> Weblog: http://msmvps.org/UlfBSimonWeidner > >> Website: http://www.windowsserverfaq.org > >> > >> > >> > >> > >> >-----Original Message----- > >> >From: [EMAIL PROTECTED] > >> >[mailto:[EMAIL PROTECTED] On Behalf Of joe > >> >Sent: Friday, April 28, 2006 10:36 PM > >> >To: 'Send - AD mailing list' > >> >Subject: RE: [ActiveDir] User Accounts > >> > > >> >This is a good thread, I should have kept up with it. :) > >> > > >> >I think some of the problem is resulting from language > >> >interpretation. When I visualize AD in regards to the > >topics in this > >> >thread I think of it sort of like > >> > > >> >----------- > >> >| | > >> >| "AD" | > >> >| | > >> >----------- > >> >| | > >> >| DBLAYER | > >> >| | > >> >----------- > >> >| | > >> >| DB | > >> >| | > >> >----------- > >> > > >> > > >> >Depending on who you are you make look at all three boxes as AD > >> >and truly for most everyone that is the case. However when > >> >speaking at the internal component level these are three main > >> >areas, it > >could be > >> >broken up into even more like for instance SAM, Kerb, Replication, > >> >LDAP, etc. > >> > > >> >But I think where some confusion may have come in when saying AD > >> >dblayer. To many that would read as the DB. But I am reading it as > >> >the layer that interfaces or more properly abstracts the > >the lower DB > >> >portions from the high level AD stuff. That way you could > >jack up AD > >> >and slide another DB under it say something good like > >Oracle or MySQL > >> >or notepad or something <eg> and make most adjustments at the > >> >dblayer, sort of like a HAL. So we could call the dblayer > >> >something more like DBAL. I expect the abstraction isn't that > >> >fully > >fleshed out > >> >and there is still dependencies based on the underlying DB > >tech but I > >> >expect that could be worked through rather speedily, those AD Dev > >> >guys are a generally smart bunch. > >> > > >> >Microsoft could look into a reuse system for older DNTs but > >it would > >> >be more logical, IMO, to just expand the bit size of the variable. > >> >Since again, these DNTs are local it wouldn't be an issue except > >> >in the case of IFM promos, you would now be in a situation where > >> >you could IFM from a machine with a 32 bit DNT to one with 32 > >bit DNTs or > >> >64 Bit DNTs but if you have a backup from a 64 bit machine > >you could > >> >only IFM with another > >> >64 bit machine (even that could be made to work if you could > >> >guarantee that the high half of the variable wasn't being used but > >> >you would be silly to even start going in that direction). > >> > > >> >Anyway... Chase down the guy who stole the bit and get it > >back and we > >> >double the DNTs, fire someone and get another bit and double again > >> >(and you thought bits were just small little things...). > >Get it over > >> >with and go to 64 bits or really have fun and use 128. Of > >course this > >> >has implications on performance on 32 bit machines but those > >> >should be dropping off now that we are saying people need to load > >> >64 bit OSes anyway - who is going to want to run 32 bit DCs with > >> >64 bit Exchange pounding on them[1]? MS did it for Exchange, why > >> >not force the issue with AD as well in LH? Exchange 12 is due out > >> >before LH isn't it? Everyone should be used to being slapped and > >> >told > >they have > >> >to say they like it by then. :) > >> > > >> > joe > >> > > >> > > >> > > >> >[1] Being facetious here, though I don't really expect MS > >Exch Dev to > >> >change how they recommend DC hardware for Exchange. > >> > > >> >-- > >> >O'Reilly Active Directory Third Edition - > >> >http://www.joeware.net/win/ad3e.htm > >> > > >> > > >> >-----Original Message----- > >> >From: [EMAIL PROTECTED] > >> >[mailto:[EMAIL PROTECTED] On Behalf Of Dean > >> >Wells > >> >Sent: Wednesday, April 19, 2006 10:46 AM > >> >To: Send - AD mailing list > >> >Subject: RE: [ActiveDir] User Accounts > >> > > >> >Inline ... > >> > > >> >-- > >> >Dean Wells > >> >MSEtechnology > >> >* Email: [EMAIL PROTECTED] > >> >http://msetechnology.com > >> > > >> > > >> > > >> >> -----Original Message----- > >> >> From: [EMAIL PROTECTED] > >> >> [mailto:[EMAIL PROTECTED] On Behalf Of Ulf B. > >> >> Simon-Weidner > >> >> Sent: Wednesday, April 19, 2006 2:40 AM > >> >> To: ActiveDir@mail.activedir.org > >> >> Subject: RE: [ActiveDir] User Accounts > >> >> > >> >> >* DNTs (to me) are _not_ a component of the directory > >> >> > >> >> IIRC they are like a (primary/foreign) key in a database. > >> >> Technically not needed by the database layer, and not > >needed by the > >> >> application, but needed to keep the data together for the > >> >application. > >> >> So if you look at AD from the outside it won't be referenced, if > >> >> you look at ESE it's just a DB and doesn't care about the data > >> >> stored within, but you still need it in between to store > >the AD in the ESE. > >> >> Right? > >> > > >> >Heh, depends since the dblayer _is_ the component that implements > >> >them, not ESE. > >> > > >> >> >* DNTs are not reusable > >> >> > >> >> Unique per Server and don't provide any reference across servers. > >> >> If AD looks for a parent object by looking up it's known DNT > >> >(stored with > >> >> the child), ESE would fail in that moment, AD would not able > >> >to go to > >> >> another server and look up the same DNT in it's database. > >The AD is > >> >> distributed, the ESE is local, and DNTs are part of the > >local table. > >> > > >> >The DN of an AD object is the result of its DNT (or > >> >P[parent]DNT) ancestry, right the way back to a number of > >structural > >> >entries (I believe they're typically referred to as "structural > >> >phantoms" but don't quote me on that) that define the labels > >> >comprising the NC head. > >> > > >> >> If I understand correctly: > >> >> DNTs are reusable in ESE, however ADs implementation does > >not allow > >> >> DNTs to be released / reused on a single server > >> > > >> >Since DNTs are not a natural component of ESE, the answer is > >> >implementation specific. > >> > > >> >> , and > >> >> the database will only "reuse" > >> >> them if you recreate the DB by repromoting (cause the data is > >> >> replicated from other servers into a virgin ESE, and DNTs > >> >are assigned > >> >> from the beginning at this point). > >> > > >> >The re-promotion aspect is of course true, assuming non-IFM. > >> > > >> >> Right? > >> >> > >> >> Gruesse - Sincerely, > >> >> > >> >> Ulf B. Simon-Weidner > >> >> > >> >> MVP-Book "Windows XP - Die Expertentipps": > >http://tinyurl.com/44zcz > >> >> Weblog: http://msmvps.org/UlfBSimonWeidner > >> >> Website: http://www.windowsserverfaq.org > >> >> Profile: > >> >> http://mvp.support.microsoft.com/profile=35E388DE-4885-4308-B4 > >> >89-F2F1214C811 > >> >> D > >> >> > >> >> > >> >> > >> >> |-----Original Message----- > >> >> |From: [EMAIL PROTECTED] > >> >> |[mailto:[EMAIL PROTECTED] On Behalf Of Dean > >> >> |Wells > >> >> |Sent: Wednesday, April 19, 2006 1:18 AM > >> >> |To: Send - AD mailing list > >> >> |Subject: RE: [ActiveDir] User Accounts > >> >> | > >> >> |Inline is my take on an IM conv. Brett and I just had, the > >> >> result and > >> >> |content of which turned up some interesting (to me at least) > >> >> |implementation details. The short story is - > >> >> | > >> >> |* DNTs (to me) are _not_ a component of the directory > >> >> | - they _are_ a component of the layer that bridges the > >> >> two (dblayer) > >> >> | - to Brett, I believe he sees them within the sum of > >> >> "what is the > >> >> |directory" > >> >> |* DNTs (to both Brett and I) are not part of ESE > >> >> |* DNTs are limited (as Eric says) to 2^31 (~2.1 billion rows) > >> >> |* DNTs are not reusable > >> >> | > >> >> |I hope the summary and conversational text inline proves useful. > >> >> | > >> >> |-- > >> >> |Dean Wells > >> >> |MSEtechnology > >> >> |* Email: [EMAIL PROTECTED] http://msetechnology.com > >> >> | > >> >> | > >> >> | > >> >> |> -----Original Message----- > >> >> |> From: [EMAIL PROTECTED] > >> >> |> [mailto:[EMAIL PROTECTED] On Behalf Of > >> >> |Brett Shirley > >> >> |> Sent: Tuesday, April 18, 2006 5:11 PM > >> >> |> To: ActiveDir@mail.activedir.org > >> >> |> Cc: Send - AD mailing list > >> >> |> Subject: RE: [ActiveDir] User Accounts > >> >> |> > >> >> |> > >> >> |> Dean, I didn't understand this comment ... > >> >> |> > But, dude, seriously, you weren't aware that AD's ESE > >> >> |used a 32 bit > >> >> |> DNT? > >> >> |> > Methinks perhaps you're muddling in the realms of personal > >> >> |> interpretation > ... though I'm quite certain you'll > >> >> argue that too > >> >> |> ... ESE purist :0p > >> >> |> > >> >> |> Are you claiming that ESE knows what a DNT is? > >> >> | > >> >> |Not at all ... but IMO, neither does the directory ... and > >> >> per our IM, > >> >> |the dblayer knows what they are (after all, DNT = > >> >distinguished name > >> >> |tag ... > >> >> |blatantly not an ESE term ... and dblayer = database layer ... > >> >> |not a directory term ... hmmm) > >> >> | > >> >> |> A DNT is an entirely AD concept, ESE has no idea what a DNT is. > >> >> | > >> >> |Nod. > >> >> | > >> >> |> ESE also has no concept of linked-values, or the link_table. > >> >> | > >> >> |Now this was news to me, so here's the summary: ESE has tables > >> >> |+ columns + indices over columns. The dblayer forms the > >> >> |bridge between two technologies, one molding the behavior of > >> >> the other > >> >> |(dblayer molds ESE). > >> >> |ESE maintains no referential integrity, the dblayer does > >this ... > >> >> |including link-pairs <-- this part was especially > >surprising to me. > >> >> | > >> >> |> This is the 2nd time you've confused the AD dblayer (what > >> >> maintains > >> >> |> the AD schema on an ESE > >> >> |> database) and the ESE database layer. > >> >> | > >> >> |Don't know that I'd agree with that since on neither > >> >> occasion was the > >> >> |dblayer specifically referenced .. but it's moot for the > >> >> moment since > >> >> |I'm still mulling over whether my new-found knowledge > >> >> |pertaining to link-pairs influences my opinion on where DNTs > >> >> |lie; directory or database. > >> >> | > >> >> | > >> >> | > >> >> |List info : http://www.activedir.org/List.aspx > >> >> |List FAQ : http://www.activedir.org/ListFAQ.aspx > >> >> |List archive: > >> >> |http://www.mail-archive.com/activedir%40mail.activedir.org/ > >> >> > >> >> List info : http://www.activedir.org/List.aspx > >> >> List FAQ : http://www.activedir.org/ListFAQ.aspx > >> >> List archive: > >> >> http://www.mail-archive.com/activedir%40mail.activedir.org/ > >> >> > >> >> > >> > > >> > > >> >List info : http://www.activedir.org/List.aspx > >> >List FAQ : http://www.activedir.org/ListFAQ.aspx > >> >List archive: > >> >http://www.mail-archive.com/activedir%40mail.activedir.org/ > >> > > >> >List info : http://www.activedir.org/List.aspx > >> >List FAQ : http://www.activedir.org/ListFAQ.aspx > >> >List archive: > >> >http://www.mail-archive.com/activedir%40mail.activedir.org/ > >> > >> List info : http://www.activedir.org/List.aspx > >> List FAQ : http://www.activedir.org/ListFAQ.aspx > >> List archive: > >> http://www.mail-archive.com/activedir%40mail.activedir.org/ > >> > > > >List info : http://www.activedir.org/List.aspx > >List FAQ : http://www.activedir.org/ListFAQ.aspx > >List archive: > >http://www.mail-archive.com/activedir%40mail.activedir.org/ > > List info : http://www.activedir.org/List.aspx > List FAQ : http://www.activedir.org/ListFAQ.aspx > List archive: > http://www.mail-archive.com/activedir%40mail.activedir.org/ > List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/ List info : http://www.activedir.org/List.aspx List FAQ : http://www.activedir.org/ListFAQ.aspx List archive: http://www.mail-archive.com/activedir%40mail.activedir.org/
