>> >>> >>> If your concern is that sysadmins will blindly trust LLM output without >>> verification - that's a valid concern, but it's also true of any >>> documentation, any Stack Overflow answer, any vendor support response. The >>> tool surfaces information; it doesn't replace the judgment needed to act on >>> it. That judgment was always required - with or without MCP. For >>> experienced admins who understand what they're looking at, I think it can >>> be a useful convenience. >> >> As an experienced admin, this would be a hinderance - not a convenience. I >> would forever be doubting if the surfaced information is truthful or not. >> >> There are better ways we can surface information about 389-ds to promote >> transparency and understanding for our admins and users. Again, in my view, >> this is not it. > > I don't think it's a hindrance - let me describe the workflow I have in mind. > > You're investigating an issue. You ask the model to check something > (replication status, healthcheck, performance metrics, etc.) across the whole > topology. It makes the lib389 calls, gathers the data, and presents you with > a graph or table giving you a basic picture of the issue you're > investigating. Then you go to dsconf and confirm what you're seeing. (I might > add dsconf commands to the output that would produce the same result, where > such commands are available.) The admin still does the real work - the MCP > layer just gets you to the starting point faster. > > About plausibility: the key thing is that we have the lib389 data. It's > accurate data from the server. Just to make my point - ask any modern LLM to > remember a number, then add 200 to it. It'll get it right 99.99% of the time. > That's essentially what this does: user asks a question, lib389 gathers > accurate data, LLM represents those accurate numbers in a more digestible > format.
But if I use calc.exe it shows me the correct answer 100% of the time. Not 99.99%. 100%. If there is even a 0.01% chance of failure, that means that 1 in every 10,000 occurrences an error will occur. 10,000 is not a large number. So now consider, how many users of 389-ds do we have? How often do they run the tools we wrote? How many "errors" are acceptable for us to put in front of our users? Are we really happy with a 1 in 10,000 failure rate? And to further this you are assuming that the LLM will point you into the correct direction where the error is. You have to always assume that it *may not*. LLMS are a non-deterministic sentence generator. They are not able to process data like a human. They do not understand context, or your environment. And that's what's so important here - they aren't deterministic, they can and will change the data they are given, and they can and will misrepresent things. > With current tech, there's arguably less chance of error than an unfocused > administrator parsing raw output after a night with no sleep. No - the admin still needs to parse and process the output regardless of if it's from an LDAP command or from MCP/LLM. The human still can make a mistake. The LLM just makes this more likely because then instead of having to build a mental model and understand, humans will potentially become complacent and blindly trust the LLM, even though it is an unreliable tool. We already see this in coding with the advent of "AI slop" PR's. > > And to be clear - with that information, the admin should do nothing except > go to their server with dsconf and continue the investigation there. This > doesn't replace that step; it accelerates getting to the specifics of it. In > some cases it might save hours, not just minutes. > > For something simple? No need for the LDAP MCP Assistant. For more complex > and specific cases? Can an admin go through the topology manually with > dsconf, collect reports, gather attributes, and juggle through all that > information for their specific case? Of course they can. Will it be tedious? > Probably. We (389 DS devs) might cover some situations where that initial > information representation saves time, we might not. But for the cases where > it does help, MCP saves a lot of initial kickstarter time for the > investigation. > > The tools will be tested and tuned, and the docs will advise caution. The MCP > goal in general is to help engineers design tools that are as precise (hence > helpful) as possible. From my testing, I think it provides enough benefit to > be used as a helper tool - a "junior admin" you ask for simple tasks, who may > save you time in complex data-gathering situations. In a complex data gathering situation, where the consequences are so high, I don't want an LLM collecting data given the chance that it will alter, redact, manipulate or just make up garbage. It is critical to an administrator to gather evidence carefully, and in a thorough manner to allow understanding the situation at hand. I think that's the fundamental difference here: You are seeing a hopeful case - I am seeing the worst case. And unfortunately in our line of work, we must consider the worst case as the consequences on our users, and the personal data stored in LDAP can be catastrophic. So I will ask - is it ethical and responsible to put a potentially inaccurate and unreliable tool in front of our users for some of the most critical infrastructure in their organisation? -- Sincerely, William Brown Senior Software Engineer, Identity and Access Management SUSE Labs, Australia
-- _______________________________________________ 389-users mailing list -- [email protected] To unsubscribe send an email to [email protected] Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/[email protected] Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
