I have not as yet created the visual or really mapped anything out yet, it has been a somewhat oddly long working strectch for me this last week, so I will have some time to myself on Sunday of this week to work solely on this effort.
garotconk...@yahoo.com ________________________________ From: "webmas...@lisol.co.uk" <webmas...@lisol.co.uk> To: 'Garot Conklin' <garotconk...@yahoo.com>; 'Martin Gainty' <mgai...@hotmail.com>; ag4ve...@gmail.com Cc: mysql@lists.mysql.com Sent: Friday, August 24, 2012 4:04 AM Subject: RE: Machine Learning Hi Garot, How is the visual diagram coming along ? I'd be good to have a good idea of core system specifics so that DB design can be tailored to that ... of course, knowing that things will change as the concept evolves. Thanks, Justin -----Original Message----- From: Garot Conklin [mailto:garotconk...@yahoo.com] Sent: 23 August 2012 20:14 To: Martin Gainty; ag4ve...@gmail.com Cc: webmas...@lisol.co.uk; mysql@lists.mysql.com Subject: Re: Machine Learning That is in fact a very interesting approach... I like it very much. Any architectural thoughts on an auto-population attempt? Odly I have not created a db like this before, not from this large of an import anyway... Please share some thoughts there and I will get that rolling... seems that if we can get something physical going I would like to offer it up as a sandbox to all those interested. I have a few X336's lying around my house just collecting dust, they would be a great testing environment for all to work on. garotconk...@yahoo.com ________________________________ From: Martin Gainty <mgai...@hotmail.com> To: ag4ve...@gmail.com; garotconk...@yahoo.com Cc: webmas...@lisol.co.uk; "mysql@lists.mysql.com" <mysql@lists.mysql.com> Sent: Thursday, August 23, 2012 11:30 AM Subject: RE: Machine Learning Shawn and Garot I like the parsing capability of the lucene and its ability to stem incoming queries.. If you are able to start populating your data then we *should* be able to start identifying which root terms we can pull for building lucene-indexes I have an upcoming stuck-in-JFK-airport-on-labor-day-waiting-for-the-next-am-flight-of-USscare for an interrupted timeframe(i never learned how to 'sleep' in airport terminal) as long as I can find a wifi connection (and an AC connection within 6 hours) and use a working MySQL JDBC connection string..I should be able to bang out a quick prototype.. Let me know how i can help out, Martin Gainty ______________________________________________ ..place long-winded disclaimer here.. > From: ag4ve...@gmail.com > Date: Thu, 23 Aug 2012 10:42:55 -0400 > Subject: Re: Machine Learning > To: garotconk...@yahoo.com > CC: webmas...@lisol.co.uk; mgai...@hotmail.com; mysql@lists.mysql.com > > just a few thoughts (things to look into) > > if you want to populate a db with command parameters, i'd mine the man > pages. it's a consistent format and you should be able to find a > parser for whatever language you prefer. > > if you want to see what has been entered and statistics about that > process, look at the audit framework. it won't tell you success or > failure, but you might be able to derive that based on the time, > memory, and process usage that you can get from audit. auditd also > maintains a db - you might look into the engine and schema they use. > > you might also find some interesting topics looking into language > processing (after all, commands are a type of language). people > generally use other database engines to do this type of thing - > elastic search or some other things people have done with lucene > maybe. > > do let us know where you go with this. it sounds quite interesting > (and might help me with something i've been looking into)... and > getting zsh to autocomplete options might be a bonus if you parse man > pages and shove it into a db (i might do that part for you if i get a > lazy afternoon). > > On Wed, Aug 22, 2012 at 7:51 PM, Garot Conklin <garotconk...@yahoo.com> wrote: > > I am going to work up a visual diagram for this to better... > > visualize (LOL) the lay of the land. I will forward it when i have > > it completed then I hope we can continue the architectural > > discussion here! Thanks > > > > -Garot > > > > > > garotconk...@yahoo.com > > > > > > ________________________________ > > From: Garot Conklin <garotconk...@yahoo.com> > > To: Garot Conklin <garotconk...@yahoo.com>; "webmas...@lisol.co.uk" > ><webmas...@lisol.co.uk>; 'Martin Gainty' <mgai...@hotmail.com>; > >"mysql@lists.mysql.com" <mysql@lists.mysql.com> > > Sent: Tuesday, August 21, 2012 4:40 PM > > Subject: Re: Machine Learning > > > > to elaborate: > > > > I would almost rather qualify from the OS perspective what will work in > > advance, but take into consideration any failing condition. I may be > > getting a bit too specific at this juncture however as I am already apply > > this logic to issues I see in MY environment rather than total ambiguity, > > which is a best position here. > > > > I am ultimately expecting the learning algorithm to create new and > > excitingly effective ways of stringing commands together to solve issues, > > this being a notable aspect in and of itself, then creating some > > centralization based on what the machine determines as a best solution. > > > > Seems a relative "snapshot of the system" pre and post execution would be > > favorable as well so long as resources are not taxed as a result. This > > discussion is leading me to believe that this will be a multifaceted db > > solution without a doubt comprised of multiple layers of abstraction > > ultimately rolling up into a single master cluster for all the higher level > > analytics and internal testing cycles to then roll out the new primary > > protocols, which may themselves take place in a sub-instance of each > > possible scenario. > > > > I have also though of populating a db or multiple db's with actual text in > > the form of .pdf's of actual technical manuals... not certain the benefit > > of the written language abstraction there but the thought process > > extraction might be an interesting end indeed. > > > > > > > > garotconk...@yahoo.com > > > > > > ________________________________ > > From: Garot Conklin <garotconk...@yahoo.com> > > To: "webmas...@lisol.co.uk" <webmas...@lisol.co.uk>; 'Martin Gainty' > > <mgai...@hotmail.com>; "mysql@lists.mysql.com" > > <mysql@lists.mysql.com> > > Sent: Tuesday, August 21, 2012 4:09 PM > > Subject: Re: Machine Learning > > > > YES! > > > > > > garotconk...@yahoo.com > > > > > > ________________________________ > > From: "webmas...@lisol.co.uk" <webmas...@lisol.co.uk> > > To: 'Garot Conklin' <garotconk...@yahoo.com>; 'Martin Gainty' > > <mgai...@hotmail.com>; mysql@lists.mysql.com > > Sent: Tuesday, August 21, 2012 3:37 PM > > Subject: RE: Machine Learning > > > > > > Ah, > > > > Getting clearer and clearer. > > > > So these ‘nodes’ could ‘learn’ and ‘teach’ at the same time – right ? For > > instance, N1 runs a command in ‘domain’ D20 which it successful – it send > > information to node N20 that is the authority on domain D20 and N20 records > > it as success; N5 runs a command in domain D20 which goes wrong, and sends > > info to N20 and N20 records failure and sends a correct call to N5. Is this > > what you have in mind ? > > > > There are a lot of variables to be considered, for instance: > > 1) Command A version 1 can run very happily on operating system > > version 2, but fails in OS version 1 > > 2) Command A version 1 can run happily on OS version 2 on a 2G > > RAM, but fail on OS version 2 on 1G RAM > > 3) etc > > > > I think the DB design issues will become straightforward once the model is > > quite clear. > > > > Justin > > > > From:Garot Conklin [mailto:garotconk...@yahoo.com] > > Sent: 21 August 2012 17:14 > > To: webmas...@lisol.co.uk; 'Martin Gainty'; mysql@lists.mysql.com > > Subject: Re: Machine Learning > > > > 1)Refer to it for ‘knowledge’ (for instance, of what the latest > >version of a command is) > > It would most likely end up being "central" in this sense: > > A distributed collection of systems; i.e. (possible defined > >in roles) > > DB's > > FE's > > REPL's > > Cache's > > Each DB would have its own collection of remediation's that would > >then be indexed to populate a central db for trending/correlation etc... > > The "Collective" itself would function as a single conceptual > >implementation. A VIP for example might be associated with a specific role, > >say Web FE's and remediate only/all of them, but only have some relative > >access to the core db of say the Network from a Primary index perspective to > >make/draw associations/conclusions to issues at hand. > > > > > > 2)Send their ‘knowledge’ (for instance, of the latest command > > versions) to it for storage and distribution to others > > Rather than "latest command versions" I envision this to be more > >encapsulated as "latest successful invocation of the command string" and it > >inverse as well (to trend/metric-ize the failures thus lending to perpetual > >optimization). > > > > I like how this is fleshing out... This is helping me to define what I am > > really trying to accomplish. Thanks very much for everyone responding here, > > this is wonderful, please keep this going... > > > > garotconk...@yahoo.com > > > > ________________________________ > > > > From:"webmas...@lisol.co.uk" <webmas...@lisol.co.uk> > > To: 'Martin Gainty' <mgai...@hotmail.com>; garotconk...@yahoo.com; > > mysql@lists.mysql.com > > Sent: Tuesday, August 21, 2012 11:19 AM > > Subject: RE: Machine Learning > > > > Hi Garot, > > > > > > > > Ok, the concept is getting clearer, but let’s bring this down to > > earth a little bit more. I love DB design and problem-solving and am > > quite curious about this. > > > > > > > > Is the idea that you have a central computer (not HAL J) somewhere > > so that other computers can: > > > > 1) Refer to it for ‘knowledge’ (for instance, of what the > > latest version of a command is) > > > > 2) Send their ‘knowledge’ (for instance, of the latest command > > versions) to it for storage and distribution to others > > > > > > > > If this is the model, then the knowledge base can build up > > organically over time – I think. Or is this too simplistic ? > > > > > > > > Thanks, > > > > Justin > > > > > > > > From: Martin Gainty [mailto:mgai...@hotmail.com] > > Sent: 21 August 2012 00:25 > > To: garotconk...@yahoo.com; webmas...@lisol.co.uk; > > mysql@lists.mysql.com > > Subject: RE: Machine Learning > > > > > > > > When I hear 'AI' I always imagine theres a HAL 9001 behind the > > scenes that is running the show constantly admonishing its creator > > to "take another stress pill" > > > > Sounds like a fun project > > > > Keep us apprised, > > Martin Gainty > > ______________________________________________ > > Verzicht und Vertraulichkeitanmerkung/Note de déni et de > > confidentialité > > > > > > Diese Nachricht ist vertraulich. Sollten Sie nicht der vorgesehene > > Empfaenger sein, so bitten wir hoeflich um eine Mitteilung. Jede > > unbefugte Weiterleitung oder Fertigung einer Kopie ist unzulaessig. > > Diese Nachricht dient lediglich dem Austausch von Informationen und > > entfaltet keine rechtliche Bindungswirkung. Aufgrund der leichten > > Manipulierbarkeit von E-Mails koennen wir keine Haftung fuer den Inhalt > > uebernehmen. > > > > Ce message est confidentiel et peut être privilégié. Si vous n'êtes > > pas le destinataire prévu, nous te demandons avec bonté que pour > > satisfaire informez l'expéditeur. N'importe quelle diffusion non > > autorisée ou la copie de ceci est interdite. Ce message sert à > > l'information seulement et n'aura pas n'importe quel effet > > légalement obligatoire. Étant donné que les email peuvent facilement > > être sujets à la manipulation, nous ne pouvons accepter aucune > > responsabilité pour le contenu fourni. > > > > > > > > > > > > _____ > > > > Date: Mon, 20 Aug 2012 13:50:04 -0700 > > From: garotconk...@yahoo.com > > Subject: Re: Machine Learning > > To: webmas...@lisol.co.uk; mgai...@hotmail.com; > > mysql@lists.mysql.com > > > > Ya the idea is not anything new, but must be apparently quit > > difficult or not a priority as I have yet to find it already > > implemented anywhere... Far be it from me to not make some attempt here > > anyway... > > > > > > > > I am creating a fully automated framework from which a distributed > > infrastructure can be maintained. I have been writing automation > > scripts/code for some time now and the logical progression is to > > embark on a full concept of systems health auto remediation. I have > > numerous "monitoring" solutions under my control however none that > > properly (in my > > opinion) implements any real learning algorithms from which to draw > > even a minimalists view of automation. I like mySQL therefor began > > thinking about creating the aspects (lobes) of the "brain" as a > > relational database(s). So this is only one facet of what I am > > trying to do, however leveraging a full command set of shell > > utilities/commands/programs seemed to be a good starting point before I get > > into the "hard" stuff ! > > > > > > > > -Garot > > > > > > > > garotconk...@yahoo.com > > > > _____ > > > > From: "webmas...@lisol.co.uk" <webmas...@lisol.co.uk> > > To: 'Martin Gainty' <mgai...@hotmail.com>; garotconk...@yahoo.com; > > mysql@lists.mysql.com > > Sent: Monday, August 20, 2012 3:55 PM > > Subject: RE: Machine Learning > > > > > > Hi Garot, > > > > > > > > You'll have to elaborate some more ... I understand you may want to > > protect the idea as well, so if you can narrow it down to some > > technical specifics then it'll help. > > > > > > > > What is the objective of this system, for instance ? > > > > > > > > Thanks, > > > > Justin > > > > > > > > From: Martin Gainty [mailto:mgai...@hotmail.com] > > Sent: 20 August 2012 19:23 > > To: garotconk...@yahoo.com; webmas...@lisol.co.uk; > > mysql@lists.mysql.com > > Subject: RE: Machine Learning > > > > > > > > > > > > From: garotconk...@yahoo.com > > Subject: Re: Machine Learning > > To: webmas...@lisol.co.uk; mgai...@hotmail.com; > > mysql@lists.mysql.com > > > > > > My initial thought was to propagate the db with everything and allow > > the algorithm to then begin to determin trends/patterns > > MG>which trends or patterns will you be modelling? > > > > and begin either an indexing methodology > > MG>which indexes are you considering: Unique index, primary index or > > MG>foreign > > index? > > > > additional table/db creation process or both to further optimize the > > calls being made > > MG>optimize based on execution time or diskspace allocated, > > MG>EliminatingFTS > > or some other criteria? > > MG>https://dev.mysql.com/doc/refman/5.5/en/optimization.html > > > > and build in some internal levels of redundancy. > > MG>what about replication > > MG>http://dev.mysql.com/doc/refman/5.5/en/replication.html > > > > I am actually approaching this with some degree of biological > > conception in the multipathing within our own brains however until I > > have something up and running under some substantial load however I > > may not get a complete picture. > > > > Thanks, > > > > Garot > > > > > > Interesting > > Martin > > > > _____ > > > > From: webmas...@lisol.co.uk <webmas...@lisol.co.uk>; > > To: 'Garot Conklin' <garotconk...@yahoo.com>; 'Martin Gainty' > > <mgai...@hotmail.com>; <mysql@lists.mysql.com>; > > Subject: RE: Machine Learning > > Sent: Mon, Aug 20, 2012 7:13:25 AM > > > > > > Hi Garot, > > > > This sounds an interesting idea. > > > > Are you looking to store all known commands and their options or are > > you looking for a 'formula' for calling any unix command ? > > > > The reason for my question is that, at the end of the day, a unix > > command is just a program that is run in the operating system. Each > > program comes with its own options and acceptable inputs. I don't > > know if there is a rule or convention for structuring these commands. > > > > Are you then looking to build a system that 'knows' all commands and > > 'how to' call them ? > > > > Thanks, > > Justin > > > > -----Original Message----- > > From: Garot Conklin [mailto:garotconk...@yahoo.com] > > Sent: 20 August 2012 03:39 > > To: Martin Gainty; mysql@lists.mysql.com > > Subject: Re: Machine Learning > > > > The initial goal is to provide a working framework from which to > > call all UNIX shell command combinations as the underlying storage > > mechanism for a machine learning algorithm. I would like to build a > > completely self aware instantiation that will maintain itself on all > > levels... I postulate that the first place to start would be in > > determining a method for maintaining all possible remediation > > combinations including the unknown to eventually be learned from and > > populate new knowledge into the database. Thank you for the reply, > > > > Garot > > -- > MySQL General Mailing List > For list archives: http://lists.mysql.com/mysql To unsubscribe: > http://lists.mysql.com/mysql > -- MySQL General Mailing List For list archives: http://lists.mysql.com/mysql To unsubscribe: http://lists.mysql.com/mysql
