Great idea! We already use ZK to track storage plugins. ZK is perhaps better suited to register each jar and/or function that using files in DFS. Still need to work out the proper sequencing. But you are right, this is the kind of thing that ZK is supposed to solve.
- Paul > On Jun 27, 2016, at 2:01 PM, Parth Chandra <par...@apache.org> wrote: > > Reading thru some of Paul's comments on maintaining a consistent state for > the registration of the UDF, it looks like we need a consensus protocol for > determining that all the Drillbits have the UDF deployed. > I believe Zookeeper can provide a stronger guarantee than a 2 phase > approach. Should we look into that? > > On Fri, Jun 24, 2016 at 10:00 AM, Arina Yelchiyeva < > arina.yelchiy...@gmail.com> wrote: > >> Hi all! >> >> I have updated design document. >> Main changes: >> 1. Add to Drill’s config цшер the staging and registration DFS locations. >> 2. User is no longer is responsible for copying jars into drillbit nodes. >> Now user needs to copy jars into staging DFS location from where drillbits >> will copy them to local fs. >> 2. During UDFs registration jars will be moved to DFS registration area. >> 3. During start up drillbit will copy all jars from registration area, so >> newly added drillbit will have all UDFs as others. >> 4. Security issues - probably they will be added later as enhancement. >> >> More detains in the document: >> >> https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit >> >> Kind regards >> Arina >> >> On Fri, Jun 17, 2016 at 1:25 AM Paul Rogers <prog...@maprtech.com> wrote: >> >>> Hi All, >>> >>> To answer Arina on item 3: there is actually no good location on any >> local >>> node to put the UDFs. Reason: DoY allows the admin to start a Drillbit on >>> any available node. When it starts, a new, fresh copy of Drill will be >>> downloaded, and this can happen after the user issued the CREATE command. >>> >>> What we need is a shared, secure distributed storage location from which >>> Drillbits can download the needed jar files. Something like… DFS! Indeed, >>> this is how YARN stores the Drill archive from which it creates the Drill >>> install directory on each node. We can’t quite use YARN’s mechanism (YARN >>> is aware only of the files uploaded when launching an app), but we can do >>> something similar. >>> >>> So, brainstorming a bit… >>> >>> 1. Store the UDF jar in a pre-defined DFS location. >>> >>> 2. The CREATE function 1) uploads the jar to the DFS location, and 2) >>> creates some kind of registry entry. >>> >>> 3. The DELETE function 1) deregisters the jar (and function), but 2) does >>> not delete the jar (this allows in-flight queries to complete.) >>> >>> 3. Drillbits periodically check DFS for changed registrations, >> downloading >>> any needed jars. (YARN, Spark, Storm and others already do something >>> similar.) >>> >>> 4. Registry check is “forced” when processing a query with a function >> that >>> is not currently registered. (Doing so resolves any possible race >>> conditions.) >>> >>> 5. Some process (perhaps time based) removes old, unregistered jar files. >>> (Or, we could get fancy and use reference counts. The reference count >> would >>> be required if the user wants to delete, then recreate, the same function >>> and jar to avoid conflict with in-flight queries.) >>> >>> We can build security on this as follows: >>> >>> 1. Define permissions for who can write to the DFS location. Or, indeed, >>> have subdirectories by user and grant each user permission only on their >>> own UDF directory. >>> >>> 2. Provide separate registries for per-user functions (private) and >> global >>> functions (public). Only the admin can add global functions. But, only >> the >>> user that uploads a private function can use it. >>> >>> 3. Leverage the Java class loader to isolate UDFs in their own name space >>> (see Eclipse & Tomcat for examples). That is, Drill can call into a UDF, >>> UDFs can call selected Drill code, but UDFs can’t shadow Drill classes >>> (accidentally or maliciously.) Plus, my function Foo won’t clash with >> your >>> function Foo if both are private. >>> >>> Sorry that this has wandered a bit far from the original simple design, >>> but the above may capture much of what folks expect in modern distributed >>> big data systems. >>> >>> I wonder if a good next step might be to review the notes in the design >>> doc, in the JIRA, and in this e-mail chain and to prepare a summary of >>> technical requirements, and a proposed design. Postpone, at least for >> now, >>> concerns about the amount of work; we can worry about that once folks >> agree >>> on your revised design. >>> >>> Thanks, >>> >>> - Paul >>> >>> >>>> On Jun 21, 2016, at 9:48 AM, Arina Yelchiyeva < >>> arina.yelchiy...@gmail.com> wrote: >>>> >>>> 4. Authorization model mentioned by Julia and John >>>> If user won't have rights to copy jars to UDF classpath, which can be >>>> restricted by file system, he won't be able to do much harm by running >>>> CREATE command. If UDFs from jar were already registered, CREATE >>> statement >>>> will fail. CREATE OR REPLACE will just re-register UDFs. >>>> But DELETE command is not safe. If user knows jar name, he can delete >> all >>>> associated with it UDFs, as well as the binary and source jars. That's >>>> where we'll probably need to impose restrictions. >>>> >>>> On Tue, Jun 21, 2016 at 7:34 PM Arina Yelchiyeva < >>> arina.yelchiy...@gmail.com> >>>> wrote: >>>> >>>>> 1. DELETE command - I missed to indicate it document but had it in my >>>>> mind. When user issues DELETE command, all UDF associated with >> indicated >>>>> jar is removed from DrillFunctionRegistry. And then binary and source >>>>> files are also deleted from UDF classpath. >>>>> >>>>> 2. Distribution race condition described by Paul >>>>> User issues CREATE command and gets confirmation that UDFs is >> registered >>>>> only if all drilllbits have confirmed that registration was >> successful. >>>>> I don't expect user to start using UDFs in queries prior to CREATE >>> command >>>>> success / failure result, which is possible but strange. >>>>> >>>>> 3. DoY >>>>> @Paul >>>>> If instead of using $DRILL_HOME/jars/3rdparty/udf directly we use >>>>> $DRILL_UDF environment variable which will be set during drillbit >> start >>>>> (like $DRILL_LOG_DIR). Location stored in this variable will be added >> to >>>>> Drill classpath during start. >>>>> Will it ease DoY integration somehow? >>>>> >>>>> Kind regards >>>>> Arina >>>>> >>>>> On Tue, Jun 21, 2016 at 7:15 PM yuliya Feldman >>> <yufeld...@yahoo.com.invalid> >>>>> wrote: >>>>> >>>>>> Just thoughts: >>>>>> You can try to reuse distributed cache Let Drill AM do the needful in >>>>>> terms of orchestrating UDF jars distribution. >>>>>> But >>>>>> I would be inclined to have a common path that is independent of the >>> fact >>>>>> that it is Drill on YARN or not, as maintaining two separate ways of >>>>>> dealing with loading/unloading UDFs will be painful and error prone. >>>>>> One more note (I left a comment in the doc) - not sure about >>>>>> authorization model here - we need to have some. >>>>>> Just my 2cThanks >>>>>> >>>>>> From: Paul Rogers <prog...@maprtech.com> >>>>>> To: "dev@drill.apache.org" <dev@drill.apache.org> >>>>>> Sent: Monday, June 20, 2016 7:32 PM >>>>>> Subject: Re: Dynamic UDFs support >>>>>> >>>>>> Hi Neeraja, >>>>>> >>>>>> The proposal calls for the user to copy the jar file to each Drillbit >>>>>> node. The jar would go into a new $DRILL_HOME/jars/3rdparty/udf >>> directory. >>>>>> >>>>>> In Drill-on-YARN (DoY), YARN is responsible for copying Drill code to >>>>>> each node (which is good.) YARN puts that code in a location known >>> only to >>>>>> YARN. Since the location is private to YARN, the user can’t easily >> hunt >>>>>> down the location in order to add the udf jar. Even if the user did >>> find >>>>>> the location, the next Drillbit to start would create a new copy of >> the >>>>>> Drill software, without the udf jar. >>>>>> >>>>>> Second, in DoY we have separated user files from Drill software. This >>>>>> makes it much easier to distribute the software to each node: we give >>> the >>>>>> Drill distribution tar archive to YARN, and YARN copies it to each >>> node and >>>>>> untars the Drill files. We make a separate copy of the (far smaller) >>> set of >>>>>> user config files. >>>>>> >>>>>> If the udf jar goes into a Drill folder >>> ($DRILL_HOME/jars/3rdparty/udf), >>>>>> then the user would have to rebuild the Drill tar file each time they >>> add a >>>>>> udf jar. When I tried this myself when building DoY, I found it to be >>> slow >>>>>> and error-prone. >>>>>> >>>>>> So, the solution is to place the udf code in the new “site” >> directory: >>>>>> $DRILL_SITE/jars. That’s what that is for. Then, let DoY >> automatically >>>>>> distribute the code to every node. Perfect! Except that it does not >>> work to >>>>>> dynamically distribute code after Drill starts. >>>>>> >>>>>> For DoY, the solution requirements are: >>>>>> >>>>>> 1. Distribute code using Drill itself, rather than manually copying >>> jars >>>>>> to (unknown) Drill directories. >>>>>> 2. Ensure the solution works even if another Drillbit is spun up >> later, >>>>>> and uses the original Drill tar file. >>>>>> >>>>>> I’m thinking we want to leverage DFS: place udf files into a >> well-known >>>>>> DFS directory. Register the udf into, say, ZK. When a new Drillbit >>> starts, >>>>>> it looks for new udf jars in ZK, copies the file to a temporary >>> location, >>>>>> and launches. An existing Drill is notified of the change and does >> the >>> same >>>>>> download process. Clean-up is needed at some point to remove ZK >>> entries if >>>>>> the udf jar becomes statically available on the next launch. That >> needs >>>>>> more thought. >>>>>> >>>>>> We’d still need the phases mentioned earlier to ensure consistency. >>>>>> >>>>>> Suggestions anyone as to how to do this super simply & still get it >> to >>>>>> work with DoY? >>>>>> >>>>>> Thanks, >>>>>> >>>>>> - Paul >>>>>> >>>>>>> On Jun 20, 2016, at 7:18 PM, Neeraja Rentachintala < >>>>>> nrentachint...@maprtech.com> wrote: >>>>>>> >>>>>>> This will need to work with YARN (Once Drill is YARN enabled, I >> would >>>>>>> expect a lot of users using it in conjunction with YARN). >>>>>>> Paul, I am not clear why this wouldn't work with YARN. Can you >>>>>> elaborate. >>>>>>> >>>>>>> -Neeraja >>>>>>> >>>>>>> On Mon, Jun 20, 2016 at 7:01 PM, Paul Rogers <prog...@maprtech.com> >>>>>> wrote: >>>>>>> >>>>>>>> Good enough, as long as we document the limitation that this >> feature >>>>>> can’t >>>>>>>> work with YARN deployment as users generally do not have access to >>> the >>>>>>>> temporary “localization” directories where the Drill code is placed >>> by >>>>>> YARN. >>>>>>>> >>>>>>>> Note that the jar distribution race condition issue occurs with the >>>>>>>> proposed design: I believe I sketched out a scenario in one of the >>>>>> earlier >>>>>>>> comments. Drillbit A receives the CREATE FUNCTION command. It tells >>>>>>>> Drillbit B. While informing the other Drillbits, Drillbit B plans >> and >>>>>>>> launches a query that uses the function. Drillbit Z starts >> execution >>>>>> of the >>>>>>>> query before it learns from A about the new function. This will be >>>>>> rare — >>>>>>>> just rare enough to create very hard to reproduce bugs. >>>>>>>> >>>>>>>> The only reliable solution is to do the work in multiple passes: >>>>>>>> >>>>>>>> Pass 1: Ask each node to load the function, but not make it >> available >>>>>> to >>>>>>>> the planner. (it would be available to the execution engine.) >>>>>>>> Pass 2: Await confirmation from each node that this is done. >>>>>>>> Pass 3: Alert every node that it is now free to plan queries with >> the >>>>>>>> function. >>>>>>>> >>>>>>>> Finally, I wonder if we should design the SQL syntax based on a >>>>>> long-term >>>>>>>> design, even if the feature itself is a short-term work-around. >>>>>> Changing >>>>>>>> the syntax later might break scripts that users might write. >>>>>>>> >>>>>>>> So, the question for the group is this: is the value of >> semi-complete >>>>>>>> feature sufficient to justify the potential problems? >>>>>>>> >>>>>>>> - Paul >>>>>>>> >>>>>>>>> On Jun 20, 2016, at 6:15 PM, Parth Chandra <pchan...@maprtech.com >>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Moving discussion to dev. >>>>>>>>> >>>>>>>>> I believe the aim is to do a simple implementation without the >>>>>> complexity >>>>>>>>> of distributing the UDF. I think the document should make this >>>>>> limitation >>>>>>>>> clear. >>>>>>>>> >>>>>>>>> Per Paul's point on there being a simpler solution of just having >>> each >>>>>>>>> drillbit detect the if a UDF is present, I think the problem is >> if a >>>>>> UDF >>>>>>>>> get's deployed to some but not all drillbits. A query can then >> start >>>>>>>>> executing but not run successfully. The intent of the create >>> commands >>>>>>>> would >>>>>>>>> be to ensure that all drillbits have the UDF or none would. >>>>>>>>> >>>>>>>>> I think Jacques' point about ownership conflicts is not addressed >>>>>>>> clearly. >>>>>>>>> Also, the unloading is not clear. The delete command should >> probably >>>>>>>> remove >>>>>>>>> the UDF and unload it. >>>>>>>>> >>>>>>>>> >>>>>>>>> On Fri, Jun 17, 2016 at 11:19 AM, Paul Rogers < >> prog...@maprtech.com >>>> >>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Reviewed the spec; many comments posted. Three primary comments >> for >>>>>> the >>>>>>>>>> community to consider. >>>>>>>>>> >>>>>>>>>> 1. The design conflicts with the Drill-on-YARN project. Is this a >>>>>>>> specific >>>>>>>>>> fix for one unique problem, or is it worth expanding the solution >>> to >>>>>>>> work >>>>>>>>>> with Drill-on-YARN deployments? Might be hard to make the two >> work >>>>>>>> together >>>>>>>>>> later. See comments in docs for details. >>>>>>>>>> >>>>>>>>>> 2. Have we, by chance, looked at how other projects handle code >>>>>>>>>> distribution? Spark, Storm and others automatically deploy code >>>>>> across >>>>>>>> the >>>>>>>>>> cluster; no manual distribution to each node. The key difference >>>>>> between >>>>>>>>>> Drill and others is that, for Storm, say, code is associated >> with a >>>>>> job >>>>>>>>>> (“topology” in Storm terms.) But, in Drill, functions are global >>> and >>>>>>>> have >>>>>>>>>> no obvious life cycle that suggests when the code can be >> unloaded. >>>>>>>>>> >>>>>>>>>> 3. Have considered the class loader, dependency and name space >>>>>> isolation >>>>>>>>>> issues addressed by such products as Tomcat (web apps) or Eclipse >>>>>>>>>> (plugins)? Putting user code in the same namespace as Drill code >>> is >>>>>>>> quick >>>>>>>>>> & dirty. It turns out, however, that doing so leads to problems >>> that >>>>>>>>>> require long, frustrating debugging sessions to resolve. >>>>>>>>>> >>>>>>>>>> Addressing item 1 might expand scope a bit. Addressing items 2 >> and >>> 3 >>>>>>>> are a >>>>>>>>>> big increase in scope, so I won’t be surprised if we leave those >>>>>> issues >>>>>>>> for >>>>>>>>>> later. (Though, addressing item 2 might be the best way to >> address >>>>>> item >>>>>>>> 1.) >>>>>>>>>> >>>>>>>>>> If we want a very simple solution that requires minimal change, >>>>>> perhaps >>>>>>>> we >>>>>>>>>> can use an even simpler solution. In the proposed design, the >> user >>>>>> still >>>>>>>>>> must distribute code to all the nodes. The primary change is to >>> tell >>>>>>>> Drill >>>>>>>>>> to load (or unload) that code. Can accomplish the same result >>> easier >>>>>>>> simply >>>>>>>>>> by having Drill periodically scan certain directories looking for >>> new >>>>>>>> (or >>>>>>>>>> removed) jars? Still won’t work with YARN, or solve the name >> space >>>>>>>> issues, >>>>>>>>>> but will work for existing non-YARN Drill users without new SQL >>>>>> syntax. >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> >>>>>>>>>> - Paul >>>>>>>>>> >>>>>>>>>>> On Jun 16, 2016, at 2:07 PM, Jacques Nadeau <jacq...@dremio.com >>> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Two quick thoughts: >>>>>>>>>>> >>>>>>>>>>> - (user) In the design document I didn't see any discussion of >>>>>>>>>>> ownership/conflicts or unloading. Would be helpful to see the >>>>>> thinking >>>>>>>>>> there >>>>>>>>>>> - (dev) There is a row oriented facade via the >>>>>>>>>>> FieldReader/FieldWriter/ComplexWriter classes. That would be a >>> good >>>>>>>> place >>>>>>>>>>> to start when trying to implement an alternative interface. >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Jacques Nadeau >>>>>>>>>>> CTO and Co-Founder, Dremio >>>>>>>>>>> >>>>>>>>>>> On Thu, Jun 16, 2016 at 11:32 AM, John Omernik < >> j...@omernik.com> >>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>>> Honestly, I don't see it as a priority issue. I think some of >> the >>>>>>>> ideas >>>>>>>>>>>> around community java UDFs could be a better approach. I'd hate >>> to >>>>>>>> take >>>>>>>>>>>> away from other work to hack in something like this. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> On Thu, Jun 16, 2016 at 1:19 PM, Paul Rogers < >>> prog...@maprtech.com >>>>>>> >>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>>> Ted refers to source code transformation. Drill gains its >> speed >>>>>> from >>>>>>>>>>>> value >>>>>>>>>>>>> vectors. However, VVs are a far cry from the row-based >> interface >>>>>> that >>>>>>>>>>>> most >>>>>>>>>>>>> mere mortals are accustomed to using. Since VVs are very type >>>>>>>> specific, >>>>>>>>>>>>> code is typically generated to handle the specifics of each >>> type. >>>>>>>>>>>> Accessing >>>>>>>>>>>>> VVs in Jython may be a bit of a challenge because of the >>>>>> "impedence >>>>>>>>>>>>> mismatch" between how VVs work and the row-and-column view >>>>>> expected >>>>>>>> by >>>>>>>>>>>> most >>>>>>>>>>>>> (non-Drill) developers. >>>>>>>>>>>>> >>>>>>>>>>>>> I wonder if we've considered providing a row-oriented "facade" >>>>>> that >>>>>>>> can >>>>>>>>>>>> be >>>>>>>>>>>>> used by roll-your own data sources and user-defined row >>>>>> transforms? >>>>>>>>>> Might >>>>>>>>>>>>> be a hiccup in the fast VV pipeline, but might be handy for >>> users >>>>>>>>>> willing >>>>>>>>>>>>> to trade a bit of speed for convenience. With such a facade, >> the >>>>>>>> Jython >>>>>>>>>>>> row >>>>>>>>>>>>> transforms that John mentions could be quite simple. >>>>>>>>>>>>> >>>>>>>>>>>>> On Thu, Jun 16, 2016 at 10:36 AM, Ted Dunning < >>>>>> ted.dunn...@gmail.com >>>>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>>> Since UDF's use source code transformation, using Jython >> would >>> be >>>>>>>>>>>>>> difficult. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Thu, Jun 16, 2016 at 9:42 AM, Arina Yelchiyeva < >>>>>>>>>>>>>> arina.yelchiy...@gmail.com> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi Charles, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> not that I am aware of. Proposed solution doesn't invent >>>>>> anything >>>>>>>>>>>> new, >>>>>>>>>>>>>> just >>>>>>>>>>>>>>> adds possibility to add UDFs without drillbit restart. But >>>>>>>>>>>>> contributions >>>>>>>>>>>>>>> are welcomed. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Thu, Jun 16, 2016 at 4:52 PM Charles Givre < >>> cgi...@gmail.com >>>>>>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Arina, >>>>>>>>>>>>>>>> Has there been any discussion about making it possible via >>>>>> Jython >>>>>>>>>>>> or >>>>>>>>>>>>>>>> something for users to write simple UDFs in Python? >>>>>>>>>>>>>>>> My ideal would be to have this capability integrated in the >>> web >>>>>>>> GUI >>>>>>>>>>>>>> such >>>>>>>>>>>>>>>> that a user could write their UDF (in Python) right there, >>>>>> submit >>>>>>>>>>>> it >>>>>>>>>>>>>> and >>>>>>>>>>>>>>> it >>>>>>>>>>>>>>>> would be deployed to Drill if it passes validation tests. >>>>>>>>>>>>>>>> —C >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Jun 16, 2016, at 09:34, Arina Yelchiyeva < >>>>>>>>>>>>>>> arina.yelchiy...@gmail.com> >>>>>>>>>>>>>>>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi all! >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I have created Jira to allow dynamic UDFs support in >> Drill ( >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/DRILL-4726). There >>> is a >>>>>>>>>>>> link >>>>>>>>>>>>>> to >>>>>>>>>>>>>>>>> design document in Jira description. >>>>>>>>>>>>>>>>> Comments or suggestions are welcomed. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Kind regards >>>>>>>>>>>>>>>>> Arina >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>> >>>>>>>> >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>> >>> >>
