I also agree on using Zookeeper. I have re-worked dynamic UDF support document taking into account Zookeeper usage.
Link to the document - https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit Kind regards Arina On Tue, Jun 28, 2016 at 12:55 AM Paul Rogers <prog...@maprtech.com> wrote: > Great idea! We already use ZK to track storage plugins. ZK is perhaps > better suited to register each jar and/or function that using files in DFS. > Still need to work out the proper sequencing. But you are right, this is > the kind of thing that ZK is supposed to solve. > > - Paul > > > > On Jun 27, 2016, at 2:01 PM, Parth Chandra <par...@apache.org> wrote: > > > > Reading thru some of Paul's comments on maintaining a consistent state > for > > the registration of the UDF, it looks like we need a consensus protocol > for > > determining that all the Drillbits have the UDF deployed. > > I believe Zookeeper can provide a stronger guarantee than a 2 phase > > approach. Should we look into that? > > > > On Fri, Jun 24, 2016 at 10:00 AM, Arina Yelchiyeva < > > arina.yelchiy...@gmail.com> wrote: > > > >> Hi all! > >> > >> I have updated design document. > >> Main changes: > >> 1. Add to Drill’s config цшер the staging and registration DFS > locations. > >> 2. User is no longer is responsible for copying jars into drillbit > nodes. > >> Now user needs to copy jars into staging DFS location from where > drillbits > >> will copy them to local fs. > >> 2. During UDFs registration jars will be moved to DFS registration area. > >> 3. During start up drillbit will copy all jars from registration area, > so > >> newly added drillbit will have all UDFs as others. > >> 4. Security issues - probably they will be added later as enhancement. > >> > >> More detains in the document: > >> > >> > https://docs.google.com/document/d/1MluM17EKajvNP_x8U4aymcOihhUm8BMm8t_hM0jEFWk/edit > >> > >> Kind regards > >> Arina > >> > >> On Fri, Jun 17, 2016 at 1:25 AM Paul Rogers <prog...@maprtech.com> > wrote: > >> > >>> Hi All, > >>> > >>> To answer Arina on item 3: there is actually no good location on any > >> local > >>> node to put the UDFs. Reason: DoY allows the admin to start a Drillbit > on > >>> any available node. When it starts, a new, fresh copy of Drill will be > >>> downloaded, and this can happen after the user issued the CREATE > command. > >>> > >>> What we need is a shared, secure distributed storage location from > which > >>> Drillbits can download the needed jar files. Something like… DFS! > Indeed, > >>> this is how YARN stores the Drill archive from which it creates the > Drill > >>> install directory on each node. We can’t quite use YARN’s mechanism > (YARN > >>> is aware only of the files uploaded when launching an app), but we can > do > >>> something similar. > >>> > >>> So, brainstorming a bit… > >>> > >>> 1. Store the UDF jar in a pre-defined DFS location. > >>> > >>> 2. The CREATE function 1) uploads the jar to the DFS location, and 2) > >>> creates some kind of registry entry. > >>> > >>> 3. The DELETE function 1) deregisters the jar (and function), but 2) > does > >>> not delete the jar (this allows in-flight queries to complete.) > >>> > >>> 3. Drillbits periodically check DFS for changed registrations, > >> downloading > >>> any needed jars. (YARN, Spark, Storm and others already do something > >>> similar.) > >>> > >>> 4. Registry check is “forced” when processing a query with a function > >> that > >>> is not currently registered. (Doing so resolves any possible race > >>> conditions.) > >>> > >>> 5. Some process (perhaps time based) removes old, unregistered jar > files. > >>> (Or, we could get fancy and use reference counts. The reference count > >> would > >>> be required if the user wants to delete, then recreate, the same > function > >>> and jar to avoid conflict with in-flight queries.) > >>> > >>> We can build security on this as follows: > >>> > >>> 1. Define permissions for who can write to the DFS location. Or, > indeed, > >>> have subdirectories by user and grant each user permission only on > their > >>> own UDF directory. > >>> > >>> 2. Provide separate registries for per-user functions (private) and > >> global > >>> functions (public). Only the admin can add global functions. But, only > >> the > >>> user that uploads a private function can use it. > >>> > >>> 3. Leverage the Java class loader to isolate UDFs in their own name > space > >>> (see Eclipse & Tomcat for examples). That is, Drill can call into a > UDF, > >>> UDFs can call selected Drill code, but UDFs can’t shadow Drill classes > >>> (accidentally or maliciously.) Plus, my function Foo won’t clash with > >> your > >>> function Foo if both are private. > >>> > >>> Sorry that this has wandered a bit far from the original simple design, > >>> but the above may capture much of what folks expect in modern > distributed > >>> big data systems. > >>> > >>> I wonder if a good next step might be to review the notes in the design > >>> doc, in the JIRA, and in this e-mail chain and to prepare a summary of > >>> technical requirements, and a proposed design. Postpone, at least for > >> now, > >>> concerns about the amount of work; we can worry about that once folks > >> agree > >>> on your revised design. > >>> > >>> Thanks, > >>> > >>> - Paul > >>> > >>> > >>>> On Jun 21, 2016, at 9:48 AM, Arina Yelchiyeva < > >>> arina.yelchiy...@gmail.com> wrote: > >>>> > >>>> 4. Authorization model mentioned by Julia and John > >>>> If user won't have rights to copy jars to UDF classpath, which can be > >>>> restricted by file system, he won't be able to do much harm by running > >>>> CREATE command. If UDFs from jar were already registered, CREATE > >>> statement > >>>> will fail. CREATE OR REPLACE will just re-register UDFs. > >>>> But DELETE command is not safe. If user knows jar name, he can delete > >> all > >>>> associated with it UDFs, as well as the binary and source jars. That's > >>>> where we'll probably need to impose restrictions. > >>>> > >>>> On Tue, Jun 21, 2016 at 7:34 PM Arina Yelchiyeva < > >>> arina.yelchiy...@gmail.com> > >>>> wrote: > >>>> > >>>>> 1. DELETE command - I missed to indicate it document but had it in my > >>>>> mind. When user issues DELETE command, all UDF associated with > >> indicated > >>>>> jar is removed from DrillFunctionRegistry. And then binary and source > >>>>> files are also deleted from UDF classpath. > >>>>> > >>>>> 2. Distribution race condition described by Paul > >>>>> User issues CREATE command and gets confirmation that UDFs is > >> registered > >>>>> only if all drilllbits have confirmed that registration was > >> successful. > >>>>> I don't expect user to start using UDFs in queries prior to CREATE > >>> command > >>>>> success / failure result, which is possible but strange. > >>>>> > >>>>> 3. DoY > >>>>> @Paul > >>>>> If instead of using $DRILL_HOME/jars/3rdparty/udf directly we use > >>>>> $DRILL_UDF environment variable which will be set during drillbit > >> start > >>>>> (like $DRILL_LOG_DIR). Location stored in this variable will be added > >> to > >>>>> Drill classpath during start. > >>>>> Will it ease DoY integration somehow? > >>>>> > >>>>> Kind regards > >>>>> Arina > >>>>> > >>>>> On Tue, Jun 21, 2016 at 7:15 PM yuliya Feldman > >>> <yufeld...@yahoo.com.invalid> > >>>>> wrote: > >>>>> > >>>>>> Just thoughts: > >>>>>> You can try to reuse distributed cache Let Drill AM do the needful > in > >>>>>> terms of orchestrating UDF jars distribution. > >>>>>> But > >>>>>> I would be inclined to have a common path that is independent of the > >>> fact > >>>>>> that it is Drill on YARN or not, as maintaining two separate ways of > >>>>>> dealing with loading/unloading UDFs will be painful and error prone. > >>>>>> One more note (I left a comment in the doc) - not sure about > >>>>>> authorization model here - we need to have some. > >>>>>> Just my 2cThanks > >>>>>> > >>>>>> From: Paul Rogers <prog...@maprtech.com> > >>>>>> To: "dev@drill.apache.org" <dev@drill.apache.org> > >>>>>> Sent: Monday, June 20, 2016 7:32 PM > >>>>>> Subject: Re: Dynamic UDFs support > >>>>>> > >>>>>> Hi Neeraja, > >>>>>> > >>>>>> The proposal calls for the user to copy the jar file to each > Drillbit > >>>>>> node. The jar would go into a new $DRILL_HOME/jars/3rdparty/udf > >>> directory. > >>>>>> > >>>>>> In Drill-on-YARN (DoY), YARN is responsible for copying Drill code > to > >>>>>> each node (which is good.) YARN puts that code in a location known > >>> only to > >>>>>> YARN. Since the location is private to YARN, the user can’t easily > >> hunt > >>>>>> down the location in order to add the udf jar. Even if the user did > >>> find > >>>>>> the location, the next Drillbit to start would create a new copy of > >> the > >>>>>> Drill software, without the udf jar. > >>>>>> > >>>>>> Second, in DoY we have separated user files from Drill software. > This > >>>>>> makes it much easier to distribute the software to each node: we > give > >>> the > >>>>>> Drill distribution tar archive to YARN, and YARN copies it to each > >>> node and > >>>>>> untars the Drill files. We make a separate copy of the (far smaller) > >>> set of > >>>>>> user config files. > >>>>>> > >>>>>> If the udf jar goes into a Drill folder > >>> ($DRILL_HOME/jars/3rdparty/udf), > >>>>>> then the user would have to rebuild the Drill tar file each time > they > >>> add a > >>>>>> udf jar. When I tried this myself when building DoY, I found it to > be > >>> slow > >>>>>> and error-prone. > >>>>>> > >>>>>> So, the solution is to place the udf code in the new “site” > >> directory: > >>>>>> $DRILL_SITE/jars. That’s what that is for. Then, let DoY > >> automatically > >>>>>> distribute the code to every node. Perfect! Except that it does not > >>> work to > >>>>>> dynamically distribute code after Drill starts. > >>>>>> > >>>>>> For DoY, the solution requirements are: > >>>>>> > >>>>>> 1. Distribute code using Drill itself, rather than manually copying > >>> jars > >>>>>> to (unknown) Drill directories. > >>>>>> 2. Ensure the solution works even if another Drillbit is spun up > >> later, > >>>>>> and uses the original Drill tar file. > >>>>>> > >>>>>> I’m thinking we want to leverage DFS: place udf files into a > >> well-known > >>>>>> DFS directory. Register the udf into, say, ZK. When a new Drillbit > >>> starts, > >>>>>> it looks for new udf jars in ZK, copies the file to a temporary > >>> location, > >>>>>> and launches. An existing Drill is notified of the change and does > >> the > >>> same > >>>>>> download process. Clean-up is needed at some point to remove ZK > >>> entries if > >>>>>> the udf jar becomes statically available on the next launch. That > >> needs > >>>>>> more thought. > >>>>>> > >>>>>> We’d still need the phases mentioned earlier to ensure consistency. > >>>>>> > >>>>>> Suggestions anyone as to how to do this super simply & still get it > >> to > >>>>>> work with DoY? > >>>>>> > >>>>>> Thanks, > >>>>>> > >>>>>> - Paul > >>>>>> > >>>>>>> On Jun 20, 2016, at 7:18 PM, Neeraja Rentachintala < > >>>>>> nrentachint...@maprtech.com> wrote: > >>>>>>> > >>>>>>> This will need to work with YARN (Once Drill is YARN enabled, I > >> would > >>>>>>> expect a lot of users using it in conjunction with YARN). > >>>>>>> Paul, I am not clear why this wouldn't work with YARN. Can you > >>>>>> elaborate. > >>>>>>> > >>>>>>> -Neeraja > >>>>>>> > >>>>>>> On Mon, Jun 20, 2016 at 7:01 PM, Paul Rogers <prog...@maprtech.com > > > >>>>>> wrote: > >>>>>>> > >>>>>>>> Good enough, as long as we document the limitation that this > >> feature > >>>>>> can’t > >>>>>>>> work with YARN deployment as users generally do not have access to > >>> the > >>>>>>>> temporary “localization” directories where the Drill code is > placed > >>> by > >>>>>> YARN. > >>>>>>>> > >>>>>>>> Note that the jar distribution race condition issue occurs with > the > >>>>>>>> proposed design: I believe I sketched out a scenario in one of the > >>>>>> earlier > >>>>>>>> comments. Drillbit A receives the CREATE FUNCTION command. It > tells > >>>>>>>> Drillbit B. While informing the other Drillbits, Drillbit B plans > >> and > >>>>>>>> launches a query that uses the function. Drillbit Z starts > >> execution > >>>>>> of the > >>>>>>>> query before it learns from A about the new function. This will be > >>>>>> rare — > >>>>>>>> just rare enough to create very hard to reproduce bugs. > >>>>>>>> > >>>>>>>> The only reliable solution is to do the work in multiple passes: > >>>>>>>> > >>>>>>>> Pass 1: Ask each node to load the function, but not make it > >> available > >>>>>> to > >>>>>>>> the planner. (it would be available to the execution engine.) > >>>>>>>> Pass 2: Await confirmation from each node that this is done. > >>>>>>>> Pass 3: Alert every node that it is now free to plan queries with > >> the > >>>>>>>> function. > >>>>>>>> > >>>>>>>> Finally, I wonder if we should design the SQL syntax based on a > >>>>>> long-term > >>>>>>>> design, even if the feature itself is a short-term work-around. > >>>>>> Changing > >>>>>>>> the syntax later might break scripts that users might write. > >>>>>>>> > >>>>>>>> So, the question for the group is this: is the value of > >> semi-complete > >>>>>>>> feature sufficient to justify the potential problems? > >>>>>>>> > >>>>>>>> - Paul > >>>>>>>> > >>>>>>>>> On Jun 20, 2016, at 6:15 PM, Parth Chandra < > pchan...@maprtech.com > >>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>> Moving discussion to dev. > >>>>>>>>> > >>>>>>>>> I believe the aim is to do a simple implementation without the > >>>>>> complexity > >>>>>>>>> of distributing the UDF. I think the document should make this > >>>>>> limitation > >>>>>>>>> clear. > >>>>>>>>> > >>>>>>>>> Per Paul's point on there being a simpler solution of just having > >>> each > >>>>>>>>> drillbit detect the if a UDF is present, I think the problem is > >> if a > >>>>>> UDF > >>>>>>>>> get's deployed to some but not all drillbits. A query can then > >> start > >>>>>>>>> executing but not run successfully. The intent of the create > >>> commands > >>>>>>>> would > >>>>>>>>> be to ensure that all drillbits have the UDF or none would. > >>>>>>>>> > >>>>>>>>> I think Jacques' point about ownership conflicts is not addressed > >>>>>>>> clearly. > >>>>>>>>> Also, the unloading is not clear. The delete command should > >> probably > >>>>>>>> remove > >>>>>>>>> the UDF and unload it. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> On Fri, Jun 17, 2016 at 11:19 AM, Paul Rogers < > >> prog...@maprtech.com > >>>> > >>>>>>>> wrote: > >>>>>>>>> > >>>>>>>>>> Reviewed the spec; many comments posted. Three primary comments > >> for > >>>>>> the > >>>>>>>>>> community to consider. > >>>>>>>>>> > >>>>>>>>>> 1. The design conflicts with the Drill-on-YARN project. Is this > a > >>>>>>>> specific > >>>>>>>>>> fix for one unique problem, or is it worth expanding the > solution > >>> to > >>>>>>>> work > >>>>>>>>>> with Drill-on-YARN deployments? Might be hard to make the two > >> work > >>>>>>>> together > >>>>>>>>>> later. See comments in docs for details. > >>>>>>>>>> > >>>>>>>>>> 2. Have we, by chance, looked at how other projects handle code > >>>>>>>>>> distribution? Spark, Storm and others automatically deploy code > >>>>>> across > >>>>>>>> the > >>>>>>>>>> cluster; no manual distribution to each node. The key difference > >>>>>> between > >>>>>>>>>> Drill and others is that, for Storm, say, code is associated > >> with a > >>>>>> job > >>>>>>>>>> (“topology” in Storm terms.) But, in Drill, functions are global > >>> and > >>>>>>>> have > >>>>>>>>>> no obvious life cycle that suggests when the code can be > >> unloaded. > >>>>>>>>>> > >>>>>>>>>> 3. Have considered the class loader, dependency and name space > >>>>>> isolation > >>>>>>>>>> issues addressed by such products as Tomcat (web apps) or > Eclipse > >>>>>>>>>> (plugins)? Putting user code in the same namespace as Drill code > >>> is > >>>>>>>> quick > >>>>>>>>>> & dirty. It turns out, however, that doing so leads to problems > >>> that > >>>>>>>>>> require long, frustrating debugging sessions to resolve. > >>>>>>>>>> > >>>>>>>>>> Addressing item 1 might expand scope a bit. Addressing items 2 > >> and > >>> 3 > >>>>>>>> are a > >>>>>>>>>> big increase in scope, so I won’t be surprised if we leave those > >>>>>> issues > >>>>>>>> for > >>>>>>>>>> later. (Though, addressing item 2 might be the best way to > >> address > >>>>>> item > >>>>>>>> 1.) > >>>>>>>>>> > >>>>>>>>>> If we want a very simple solution that requires minimal change, > >>>>>> perhaps > >>>>>>>> we > >>>>>>>>>> can use an even simpler solution. In the proposed design, the > >> user > >>>>>> still > >>>>>>>>>> must distribute code to all the nodes. The primary change is to > >>> tell > >>>>>>>> Drill > >>>>>>>>>> to load (or unload) that code. Can accomplish the same result > >>> easier > >>>>>>>> simply > >>>>>>>>>> by having Drill periodically scan certain directories looking > for > >>> new > >>>>>>>> (or > >>>>>>>>>> removed) jars? Still won’t work with YARN, or solve the name > >> space > >>>>>>>> issues, > >>>>>>>>>> but will work for existing non-YARN Drill users without new SQL > >>>>>> syntax. > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> > >>>>>>>>>> - Paul > >>>>>>>>>> > >>>>>>>>>>> On Jun 16, 2016, at 2:07 PM, Jacques Nadeau < > jacq...@dremio.com > >>> > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>> Two quick thoughts: > >>>>>>>>>>> > >>>>>>>>>>> - (user) In the design document I didn't see any discussion of > >>>>>>>>>>> ownership/conflicts or unloading. Would be helpful to see the > >>>>>> thinking > >>>>>>>>>> there > >>>>>>>>>>> - (dev) There is a row oriented facade via the > >>>>>>>>>>> FieldReader/FieldWriter/ComplexWriter classes. That would be a > >>> good > >>>>>>>> place > >>>>>>>>>>> to start when trying to implement an alternative interface. > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -- > >>>>>>>>>>> Jacques Nadeau > >>>>>>>>>>> CTO and Co-Founder, Dremio > >>>>>>>>>>> > >>>>>>>>>>> On Thu, Jun 16, 2016 at 11:32 AM, John Omernik < > >> j...@omernik.com> > >>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Honestly, I don't see it as a priority issue. I think some of > >> the > >>>>>>>> ideas > >>>>>>>>>>>> around community java UDFs could be a better approach. I'd > hate > >>> to > >>>>>>>> take > >>>>>>>>>>>> away from other work to hack in something like this. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> On Thu, Jun 16, 2016 at 1:19 PM, Paul Rogers < > >>> prog...@maprtech.com > >>>>>>> > >>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>>> Ted refers to source code transformation. Drill gains its > >> speed > >>>>>> from > >>>>>>>>>>>> value > >>>>>>>>>>>>> vectors. However, VVs are a far cry from the row-based > >> interface > >>>>>> that > >>>>>>>>>>>> most > >>>>>>>>>>>>> mere mortals are accustomed to using. Since VVs are very type > >>>>>>>> specific, > >>>>>>>>>>>>> code is typically generated to handle the specifics of each > >>> type. > >>>>>>>>>>>> Accessing > >>>>>>>>>>>>> VVs in Jython may be a bit of a challenge because of the > >>>>>> "impedence > >>>>>>>>>>>>> mismatch" between how VVs work and the row-and-column view > >>>>>> expected > >>>>>>>> by > >>>>>>>>>>>> most > >>>>>>>>>>>>> (non-Drill) developers. > >>>>>>>>>>>>> > >>>>>>>>>>>>> I wonder if we've considered providing a row-oriented > "facade" > >>>>>> that > >>>>>>>> can > >>>>>>>>>>>> be > >>>>>>>>>>>>> used by roll-your own data sources and user-defined row > >>>>>> transforms? > >>>>>>>>>> Might > >>>>>>>>>>>>> be a hiccup in the fast VV pipeline, but might be handy for > >>> users > >>>>>>>>>> willing > >>>>>>>>>>>>> to trade a bit of speed for convenience. With such a facade, > >> the > >>>>>>>> Jython > >>>>>>>>>>>> row > >>>>>>>>>>>>> transforms that John mentions could be quite simple. > >>>>>>>>>>>>> > >>>>>>>>>>>>> On Thu, Jun 16, 2016 at 10:36 AM, Ted Dunning < > >>>>>> ted.dunn...@gmail.com > >>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Since UDF's use source code transformation, using Jython > >> would > >>> be > >>>>>>>>>>>>>> difficult. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On Thu, Jun 16, 2016 at 9:42 AM, Arina Yelchiyeva < > >>>>>>>>>>>>>> arina.yelchiy...@gmail.com> wrote: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi Charles, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> not that I am aware of. Proposed solution doesn't invent > >>>>>> anything > >>>>>>>>>>>> new, > >>>>>>>>>>>>>> just > >>>>>>>>>>>>>>> adds possibility to add UDFs without drillbit restart. But > >>>>>>>>>>>>> contributions > >>>>>>>>>>>>>>> are welcomed. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> On Thu, Jun 16, 2016 at 4:52 PM Charles Givre < > >>> cgi...@gmail.com > >>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Arina, > >>>>>>>>>>>>>>>> Has there been any discussion about making it possible via > >>>>>> Jython > >>>>>>>>>>>> or > >>>>>>>>>>>>>>>> something for users to write simple UDFs in Python? > >>>>>>>>>>>>>>>> My ideal would be to have this capability integrated in > the > >>> web > >>>>>>>> GUI > >>>>>>>>>>>>>> such > >>>>>>>>>>>>>>>> that a user could write their UDF (in Python) right there, > >>>>>> submit > >>>>>>>>>>>> it > >>>>>>>>>>>>>> and > >>>>>>>>>>>>>>> it > >>>>>>>>>>>>>>>> would be deployed to Drill if it passes validation tests. > >>>>>>>>>>>>>>>> —C > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> On Jun 16, 2016, at 09:34, Arina Yelchiyeva < > >>>>>>>>>>>>>>> arina.yelchiy...@gmail.com> > >>>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi all! > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> I have created Jira to allow dynamic UDFs support in > >> Drill ( > >>>>>>>>>>>>>>>>> https://issues.apache.org/jira/browse/DRILL-4726). There > >>> is a > >>>>>>>>>>>> link > >>>>>>>>>>>>>> to > >>>>>>>>>>>>>>>>> design document in Jira description. > >>>>>>>>>>>>>>>>> Comments or suggestions are welcomed. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Kind regards > >>>>>>>>>>>>>>>>> Arina > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>> > >>>>> > >>> > >>> > >> > >
