Hi Ajay, Thanks for following this up. Appreciate if you could start a POC (proof-of-concept) with your ideas and enhance that over time. PLMK if you need any help in understanding the APIs in falcon. Feel free to create issues in Jira if you find an issue.
Thanks and looking forward for your contributions. On Wed, Jul 16, 2014 at 12:13 PM, Ajay Yadav <[email protected]> wrote: > Hi , > > Thanks Venkatesh for such an encouraging response. It's absolutely > wonderful to see a community where new comers are provided such deep > support for contributing to the project. > > However, I am sorry that the discussion has digressed to python vs. java. I > should have been clearer with the way I phrased it. Let me try to rephrase > it > > *Dust.js alone will not be flexible enough(and insufficient in some cases) > to build future UI requirements for Admin/Falcon pipeline designer. We need > one more server side web framework (which will be delegating core falcon > tasks to falcon through REST API)* > > Again, *I am not advocating/discouraging any client side framework like > dust.js/bootstrap, just emphasising need of a new component.* > > If I try to explain it with an analogy, as per my modest knowledge Oozie > also exposes REST api but also has a server side webapp. > > I interpreted Venkatesh's original question as "we are already using > dust.js and bootstrap and why do we need more components in the tech > stack". I tried explaining the need/advantages for a new middle level > component - a server side framework sitting between clients and REST APIs > vs. using a client side framework like dust.js on top of REST Api. If you > read my original email with this context, it will help make things clearer. > > Again to help clear Haohui's question, core tasks will still be delegated > to REST layer which already exists and will be the only place for such > tasks. However, tasks like user sessions etc. will be done and managed by > the new component. This layer will manage tasks only specific to Web UI > like user sessions etc. > > With new and ever evolving scenarios like pipeline designer, It will be > lean since it will be delegating most of tasks to the web server itself but > at the same time it gives immense flexibility to cater any possible use > case/extension through plugins in future. > > > *P.S.* Using Django won't cause change in architecture of current > deployment. It will be separate deployment. Django comes with a built in > server to aid development and though it is not recommended for production > set up, it works quite well for 50-80 users. For production level set up, > It is very flexible in terms of deployment and you can use nginx, gunicorn, > apache etc. > > > > > > On Tue, Jul 15, 2014 at 5:58 PM, Srikanth Sundarrajan <[email protected] > > > wrote: > > > Perhaps we can settle on JS + any standard MVC framework in the backend > if > > deemed necessary. > > > > Regards > > Srikanth Sundarrajan > > > > ---------------------------------------- > > > Date: Tue, 15 Jul 2014 17:25:30 +0530 > > > Subject: Re: Admin Dashboard > > > From: [email protected] > > > To: [email protected] > > > > > > Hi Ajay, > > > > > > Thanks for taking the initiative and sorry for my delayed response. I > > think > > > I largely concur with Haohui that keeping the stack lean and using JS > for > > > the front end and Java for the backend is desirable. > > > > > > Let me know what you need from serverside perspective and one of us can > > > jump in and help. Since the requirements are fairly trivial and reflect > > the > > > state in falcon, we should be able to achieve a lean implementation. > > > > > > Today Falcon does not need to run python and embeds a jetty container. > We > > > may need to change the deployment architecture to accommodate python. > > > > > > Lets take a stab at using any front end tech that you are aware of and > > > proceed. I'm not hung up on dust nor bootstrap. > > > > > > Thanks! > > > > > > > > > On Tue, Jul 15, 2014 at 3:32 AM, Haohui Mai <[email protected]> > > wrote: > > > > > >> bq. Try implementing a > > >> very basic requirement: user sessions( a logged in user doesn't need > to > > >> login again on every page) with existing stack and you will understand > > what > > >> I am trying to say. > > >> > > >> Note that even simple features like user sessions in Falcon will be > > coupled > > >> with other components in the Hadoop ecosystems. In secure set ups, > > Falcon > > >> will need to get delegation tokens from the HDFS NameNode and perform > > >> UGI-related operations. I don't see how Django will help in this case. > > >> > > >> bq. P.S. On a side note, you can use python to interact with Hadoop, > > Hive, > > >> Pig, > > >> Oozie. Again, I am not suggesting to use python instead of java in > > existing > > >> components. > > >> > > >> If this is the case, can you elaborate on how Python+Django can > benefit > > >> Falcon, and what exact requirements need to be solved? > > >> > > >> > > >> > > >> > > >> On Sun, Jul 13, 2014 at 9:26 PM, Ajay Yadav <[email protected]> > wrote: > > >> > > >>> Hi Haohui, > > >>> > > >>> Thanks for the reply. I am not suggesting to change existing java > code > > to > > >>> python or saying that we can live without javascript. All I am > > suggesting > > >>> is that we do need to use a server side web framework. Try > > implementing a > > >>> very basic requirement: user sessions( a logged in user doesn't need > to > > >>> login again on every page) with existing stack and you will > understand > > >> what > > >>> I am trying to say. > > >>> > > >>> > > >>> P.S. On a side note, you can use python to interact with Hadoop, > Hive, > > >> Pig, > > >>> Oozie. Again, I am not suggesting to use python instead of java in > > >> existing > > >>> components. > > >>> > > >>> > > >>> On Sun, Jul 13, 2014 at 11:40 PM, Haohui Mai <[email protected]> > > >> wrote: > > >>> > > >>>> Thanks for your interests in contributing in Falcon. I am one of the > > >>>> early contributors in Falcon UI, here are my two cents: > > >>>> > > >>>> Falcon has a browser-server architecture which needs to address two > > >>>> requirements: > > >>>> > > >>>> 1. Provide a feature-rich UI (e.g. FALCON-290) to the browser. > > >>>> > > >>>> 2. The server-side logics have to integrate with existing Hadoop > > >>>> ecosystems (e.g., MapReduce, Hive, Pig, and Oozie) which are heavily > > >>>> based on Java. > > >>>> > > >>>> While personally I'm pretty open to any other technologies, but I am > > >>>> yet to be convinced that how Python+Django will add any values to > > >>>> address the above requirements. More concretely: > > >>>> > > >>>> i. JavaScript is (and will continue to be) an essential part of the > UI > > >>>> to meet requirement (1). The python stack cannot replace the > > >>>> JavaScript part on the client side. > > >>>> > > >>>> ii. Falcon has to rely on Java to interact with other projects in > the > > >>>> Hadoop ecosystems, as there are few bindings in other languages for > > >>>> these projects. > > >>>> > > >>>> > > >>>> Using Python+Django might help attract attentions from the python > > >>>> community, but I think that from the Falcon points of view, it is > much > > >>>> more important to keep align with the whole Hadoop ecosystems (which > > >> are > > >>>> the main audiences of the Falcon project). Moreover, I don't think > > that > > >>> the > > >>>> merits that you mentioned for server-side technologies have to be > > >> coupled > > >>>> with Python+Django -- in fact, there are many more implementation of > > >>>> enterprise > > >>>> features like exporting to Excel in Java. > > >>>> > > >>>> In short, I think that the current choice of Java+REST+JavaScript is > > >>> quite > > >>>> reasonable. Though I'm pretty open to any other technologies, I'm > yet > > >> to > > >>> be > > >>>> convinced that Python+Django can be a superior choice. > > >>>> > > >>>> > > >>>> Thanks, > > >>>> Haohui > > >>>> > > >>>> > > >>>> On Sat, Jul 12, 2014 at 7:06 PM, Ajay Yadav <[email protected]> > > >> wrote: > > >>>> > > >>>>> Any comments? I am awaiting your decision to start working on this. > > >>>>> > > >>>>> > > >>>>> On Fri, Jul 11, 2014 at 11:10 AM, Ajay Yadav <[email protected]> > > >>> wrote: > > >>>>> > > >>>>>> Hi Venkatesh, > > >>>>>> > > >>>>>> Valid question and I am glad that you gave an opportunity to > > >> discuss > > >>>> it. > > >>>>> I > > >>>>>> have no particular bias towards using django. I have worked with > > >>> client > > >>>>>> side templates(not dust.js but more popular alternatives like > > >>> mustache, > > >>>>>> handlebars etc.) and client side MV* frameworks like backbone.js, > > >>>>> angularjs > > >>>>>> etc. It is completely possible to use both together, most common > > >>> case. > > >>>>>> > > >>>>>> *Note: *I looked at the existing code and saw that the current > work > > >>> in > > >>>>>> dust.js is very less - around15-25 lines(only dust templates part > > >> not > > >>>>>> html/css). So may I take the liberty of rephrasing your question > > >>>> slightly > > >>>>>> as below > > >>>>>> > > >>>>>> *Question*: Assuming we can choose only one to keep stack mix to > > >>>> minimum > > >>>>>> - shall we use pure client side framework(like dust.js) or pure > > >>> server > > >>>>>> side framework(like django)? > > >>>>>> > > >>>>>> *Answer*: > > >>>>>> *TLDR Version:* > > >>>>>> Server side components offer more flexibility and extensibility > > >> than > > >>> a > > >>>>>> pure client side framework. Using only client side stack will rule > > >>> out > > >>>>> use > > >>>>>> of database for dashboards, talking to other services,plugins etc. > > >> by > > >>>>> using > > >>>>>> just dust.js. Things like hardcoding of rest server end points, > > >>>> graceful > > >>>>>> degradation, number of parallel connections available in browser > > >> will > > >>>>> limit > > >>>>>> pure client side stacks. In other cases it will put a stress on > > >>> falcon > > >>>>> REST > > >>>>>> API to be modified just for the dashboard. Using python stack > > >> instead > > >>>> of > > >>>>>> javascript, will help in getting more support from community. > > >>>>>> > > >>>>>> *Long Version* > > >>>>>> > > >>>>>> - *Convenience vs. Possibility* - Server side components give us > > >>>>>> flexibility and extensibility to build for any use case in > > >> future > > >>> as > > >>>>> well. > > >>>>>> Any future feature is possible to be built without dust.js and > > >>>>> bootstrap > > >>>>>> but not other way around. Dust.js and bootstrap just make it > > >>> easier > > >>>>> to do > > >>>>>> lot of things to enhance user experience. > > >>>>>> - *Features* - Consider the login page of the dashboard. We will > > >>>> need > > >>>>>> a database to store users/ integrate with ldap. We will need to > > >>>>> generate > > >>>>>> new random passwords when users forget passwords. Same for > > >>> features > > >>>>> like > > >>>>>> bookmarks, custom dashboards etc. A lot of features like this > > >>> can't > > >>>>> be done > > >>>>>> if we are not using any server side component. > > >>>>>> - *Sanity of REST API* - REST services of falcon are not only > > >> for > > >>>> web > > >>>>>> browser consumption. They can be used by command line clients > > >> etc. > > >>>> as > > >>>>> well. > > >>>>>> As the use cases for dashboard will evolve, it will become more > > >>>>> involved to > > >>>>>> do things using js alone on client side and it will put lot of > > >>>> stress > > >>>>> on > > >>>>>> modifying REST API to suit just the client. For example consider > > >>> the > > >>>>> use > > >>>>>> case of allowing an administrator to edit a process from an HTML > > >>>>> form. REST > > >>>>>> API will return plain text/xml and it will be tedious and slow > > >> to > > >>>>> parse > > >>>>>> everything in javascript side and then recreate it. Sending a > > >> json > > >>>> and > > >>>>>> accepting json just to cater to web client will be problem for > > >>>>>> - Another use case can be to be able to export the reports like > > >>> SLAs > > >>>>>> missed in last quarter, metrics in pdf/excel format and it will > > >>> not > > >>>> be > > >>>>>> possible to do it on client side and in lack of a server side > > >>>>> component it > > >>>>>> will again put stress on REST API to be modified just for > > >>> dashboard > > >>>>> UI. > > >>>>>> - *Best Practices* - With just dust.js and simple htmls being > > >>> served > > >>>>>> through a webserver, we will have to deal with naming urls with > > >>>> .html > > >>>>>> extensions, hardcoding the REST end points in each HTML etc. > > >>>>>> - *Browser constraints* - We will have to deal with browser > > >>>>>> constraints like number of parallel connections, cross site > > >>>> requests, > > >>>>>> javascript not being enabled etc. This will also restrict the > > >>>>> development. > > >>>>>> > > >>>>>> > > >>>>>> - *Community Support* - This is purely my impression and I might > > >>> be > > >>>>>> completely wrong here. I see that more python developers are > > >>>> involved > > >>>>> with > > >>>>>> big data/data science/machine learning eco system than more > > >>>> javascript > > >>>>>> developers. Hence it will be easier to find more support for > > >>>>> maintaining > > >>>>>> and extending these components in community if we develop them > > >> in > > >>>>> django > > >>>>>> instead of dust.js. > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> > > >>>>>> On Fri, Jul 11, 2014 at 2:28 AM, Seetharam Venkatesh < > > >>>>>> [email protected]> wrote: > > >>>>>> > > >>>>>>> I think we are using bootstrap and dust templates. I'd want to > see > > >>> if > > >>>> we > > >>>>>>> need to introduce more technologies into the mix. Do we need > > >> Python > > >>>> and > > >>>>>>> Django? > > >>>>>>> > > >>>>>>> > > >>>>>>> On Thu, Jul 10, 2014 at 6:08 PM, Ajay Yadav <[email protected]> > > >>>> wrote: > > >>>>>>> > > >>>>>>>> Thanks Srikanth and Venkatesh. > > >>>>>>>> > > >>>>>>>> I had looked at the UI and the graph database related code and I > > >>>> will > > >>>>>>> reuse > > >>>>>>>> existing code where ever available. I checked for compatibility > > >> of > > >>>>> each > > >>>>>>> of > > >>>>>>>> the software being used and they are all compatible with APL 2.0 > > >>>>>>> (Python, > > >>>>>>>> Django & Bootstrap). > > >>>>>>>> > > >>>>>>>> If you can assign the task to me on JIRA also, it will be > > >>>>>>>> great(user:ajayyadava) Also, I got the balsamiq account for > > >>>>>>> falcon(thanks > > >>>>>>>> Srikant) and I will be asking all my questions on mocks there. > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> > > >>>>>>>> On Thu, Jul 10, 2014 at 2:45 PM, Seetharam Venkatesh < > > >>>>>>>> [email protected]> wrote: > > >>>>>>>> > > >>>>>>>>> Welcome aboard Ajay. Please let us know how we can help you to > > >>> get > > >>>>> up > > >>>>>>> and > > >>>>>>>>> running on this. The REST APIs are documented at: > > >>>>>>>>> > > >>> http://falcon.incubator.apache.org/docs/restapi/ResourceList.html > > >>>>>>>>> > > >>>>>>>>> Thanks! > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> On Thu, Jul 10, 2014 at 7:12 AM, Ajay Yadav < > > >> [email protected] > > >>>> > > >>>>>>> wrote: > > >>>>>>>>> > > >>>>>>>>>> Hi, > > >>>>>>>>>> > > >>>>>>>>>> I would like to volunteer for > > >>>>>>>>>> https://issues.apache.org/jira/browse/FALCON-37 > > >>>>>>>>>> > > >>>>>>>>>> I have more than 7 years of experience in building > > >> websites. I > > >>>>> have > > >>>>>>>>>> experience in both client side > > >>>>>>> technologies(javascript/HTML5/Bootstrap > > >>>>>>>>>> etc.) and server side technologies(Python/Django). > > >>>>>>>>>> > > >>>>>>>>>> I will need some help with iterating on the mocks as they > > >>> don't > > >>>>>>> seem to > > >>>>>>>>> be > > >>>>>>>>>> complete. I spent couple of hours on building a couple of > > >>> pages > > >>>>>>> using > > >>>>>>>>>> "dummy" data to get an idea of what all APIs are required. > > >>>> Please > > >>>>>>> find > > >>>>>>>>> them > > >>>>>>>>>> attached. They are just a quick and dirty prototype and UI > > >> is > > >>>>> pretty > > >>>>>>>> raw > > >>>>>>>>> as > > >>>>>>>>>> of now, to get quick feedback. I used Django,HTML5 & > > >>> Bootstrap. > > >>>>>>>>>> > > >>>>>>>>>> Please let me know if I can own this task. > > >>>>>>>>>> > > >>>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> > > >>>>>>>>> -- > > >>>>>>>>> Regards, > > >>>>>>>>> Venkatesh > > >>>>>>>>> > > >>>>>>>>> “Perfection (in design) is achieved not when there is nothing > > >>> more > > >>>>> to > > >>>>>>>> add, > > >>>>>>>>> but rather when there is nothing more to take away.” > > >>>>>>>>> - Antoine de Saint-Exupéry > > >>>>>>>>> > > >>>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> > > >>>>>>> -- > > >>>>>>> Regards, > > >>>>>>> Venkatesh > > >>>>>>> > > >>>>>>> “Perfection (in design) is achieved not when there is nothing > more > > >>> to > > >>>>> add, > > >>>>>>> but rather when there is nothing more to take away.” > > >>>>>>> - Antoine de Saint-Exupéry > > >>>>>>> > > >>>>>> > > >>>>>> > > >>>>> > > >>>> > > >>>> -- > > >>>> CONFIDENTIALITY NOTICE > > >>>> NOTICE: This message is intended for the use of the individual or > > >> entity > > >>> to > > >>>> which it is addressed and may contain information that is > > confidential, > > >>>> privileged and exempt from disclosure under applicable law. If the > > >> reader > > >>>> of this message is not the intended recipient, you are hereby > notified > > >>> that > > >>>> any printing, copying, dissemination, distribution, disclosure or > > >>>> forwarding of this communication is strictly prohibited. If you have > > >>>> received this communication in error, please contact the sender > > >>> immediately > > >>>> and delete it from your system. Thank You. > > >>>> > > >>> > > >> > > >> -- > > >> CONFIDENTIALITY NOTICE > > >> NOTICE: This message is intended for the use of the individual or > > entity to > > >> which it is addressed and may contain information that is > confidential, > > >> privileged and exempt from disclosure under applicable law. If the > > reader > > >> of this message is not the intended recipient, you are hereby notified > > that > > >> any printing, copying, dissemination, distribution, disclosure or > > >> forwarding of this communication is strictly prohibited. If you have > > >> received this communication in error, please contact the sender > > immediately > > >> and delete it from your system. Thank You. > > >> > > > > > > > > > > > > -- > > > Regards, > > > Venkatesh > > > > > > “Perfection (in design) is achieved not when there is nothing more to > > add, > > > but rather when there is nothing more to take away.” > > > - Antoine de Saint-Exupéry > > > > > -- Regards, Venkatesh “Perfection (in design) is achieved not when there is nothing more to add, but rather when there is nothing more to take away.” - Antoine de Saint-Exupéry
