Hi All, I've created following jiras encompassing the requirements (To be picked up in order): https://issues.apache.org/jira/browse/APEXMALHAR-2558 https://issues.apache.org/jira/browse/APEXMALHAR-2559
I'll start adding subtasks for subitems mentioned in the jira. I'll also create the feature branch shortly so that we can start working against it. -Chinmay. On Sat, Jun 16, 2018 at 7:07 PM Chinmay Kolhatkar <chin...@apache.org> wrote: > So here is how it'll be: > 1. ReactJS application can be compiled for production builds. The output > is set of html+css+compressed js files. It can also output a war file > directly (I think). > 2. The react app will have a configuration switch to allow user to set the > baseUrl. Default value will be "/ws/v2". > 3. These files can be bundled with stram so that it can be hosted by > stram. In this case the base url for reading data will be "/ws/v2".. i.e. > read from localhost. > 3. OR the config switch be set "http://path/to/stram/ws/v2"; using which > one can host it seperately. > > Basically, it can be reusable with configuration changes during/after > build time. > > Hope that helps. > > Thanks, > Chinmay. > > On Thu, Jun 14, 2018 at 10:09 PM Pramod Immaneni < > pramod.imman...@gmail.com> wrote: > >> I don't think they would be reusable as is without any changes externally >> as the application context would not be available. Also externally you >> would want to have a different kind of application that aggregates across >> multiple applications. >> >> Thanks >> >> On Thu, Jun 14, 2018 at 10:24 AM Vlad Rozov <vro...@apache.org> wrote: >> >> > If it is a set of static files that use existing REST API to display >> > information in a browser, where they will be hosted is irrelevant. They >> > can be hosted by stram, but it should be possible to host them >> > externally without making modificaitons. If this is the case, I don't >> > have any objections to the proposal. >> > >> > Thank you, >> > >> > Vlad >> > >> > On 6/13/18 06:09, Chinmay Kolhatkar wrote: >> > > Hi Team, >> > > >> > > I agree with Thomas. The main focus of this proposal was to provide a >> UI >> > > layer using existing REST APIs. >> > > I think we should first focus on providing a meaningful UI served by >> AM >> > to >> > > user using the same existing REST APIs. >> > > >> > > If in general, there is a agreement for the proposal, I can proceed >> with >> > > creating a JIRA and feature branch for the same. >> > > >> > > Thanks, >> > > Chinmay. >> > > >> > > >> > > On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <t...@apache.org> wrote: >> > > >> > >> Looks like there is some disconnect in this thread, a few replies >> > express >> > >> concerns that the proposal IMO doesn't imply. >> > >> >> > >> There is already a REST API that backs all of the CLI functionality. >> > Modern >> > >> UIs run in the browser, and so what was proposed (unless I >> > misunderstood) >> > >> would be based on the same REST API and nothing extra. The only >> > additional >> > >> thing the AM has to do is serve static files. And AFAIK it would >> > actually >> > >> simplify the UI implementation when files and REST API have the same >> > >> origin. >> > >> >> > >> To me where files are coming from is actually a smaller piece and >> > something >> > >> that can be augmented. But before proposing solutions outside of >> Apex or >> > >> additional deployment requirements, please consider the user >> > experience. It >> > >> should be really simple to setup and use the UI. >> > >> >> > >> I would expect that a large part of the discussion evolves around >> the UI >> > >> functionality that helps the target persona. >> > >> >> > >> Thanks, >> > >> Thomas >> > >> >> > >> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra < >> > >> sjmohapat...@gmail.com> >> > >> wrote: >> > >> >> > >>> Hi Community, >> > >>> >> > >>> I would like to propose a slightly different approach on having an >> UI >> > for >> > >>> Apex. >> > >>> >> > >>> The details are below: >> > >>> >> > >>> 1. This involves having an external web-server installed along >> > side of >> > >>> apexCli. We will basically mimic all the functionalities that >> > apexCli >> > >>> currently provides. >> > >>> 2. We will develop the necessary REST APIs for fetch/update >> > operations >> > >>> which apexCli currently supports. >> > >>> 3. Because most of the information and operations needed by a >> > typical >> > >>> user are already there in apexCli, it will of great help to end >> > users >> > >>> and >> > >>> which may increase the adoption rate. >> > >>> >> > >>> About Chinmay's proposal of having the STRAM hosting the apps, >> please >> > >> find >> > >>> my comments below: >> > >>> >> > >>> 1. It may increase load on STRAM if we were to provide real time >> > >> display >> > >>> of information. I am aware that apexCli fetches info from STRAM >> > apis >> > >> but >> > >>> continuous polling can cause extra over-head. >> > >>> 2. The UI will have limited scope for the particular >> application. >> > >> Users >> > >>> would definitely love to have a comprehensive UI for the >> > management of >> > >>> Apex >> > >>> Apps as a whole. >> > >>> 3. UI might be able to affect the whole application through >> STRAM >> > >> hence >> > >>> giving direct access to STRAM APIs might not be good idea. >> > >>> 4. Because STRAM can't have a comprehensive UI hosted inside, >> > anyways >> > >> we >> > >>> will have to develop an external one. >> > >>> >> > >>> So I would like propose the plan on action like this. >> > >>> >> > >>> 1. We develop the wrapper UI for apexCli in the first phase and >> > give >> > >>> user all the management options as REST APIs. >> > >>> 2. If some of information we require are not in the current >> apexCli >> > >>> implementation, we can add those feature to apexCli and if >> > required to >> > >>> STRAM APIs in subsequent phases. >> > >>> 3. In future we can even remove the REST server from STRAM if we >> > could >> > >>> provide something like MetricsStore functionality which was >> there >> > in >> > >>> DataTorrent RTS. >> > >>> >> > >>> -- >> > >>> Thanks and Regards, >> > >>> Shubhrajyoti Mohapatra >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <vro...@apache.org> >> wrote: >> > >>> >> > >>>> IMO it will be hard to provide UI that meets everyone expectation. >> An >> > >>>> external webUI can serve two goals >> > >>>> >> > >>>> 1. provide reference UI implementation >> > >>>> 2. ensure that REST API have all necessary entry points >> > >>>> >> > >>>> by embedding webUI directly into stram it will be tempting to >> shortcut >> > >>>> some calls to use stram classes directly. >> > >>>> >> > >>>> Thank you, >> > >>>> >> > >>>> Vlad >> > >>>> >> > >>>> On 6/11/18 12:18, Pramod Immaneni wrote: >> > >>>>> I too think that a comprehensive UI works better standalone but >> given >> > >>>> that >> > >>>>> stram already runs a webapp server, provides web services and RM >> > >>>> provides a >> > >>>>> proxy with mapped url space for it, we can extend stram to provide >> > >>> better >> > >>>>> visual output. Overall I am +1 on the idea. >> > >>>>> >> > >>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <vro...@apache.org> >> > >> wrote: >> > >>>>>> Clarification: I am -0 on making UI part of the stram. I am +1 on >> > >>>>>> providing reference webUI application outside of the stram. >> > >>>>>> >> > >>>>>> Thank you, >> > >>>>>> >> > >>>>>> Vlad >> > >>>>>> >> > >>>>>> On 6/9/18 07:05, Vlad Rozov wrote: >> > >>>>>>> -0: stram provides REST API to query runtime information and IMO >> > >> this >> > >>>>>>> is sufficient to build an external webUI application(s). >> > >>>>>>> >> > >>>>>>> Thank you, >> > >>>>>>> >> > >>>>>>> Vlad >> > >>>>>>> >> > >>>>>>> On 6/6/18 11:17, Ambarish Pande wrote: >> > >>>>>>>> +1 For this feature. It would be really helpfull for users to >> > >>>>>>>> visualize all >> > >>>>>>>> this information and the DAG. >> > >>>>>>>> >> > >>>>>>>> +1 for ReactJS + MaterialUI >> > >>>>>>>> A great choice, given its flexibility and performance. >> > >>>>>>>> >> > >>>>>>>> I would like to contribute to this effort. >> > >>>>>>>> >> > >>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar < >> > >>> chin...@apache.org> >> > >>>>>>>> wrote: >> > >>>>>>>> >> > >>>>>>>>> Hi Community, >> > >>>>>>>>> >> > >>>>>>>>> I want to propose a feature in apex-core to have a UI >> component >> > >> in >> > >>>>>>>>> stram. >> > >>>>>>>>> This is influenced by how basic runtime information is shown >> on >> > >> UI >> > >>> by >> > >>>>>>>>> spark. >> > >>>>>>>>> >> > >>>>>>>>> This includes following features: >> > >>>>>>>>> 1. Webpage will be hosted by stram and will be available for >> user >> > >>> to >> > >>>>>>>>> view >> > >>>>>>>>> in one of the two ways (I'll prefer approach b): >> > >>>>>>>>> a. Hosted on a static port in which case we'll need to >> add >> > a >> > >>> new >> > >>>>>>>>> servlet >> > >>>>>>>>> to stram >> > >>>>>>>>> b. The webpage will be hosted on the same port as that >> of >> > >>> stram >> > >>>>>>>>> webservice but at a different path. Say, http:// >> > >>>>>>>>> <stram_host>:<webservice_port>/ui >> > >>>>>>>>> >> > >>>>>>>>> 2. The webpage will be a static page and depending on the >> > >> framework >> > >>>> we >> > >>>>>>>>> chose, it can show the realtime metric data from stram. >> > >>>>>>>>> >> > >>>>>>>>> 3. There will be a categories of readonly information (maybe >> > >>>>>>>>> dynamically >> > >>>>>>>>> changing) that will be shown on the UI: >> > >>>>>>>>> a. Application level information >> > >>>>>>>>> i. Status >> > >>>>>>>>> ii. Number of tuples processed/emitted >> > >>>>>>>>> iii. Start time/ Running span >> > >>>>>>>>> iv. Stram events >> > >>>>>>>>> v. Total memory used/available >> > >>>>>>>>> vi. Number of container allocated/deployed >> > >>>>>>>>> v. Other information which is available in stram >> > >>>>>>>>> b. Logical Operator level information >> > >>>>>>>>> i. Status >> > >>>>>>>>> ii. Number of tuples processed/emitted >> > >>>>>>>>> iii. Important events related to logical operator >> > >>>>>>>>> iv. Container list in which the operator is deployed >> > >>>>>>>>> v. Any other information available in stram >> > >>>>>>>>> vi. Logical DAG View >> > >>>>>>>>> c. Container level information >> > >>>>>>>>> i. Status >> > >>>>>>>>> ii. Number of tuples processed/emitted >> > >>>>>>>>> iii. Important events related to logical operator >> > >>>>>>>>> iv. Any other information available in stram >> > >>>>>>>>> v. Physical DAG View >> > >>>>>>>>> >> > >>>>>>>>> 4. In second phase, there will be control related operations >> > >>> allowed >> > >>>>>>>>> from >> > >>>>>>>>> the UI, as follows: >> > >>>>>>>>> a. Stop/Kill application >> > >>>>>>>>> b. Stop/Kill containers >> > >>>>>>>>> c. Stack trace dump of containers >> > >>>>>>>>> d. Logs related? >> > >>>>>>>>> d. etc...?? >> > >>>>>>>>> >> > >>>>>>>>> The above implementation can be done in phases as follows: >> > >>>>>>>>> 1. Have webpage display application level information only >> > >>>> (read-only). >> > >>>>>>>>> Static display first (i.e. page needs to refresh to see latest >> > >>>>>>>>> data), next >> > >>>>>>>>> dynamically changing data. >> > >>>>>>>>> 2. Have jenkins build tools updated as needed by UI framework. >> > >>>>>>>>> 3. Update the backend (stram) to serve the UI pages >> > >>>>>>>>> 3. Extend the webage to display logical operator information. >> > >>>>>>>>> 4. Extend the webage to display physical operator information >> > >>>>>>>>> 5. Implement control operations on UI. >> > >>>>>>>>> >> > >>>>>>>>> >> > >>>>>>>>> To implement above functionality, I propose ReactJS as UI >> > >> framework >> > >>>> and >> > >>>>>>>>> MaterialUI (based on Google's material design) for >> > >>> theme/controls/CSS >> > >>>>>>>>> support. >> > >>>>>>>>> >> > >>>>>>>>> Please let me know what you think about the same. >> > >>>>>>>>> >> > >>>>>>>>> Also, please let know if anyone is interested in contributing >> to >> > >>> this >> > >>>>>>>>> feature. >> > >>>>>>>>> >> > >>>>>>>>> Thanks, >> > >>>>>>>>> Chinmay. >> > >>>>>>>>> >> > >>>> >> > >> > >> >
