Added the subtasks. I just realized that by mistake I created the jiras in APEXMALHAR instead of APEXCORE. Is there a way to move the jiras to different project? Let me know otherwise, I'll close these jiras and recreate new ones in APEXCORE.
Thanks, Chinmay. On Sat, Jun 16, 2018 at 7:50 PM Chinmay Kolhatkar <chin...@apache.org> wrote: > Hi All, > > I've created following jiras encompassing the requirements (To be picked > up in order): > https://issues.apache.org/jira/browse/APEXMALHAR-2558 > https://issues.apache.org/jira/browse/APEXMALHAR-2559 > > I'll start adding subtasks for subitems mentioned in the jira. > > I'll also create the feature branch shortly so that we can start working > against it. > > -Chinmay. > > > > On Sat, Jun 16, 2018 at 7:07 PM Chinmay Kolhatkar <chin...@apache.org> > wrote: > >> So here is how it'll be: >> 1. ReactJS application can be compiled for production builds. The output >> is set of html+css+compressed js files. It can also output a war file >> directly (I think). >> 2. The react app will have a configuration switch to allow user to set >> the baseUrl. Default value will be "/ws/v2". >> 3. These files can be bundled with stram so that it can be hosted by >> stram. In this case the base url for reading data will be "/ws/v2".. i.e. >> read from localhost. >> 3. OR the config switch be set "http://path/to/stram/ws/v2"; using which >> one can host it seperately. >> >> Basically, it can be reusable with configuration changes during/after >> build time. >> >> Hope that helps. >> >> Thanks, >> Chinmay. >> >> On Thu, Jun 14, 2018 at 10:09 PM Pramod Immaneni < >> pramod.imman...@gmail.com> wrote: >> >>> I don't think they would be reusable as is without any changes externally >>> as the application context would not be available. Also externally you >>> would want to have a different kind of application that aggregates across >>> multiple applications. >>> >>> Thanks >>> >>> On Thu, Jun 14, 2018 at 10:24 AM Vlad Rozov <vro...@apache.org> wrote: >>> >>> > If it is a set of static files that use existing REST API to display >>> > information in a browser, where they will be hosted is irrelevant. They >>> > can be hosted by stram, but it should be possible to host them >>> > externally without making modificaitons. If this is the case, I don't >>> > have any objections to the proposal. >>> > >>> > Thank you, >>> > >>> > Vlad >>> > >>> > On 6/13/18 06:09, Chinmay Kolhatkar wrote: >>> > > Hi Team, >>> > > >>> > > I agree with Thomas. The main focus of this proposal was to provide >>> a UI >>> > > layer using existing REST APIs. >>> > > I think we should first focus on providing a meaningful UI served by >>> AM >>> > to >>> > > user using the same existing REST APIs. >>> > > >>> > > If in general, there is a agreement for the proposal, I can proceed >>> with >>> > > creating a JIRA and feature branch for the same. >>> > > >>> > > Thanks, >>> > > Chinmay. >>> > > >>> > > >>> > > On Wed, Jun 13, 2018 at 12:21 AM Thomas Weise <t...@apache.org> >>> wrote: >>> > > >>> > >> Looks like there is some disconnect in this thread, a few replies >>> > express >>> > >> concerns that the proposal IMO doesn't imply. >>> > >> >>> > >> There is already a REST API that backs all of the CLI functionality. >>> > Modern >>> > >> UIs run in the browser, and so what was proposed (unless I >>> > misunderstood) >>> > >> would be based on the same REST API and nothing extra. The only >>> > additional >>> > >> thing the AM has to do is serve static files. And AFAIK it would >>> > actually >>> > >> simplify the UI implementation when files and REST API have the same >>> > >> origin. >>> > >> >>> > >> To me where files are coming from is actually a smaller piece and >>> > something >>> > >> that can be augmented. But before proposing solutions outside of >>> Apex or >>> > >> additional deployment requirements, please consider the user >>> > experience. It >>> > >> should be really simple to setup and use the UI. >>> > >> >>> > >> I would expect that a large part of the discussion evolves around >>> the UI >>> > >> functionality that helps the target persona. >>> > >> >>> > >> Thanks, >>> > >> Thomas >>> > >> >>> > >> On Tue, Jun 12, 2018, 8:05 PM Shubhrajyoti Mohapatra < >>> > >> sjmohapat...@gmail.com> >>> > >> wrote: >>> > >> >>> > >>> Hi Community, >>> > >>> >>> > >>> I would like to propose a slightly different approach on having an >>> UI >>> > for >>> > >>> Apex. >>> > >>> >>> > >>> The details are below: >>> > >>> >>> > >>> 1. This involves having an external web-server installed along >>> > side of >>> > >>> apexCli. We will basically mimic all the functionalities that >>> > apexCli >>> > >>> currently provides. >>> > >>> 2. We will develop the necessary REST APIs for fetch/update >>> > operations >>> > >>> which apexCli currently supports. >>> > >>> 3. Because most of the information and operations needed by a >>> > typical >>> > >>> user are already there in apexCli, it will of great help to end >>> > users >>> > >>> and >>> > >>> which may increase the adoption rate. >>> > >>> >>> > >>> About Chinmay's proposal of having the STRAM hosting the apps, >>> please >>> > >> find >>> > >>> my comments below: >>> > >>> >>> > >>> 1. It may increase load on STRAM if we were to provide real >>> time >>> > >> display >>> > >>> of information. I am aware that apexCli fetches info from STRAM >>> > apis >>> > >> but >>> > >>> continuous polling can cause extra over-head. >>> > >>> 2. The UI will have limited scope for the particular >>> application. >>> > >> Users >>> > >>> would definitely love to have a comprehensive UI for the >>> > management of >>> > >>> Apex >>> > >>> Apps as a whole. >>> > >>> 3. UI might be able to affect the whole application through >>> STRAM >>> > >> hence >>> > >>> giving direct access to STRAM APIs might not be good idea. >>> > >>> 4. Because STRAM can't have a comprehensive UI hosted inside, >>> > anyways >>> > >> we >>> > >>> will have to develop an external one. >>> > >>> >>> > >>> So I would like propose the plan on action like this. >>> > >>> >>> > >>> 1. We develop the wrapper UI for apexCli in the first phase and >>> > give >>> > >>> user all the management options as REST APIs. >>> > >>> 2. If some of information we require are not in the current >>> apexCli >>> > >>> implementation, we can add those feature to apexCli and if >>> > required to >>> > >>> STRAM APIs in subsequent phases. >>> > >>> 3. In future we can even remove the REST server from STRAM if >>> we >>> > could >>> > >>> provide something like MetricsStore functionality which was >>> there >>> > in >>> > >>> DataTorrent RTS. >>> > >>> >>> > >>> -- >>> > >>> Thanks and Regards, >>> > >>> Shubhrajyoti Mohapatra >>> > >>> >>> > >>> >>> > >>> >>> > >>> >>> > >>> On Tue, Jun 12, 2018 at 2:49 AM Vlad Rozov <vro...@apache.org> >>> wrote: >>> > >>> >>> > >>>> IMO it will be hard to provide UI that meets everyone >>> expectation. An >>> > >>>> external webUI can serve two goals >>> > >>>> >>> > >>>> 1. provide reference UI implementation >>> > >>>> 2. ensure that REST API have all necessary entry points >>> > >>>> >>> > >>>> by embedding webUI directly into stram it will be tempting to >>> shortcut >>> > >>>> some calls to use stram classes directly. >>> > >>>> >>> > >>>> Thank you, >>> > >>>> >>> > >>>> Vlad >>> > >>>> >>> > >>>> On 6/11/18 12:18, Pramod Immaneni wrote: >>> > >>>>> I too think that a comprehensive UI works better standalone but >>> given >>> > >>>> that >>> > >>>>> stram already runs a webapp server, provides web services and RM >>> > >>>> provides a >>> > >>>>> proxy with mapped url space for it, we can extend stram to >>> provide >>> > >>> better >>> > >>>>> visual output. Overall I am +1 on the idea. >>> > >>>>> >>> > >>>>> On Mon, Jun 11, 2018 at 8:59 AM Vlad Rozov <vro...@apache.org> >>> > >> wrote: >>> > >>>>>> Clarification: I am -0 on making UI part of the stram. I am +1 >>> on >>> > >>>>>> providing reference webUI application outside of the stram. >>> > >>>>>> >>> > >>>>>> Thank you, >>> > >>>>>> >>> > >>>>>> Vlad >>> > >>>>>> >>> > >>>>>> On 6/9/18 07:05, Vlad Rozov wrote: >>> > >>>>>>> -0: stram provides REST API to query runtime information and >>> IMO >>> > >> this >>> > >>>>>>> is sufficient to build an external webUI application(s). >>> > >>>>>>> >>> > >>>>>>> Thank you, >>> > >>>>>>> >>> > >>>>>>> Vlad >>> > >>>>>>> >>> > >>>>>>> On 6/6/18 11:17, Ambarish Pande wrote: >>> > >>>>>>>> +1 For this feature. It would be really helpfull for users to >>> > >>>>>>>> visualize all >>> > >>>>>>>> this information and the DAG. >>> > >>>>>>>> >>> > >>>>>>>> +1 for ReactJS + MaterialUI >>> > >>>>>>>> A great choice, given its flexibility and performance. >>> > >>>>>>>> >>> > >>>>>>>> I would like to contribute to this effort. >>> > >>>>>>>> >>> > >>>>>>>> On Wed, 6 Jun 2018 at 8:13 PM, Chinmay Kolhatkar < >>> > >>> chin...@apache.org> >>> > >>>>>>>> wrote: >>> > >>>>>>>> >>> > >>>>>>>>> Hi Community, >>> > >>>>>>>>> >>> > >>>>>>>>> I want to propose a feature in apex-core to have a UI >>> component >>> > >> in >>> > >>>>>>>>> stram. >>> > >>>>>>>>> This is influenced by how basic runtime information is shown >>> on >>> > >> UI >>> > >>> by >>> > >>>>>>>>> spark. >>> > >>>>>>>>> >>> > >>>>>>>>> This includes following features: >>> > >>>>>>>>> 1. Webpage will be hosted by stram and will be available for >>> user >>> > >>> to >>> > >>>>>>>>> view >>> > >>>>>>>>> in one of the two ways (I'll prefer approach b): >>> > >>>>>>>>> a. Hosted on a static port in which case we'll need to >>> add >>> > a >>> > >>> new >>> > >>>>>>>>> servlet >>> > >>>>>>>>> to stram >>> > >>>>>>>>> b. The webpage will be hosted on the same port as that >>> of >>> > >>> stram >>> > >>>>>>>>> webservice but at a different path. Say, http:// >>> > >>>>>>>>> <stram_host>:<webservice_port>/ui >>> > >>>>>>>>> >>> > >>>>>>>>> 2. The webpage will be a static page and depending on the >>> > >> framework >>> > >>>> we >>> > >>>>>>>>> chose, it can show the realtime metric data from stram. >>> > >>>>>>>>> >>> > >>>>>>>>> 3. There will be a categories of readonly information (maybe >>> > >>>>>>>>> dynamically >>> > >>>>>>>>> changing) that will be shown on the UI: >>> > >>>>>>>>> a. Application level information >>> > >>>>>>>>> i. Status >>> > >>>>>>>>> ii. Number of tuples processed/emitted >>> > >>>>>>>>> iii. Start time/ Running span >>> > >>>>>>>>> iv. Stram events >>> > >>>>>>>>> v. Total memory used/available >>> > >>>>>>>>> vi. Number of container allocated/deployed >>> > >>>>>>>>> v. Other information which is available in stram >>> > >>>>>>>>> b. Logical Operator level information >>> > >>>>>>>>> i. Status >>> > >>>>>>>>> ii. Number of tuples processed/emitted >>> > >>>>>>>>> iii. Important events related to logical operator >>> > >>>>>>>>> iv. Container list in which the operator is >>> deployed >>> > >>>>>>>>> v. Any other information available in stram >>> > >>>>>>>>> vi. Logical DAG View >>> > >>>>>>>>> c. Container level information >>> > >>>>>>>>> i. Status >>> > >>>>>>>>> ii. Number of tuples processed/emitted >>> > >>>>>>>>> iii. Important events related to logical operator >>> > >>>>>>>>> iv. Any other information available in stram >>> > >>>>>>>>> v. Physical DAG View >>> > >>>>>>>>> >>> > >>>>>>>>> 4. In second phase, there will be control related operations >>> > >>> allowed >>> > >>>>>>>>> from >>> > >>>>>>>>> the UI, as follows: >>> > >>>>>>>>> a. Stop/Kill application >>> > >>>>>>>>> b. Stop/Kill containers >>> > >>>>>>>>> c. Stack trace dump of containers >>> > >>>>>>>>> d. Logs related? >>> > >>>>>>>>> d. etc...?? >>> > >>>>>>>>> >>> > >>>>>>>>> The above implementation can be done in phases as follows: >>> > >>>>>>>>> 1. Have webpage display application level information only >>> > >>>> (read-only). >>> > >>>>>>>>> Static display first (i.e. page needs to refresh to see >>> latest >>> > >>>>>>>>> data), next >>> > >>>>>>>>> dynamically changing data. >>> > >>>>>>>>> 2. Have jenkins build tools updated as needed by UI >>> framework. >>> > >>>>>>>>> 3. Update the backend (stram) to serve the UI pages >>> > >>>>>>>>> 3. Extend the webage to display logical operator information. >>> > >>>>>>>>> 4. Extend the webage to display physical operator information >>> > >>>>>>>>> 5. Implement control operations on UI. >>> > >>>>>>>>> >>> > >>>>>>>>> >>> > >>>>>>>>> To implement above functionality, I propose ReactJS as UI >>> > >> framework >>> > >>>> and >>> > >>>>>>>>> MaterialUI (based on Google's material design) for >>> > >>> theme/controls/CSS >>> > >>>>>>>>> support. >>> > >>>>>>>>> >>> > >>>>>>>>> Please let me know what you think about the same. >>> > >>>>>>>>> >>> > >>>>>>>>> Also, please let know if anyone is interested in >>> contributing to >>> > >>> this >>> > >>>>>>>>> feature. >>> > >>>>>>>>> >>> > >>>>>>>>> Thanks, >>> > >>>>>>>>> Chinmay. >>> > >>>>>>>>> >>> > >>>> >>> > >>> > >>> >>
