Hi, I have submitted the GSoC Proposal 2015 under the title of "Visualizing CouchDB Cluster with Utilization Details on a Dashboard", including the last minute revisions I made with the feedback from Robert and Alexander on heat maps & concurrent requests.
My gratitude goes to the active Community of CouchDB, specially Alexander Shorin, Robert Kowalski, James Mutton, Robert Newson and Sebastian Rothbucher for your valuable ideas and feedback. I hope to write few blog posts on the current tasks I'm doing with regard to this project and share with you all. On Fri, Mar 27, 2015 at 7:13 PM, Robert Kowalski <r...@kowalski.gd> wrote: > yes thank you! > > On Fri, Mar 27, 2015 at 7:02 AM, madhuka udantha > <madhukaudan...@gmail.com> wrote: > > Hi, > > > > I'm sharing a sample mxn Matrix Graph[1] so you all can get an idea of > what > > my proposed solution is. This can represent high number of nodes with out > > any issue and at a glance you can get the Summarized View of the > Cluster. > > Accordingly my mxn Matrix would represent nodes in the Cluster. > > (Mockup UIs are attached to JIRA) > > > > Does this clarify the issue Robert? > > > > Your feedback / ideas mostly welcomed. > > > > [1] http://bost.ocks.org/mike/miserables/ > > > > On Wed, Mar 25, 2015 at 11:46 PM, madhuka udantha < > madhukaudan...@gmail.com> > > wrote: > >> > >> Hi, > >> > >> Sharing mockups UIs again, Since some ML-Users do not have account in > >> box.com. > >> I am sharing them via jira. > >> > >> On Wed, Mar 25, 2015 at 12:20 AM, madhuka udantha > >> <madhukaudan...@gmail.com> wrote: > >>> > >>> Hi, > >>> > >>> After communicating with couchdb community got to know that there can > be > >>> 100~ 10000 nodes in cluster. So there will 2 layouts in cluster link > >>> depending on node count in cluster. > >>> > >>> My first UI is for simple count of node in cluster (4 nodes in cluster) > >>> > >>> Here is mock UIs where it hit more than 5 to 100000 nodes, I am using > >>> matrix visual. > >>> Here is initial look of the page. [Page1v1-initial-page] > >>> https://app.box.com/s/sn84u6wuzya58trssw11ba6fwtmn862f > >> > >> > >> > https://issues.apache.org/jira/secure/attachment/12707205/Page1v1-initial-page.png > >> > >>> > >>> Features: > >>> 1. User can select the attribute eg: Disk usage, http status, data > usage, > >>> alerts. etc. > >>> 2. Chart / matrix will be draw with legend > >>> > >>> [Page2v1-searching-node] > >>> https://app.box.com/s/i5uomdhirn96co6acbbrvvviifyw95pb > >> > >> > >> > https://issues.apache.org/jira/secure/attachment/12707218/Page2v1-searching-node.png > >> > >>> > >>> Who can we search node in there? > >>> You will have search in top of it. (it allow to search the node), It > will > >>> show it with red color box. > >>> Or every node have tool tip to verify node, what is you look for. > >>> > >>> > >>> [Page3v1-selecting-node] > >>> https://app.box.com/s/rsd5ysld597gvra2nkcn7027bpok37jc > >> > >> > >> > https://issues.apache.org/jira/secure/attachment/12707219/Page3v1-selecting-node.png > >> > >>> > >>> When user click on node, node panel will be open RHS. > >>> It have all the details of the node and operation that you can perform > in > >>> node leve > >>> > >>> [Page4v1-searching-db_in_node] > >>> https://app.box.com/s/jk3ojb0khj192ldnyw0eqs1p5t7l485r > >> > >> > >> > https://issues.apache.org/jira/secure/attachment/12707220/Page4v1-searching-db_in_node.png > >> > >>> > >>> User can search Databases in nodes even > >>> This searching have advance search query support > >>> > >>> [Page5v1-db_view] > >>> https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni > >> > >> > >> > https://issues.apache.org/jira/secure/attachment/12707206/Page5v1-db_view.png > >> > >>> User can select the database and it will show database details. > >>> > >>> > >>> feed backs for UIs are welcome. > >>> > >>> > >>> On Tue, Mar 24, 2015 at 4:35 PM, madhuka udantha > >>> <madhukaudan...@gmail.com> wrote: > >>>> > >>>> Hi James, > >>>> > >>>> Thanks for you feedback. It will help me to improve my proposal and to > >>>> get better understanding about couchdb cluster > >>>> > >>>> Since I am new to couchdb and I did not know much user case that you > all > >>>> have come across. But when you share those ideas. It is help to > understand > >>>> couchdb cluster. All you feedback is important. > >>>> > >>>> On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <jmut...@akamai.com> > >>>> wrote: > >>>>> > >>>>> I can share a few quick thoughts (which I admit is rare, but I happen > >>>>> to have a few minutes and this hit the top of my stack): > >>>>> > >>>>> * How do you envision this overflowing when you have hundreds of > >>>>> databases within a cluster, or maybe 64 nodes in a cluster. > >>>> > >>>> Attached UIs can only handle very low level nodes (4 to 8) according > to > >>>> layout (we can called it expanded layout). When I am drawing I did > not > >>>> thought that level of count of nodes and databases. Therefore I will > revisit > >>>> the mockup ui. Here is updated UI > >>>> > >>>> Here you can have 1 ~ 1000 node even and each node can contain 100 > >>>> databases. > >>>> Node are listed and user can pick one of it then it will show it > details > >>>> and databases of it in list. > >>>> > >>>> UI for Listing node when nodes count is above 8 in cluster (This is > how > >>>> it look as it have 64 nodes in cluster) > >>>> https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5 > >>>> [Page10-list-nodes-more-than-8] > >>>> > >>>> Features of the UI: > >>>> > >>>> Table header support sorting (ascending and descending) by just click > on > >>>> it > >>>> User have searching option to search (node/databases by name) > >>>> It support advance search query (popup form ui) > >>>> Notification listed with icon and warning/alert level is shown by > color > >>>> of icon > >>>> User can perform node operation by clicking hyperlink in it's raw > >>>> > >>>> > >>>> Node contains 100 databases (answer) > >>>> https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st > >>>> [Page11-list-dbs-in-selected-node] > >>>> > >>>> > >>>> > >>>> Features of the UI > >>>> > >>>> In Database panel contain search that scope to databases of the > selected > >>>> node > >>>> When user select database it will show database summary details and > >>>> operations that can be perform for that database > >>>> > >>>> > >>>>> > >>>>> ** On a related note, drag and drop is cool, but level of difficulty > >>>>> involved in adding a shard from node-1 increases in direct > proportion to the > >>>>> number of nodes there are. > >>>> > >>>> Yes, we can add button ( in database/node level > >>>> > >>>>> I think the “Add Database” button on a new server should be labeled > >>>>> “Add Shard” and should present the option of selecting what shards > to add. > >>>>> The add button is also not present on the nodes that have shards > already so > >>>>> it would be good to keep it around somehow. > >>>> > >>>> Yes, It will be label as “Add Shard”. It is possible to show only > that > >>>> is not shard so far. > >>>> > >>>> > >>>>> * Speaking then of shards… They don’t appear to be captured here. > How > >>>>> does one know which shard-set each server has? Each database has a > number > >>>>> of shards that all have multiple shard copies and it is not a > requirement > >>>>> that all servers have all shards. None of that is really > represented in > >>>>> these mockups. When looking at a shard, it would be good to know if > it is > >>>>> 00000000-15555554 or aaaaaaa8-bffffffc, or even both. Knowing this > and > >>>>> being able to use it to scale/move hotspots in a cluster is pretty > important > >>>>> for operations, once databases hit a significant size. > >>>> > >>>> Thanks for pointing out. shards will add for visualization. > >>>> > >>>> Can you share resources/documentations which that help to understand > the > >>>> couchdb clustering? > >>>> I like to know what are most important operations regard couch > >>>> clustering? (that you think good to added in here. since I am new for > >>>> couchdb and some time I have miss them) > >>>> > >>>> > >>>>> > >>>>> * I don’t think doc-count and deleted-count is useful information at > >>>>> the overview page > >>>> > >>>> Thanks, We can remove doc count details from overview page. If user > need > >>>> to read such details he/she can click 'more' so there it will show > them. > >>>>> > >>>>> > >>>>> > >>>>> and runs the risk of requiring a lot of requests to fulfill (see > point > >>>>> 1 on hundreds of databases). > >>>> > >>>> Sorry, In here I did not get your point > >>>> > >>>>> * What does compact: false mean? That the database is not small? or > >>>>> that it’s not actively compacting? And if the latter, wouldn’t that > be on a > >>>>> db-shard basis, not a whole server. > >>>> > >>>> compact running is mapping to 'compact_running'[2] > >>>> > >>>> [1] https://wiki.apache.org/couchdb/Compaction > >>>> > >>>>> > >>>>> * It would be nice if cluster usage graphing page would show > http-stats > >>>>> as well. Would give people a snapshot indication of where their > load is > >>>>> hot. > >>>> > >>>> Sure, Different status codes will be notify on cluster usage graph. > >>>> (Mockups UI will shared tonight) > >>>> > >>>> Thanks JamesM for your feed backs > >>>>> > >>>>> > >>>>> </JamesM> > >>>>> > >>>>> > >>>>> On Mar 23, 2015, at 20:32, madhuka udantha <madhukaudan...@gmail.com > > > >>>>> wrote: > >>>>> > >>>>> > Hi All, > >>>>> > > >>>>> > are there any ideas and feedback for above mockup UIs? > >>>>> > > >>>>> > Community feedback will be useful in here. :) > >>>>> > > >>>>> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha > >>>>> > <madhukaudan...@gmail.com> > >>>>> > wrote: > >>>>> > > >>>>> >> Hi, > >>>>> >> > >>>>> >> Here are the mockup UIs for Visualizing the CouchDB Clusters. It > is > >>>>> >> just > >>>>> >> the initial UIs. > >>>>> >> > >>>>> >> * Page 1-cluster.png* [ > >>>>> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl > >>>>> >> > >>>>> >> This Page visualize nodes in the clusters with summary information > >>>>> >> for > >>>>> >> each node. (Name, Memory usage..etc..) with a nice web UI > interface. > >>>>> >> It is > >>>>> >> interactive (drag and drop supporting over nodes) > >>>>> >> > >>>>> >> *Page 2-cluster-notification-tip.png [* > >>>>> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb > >>>>> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>] > >>>>> >> > >>>>> >> Top of each panel of the nodes it contains a notification icon if > >>>>> >> there is > >>>>> >> notification. > >>>>> >> eg: Memory/disk space availability, Error in node etc... > >>>>> >> Notification will have a simple drop down structure as it is > >>>>> >> showing. > >>>>> >> > >>>>> >> *Page 3-cluster-warning.png* [ > >>>>> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b] > >>>>> >> > >>>>> >> Critical level alert is showen. (Alert level can be set in config > >>>>> >> and > >>>>> >> notification center) > >>>>> >> > >>>>> >> *Page 4-adding-new-node.png* [ > >>>>> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057] > >>>>> >> > >>>>> >> By clicking "add new node" you can add new nodes for clusters. UI > >>>>> >> Consistency is maintained > >>>>> >> > >>>>> >> *Page 5-adding-new-node-by-drag-drop.png*[ > >>>>> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7] > >>>>> >> > >>>>> >> User interface is supported for drag-drop of databases in nodes. > >>>>> >> (replication operation in the BE) > >>>>> >> > >>>>> >> *Page 6-selected-db-in-node.png*[ > >>>>> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9] > >>>>> >> Each node shows the databases in it. By Clicking on database you > can > >>>>> >> see > >>>>> >> detailed view of a particular database. This allows users to > >>>>> >> understand > >>>>> >> memory/data/doc distribution over DBs in particular nodes. > >>>>> >> > >>>>> >> *Page 7-custer-mgt-center.png* [ > >>>>> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z] > >>>>> >> Each node is shown with disk usage under icon and summary text of > >>>>> >> the node > >>>>> >> is also given. It is easy to identify the disk space - free > >>>>> >> availability. (In here we need some BE services help) > >>>>> >> > >>>>> >> > >>>>> >> *Page 8-custer-mgt-center-infor.png* [ > >>>>> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc] > >>>>> >> User can customize notifications for each nodes by level (warning, > >>>>> >> infor, > >>>>> >> etc..) > >>>>> >> > >>>>> >> *Page 9-charting-disk-usages.png*[ > >>>>> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz] > >>>>> >> Cluster usage of the system is shown as a bar chart and the > >>>>> >> attributes to > >>>>> >> be shown can be selected according to user's preferences. > >>>>> >> > >>>>> >> You ideas are welcome in here. > >>>>> >> Thanks > >>>>> >> > >>>>> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha > >>>>> >> <madhukaudan...@gmail.com > >>>>> >>> wrote: > >>>>> >> > >>>>> >>> Hi All, > >>>>> >>> > >>>>> >>> As I'm drafting my proposal for GSoC 2015 I went through some of > >>>>> >>> the > >>>>> >>> existing Cluster Management Systems with Visualization. I > >>>>> >>> recognized some > >>>>> >>> of the features regarding to Cluster Management HTTP API. (Some > >>>>> >>> times not > >>>>> >>> fully documented yet) > >>>>> >>> Such us partitioning, re-balance..etc. > >>>>> >>> I was unable to find them on CouchDB API Documentation(only at > the > >>>>> >>> implemented level but not documented yet? ) but I was able to > find > >>>>> >>> some of > >>>>> >>> the features such as adding nodes to clusters through dev-mailing > >>>>> >>> list. > >>>>> >>> Are some of these features regarding cluster management present > on > >>>>> >>> the > >>>>> >>> API. If so where can I find the relevant > documentations/resources? > >>>>> >>> Even > >>>>> >>> the source code is fine to me. > >>>>> >>> > >>>>> >>> > >>>>> >>> Since I was able to clearly identify those missing features can I > >>>>> >>> start a > >>>>> >>> discussion on a thread on the best feature list for cluster > >>>>> >>> management on > >>>>> >>> CouchDB to be added to it on a future release. > >>>>> >>> My current project could also include these features that are > >>>>> >>> supported > >>>>> >>> by the API else I could help on including them. > >>>>> >>> > >>>>> >>> [1] http://guide.couchdb.org/draft/clustering.html > >>>>> >>> Thanks. > >>>>> >>> BR, > >>>>> >>> > >>>>> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha < > >>>>> >>> madhukaudan...@gmail.com> wrote: > >>>>> >>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin > >>>>> >>>> <kxe...@gmail.com> > >>>>> >>>> wrote: > >>>>> >>>> > >>>>> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha > >>>>> >>>>> <madhukaudan...@gmail.com> wrote: > >>>>> >>>>>> I like to know what is *different of '_nodes' and 'node'*? > >>>>> >>>>>> > >>>>> >>>>>> http://localhost:15986/nodes > >>>>> >>>>>> -->{"db_name":"nodes","doc_count":3..} > >>>>> >>>>>> http://localhost:25986/nodes > >>>>> >>>>>> -->{"db_name":"nodes","doc_count":1..} > >>>>> >>>>>> http://localhost:35986/nodes > >>>>> >>>>> -->{"error":"not_found","reason":"no_db_file"} > >>>>> >>>>> > >>>>> >>>>> Where did you get these nodes? > >>>>> >>>>> > >>>>> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes. > (default > >>>>> >>>> it is > >>>>> >>>> 3 nodes) > >>>>> >>>> > >>>>> >>>> Here is my console print, Yes now I have four node running. > >>>>> >>>> > >>>>> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4* > >>>>> >>>> *Setup environment... ok* > >>>>> >>>> *Ensure CouchDB is built... ok* > >>>>> >>>> *Prepare configuration files... ok* > >>>>> >>>> *Start node node1... ok* > >>>>> >>>> *Start node node2... ok* > >>>>> >>>> *Start node node3... ok* > >>>>> >>>> *Start node node4... ok* > >>>>> >>>> *Ensure all nodes are run... ok* > >>>>> >>>> *Join nodes into cluster... ok* > >>>>> >>>> *Developers cluster is set up at http://127.0.0.1:15984 > >>>>> >>>> <http://127.0.0.1:15984>. Time to hack!... * > >>>>> >>>> > >>>>> >>>> Each above nodes are running in localhost in port 15986, 25986, > >>>>> >>>> 35986 > >>>>> >>>> and 45986 > >>>>> >>>> > >>>>> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count' > >>>>> >>>> value, > >>>>> >>>> is there any reason for that or is it happening randomly? > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>>> > >>>>> >>>>> -- > >>>>> >>>>> ,,,^..^,,, > >>>>> >>>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> > >>>>> >>>> -- > >>>>> >>>> Cheers, > >>>>> >>>> Madhuka Udantha > >>>>> >>>> http://madhukaudantha.blogspot.com > >>>>> >>>> > >>>>> >>> > >>>>> >>> > >>>>> >>> > >>>>> >>> -- > >>>>> >>> Cheers, > >>>>> >>> Madhuka Udantha > >>>>> >>> http://madhukaudantha.blogspot.com > >>>>> >>> > >>>>> >> > >>>>> >> > >>>>> >> > >>>>> >> -- > >>>>> >> Cheers, > >>>>> >> Madhuka Udantha > >>>>> >> http://madhukaudantha.blogspot.com > >>>>> >> > >>>>> > > >>>>> > > >>>>> > > >>>>> > -- > >>>>> > Cheers, > >>>>> > Madhuka Udantha > >>>>> > http://madhukaudantha.blogspot.com > >>>>> > >>>> > >>>> > >>>> > >>>> -- > >>>> Cheers, > >>>> Madhuka Udantha > >>>> http://madhukaudantha.blogspot.com > >>> > >>> > >>> > >>> > >>> -- > >>> Cheers, > >>> Madhuka Udantha > >>> http://madhukaudantha.blogspot.com > >> > >> > >> > >> > >> -- > >> Cheers, > >> Madhuka Udantha > >> http://madhukaudantha.blogspot.com > > > > > > > > > > -- > > Cheers, > > Madhuka Udantha > > http://madhukaudantha.blogspot.com > -- Cheers, Madhuka Udantha http://madhukaudantha.blogspot.com
