Hello Sir, 1)I created XmlNotebookRepo class, initially To start, I just copied the code from VfsNotebookRepo and changed this line in XmlNotebookRepo
this.filesystemRoot = new URI(new File( conf.getRelativeDir(filesystemRoot. getPath())).getAbsolutePath()); to this.filesystemRoot = new URI(new File( conf.getRelativeDir(filesystemRoot. getPath())+"/notebook-xml").getAbsolutePath()); 2) in zeppelin-site.xml added XmlNotebookRepo property at this point I was hoping to see two directories . obviously both will contain json files for now, since i haven't changed the XmlNotebookRepo save() method to contain JAXB code. 3)but in debugging i found that Zeppelinconfiguration fails to load that file. url = ZeppelinConfiguration.class.getResource(ZEPPELIN_SITE_XML); //url==null so default properties are loaded. In NotebookRepoSync there is for loop which takes zeppelin.notebook.storage properties but it runs only once. for VfsNotebookrepo. So what I thought I can do is change default to XmlNotebookRepo but that would not save both Json and Xml files. will try more and let you know.. Regards, Onkar Shedge. On Mon, Apr 25, 2016 at 7:03 PM, onkar shedge <shedge31on...@gmail.com> wrote: > Thank You , > > Yes seems an interesting week-long project.I am willing to accept. > > On Mon, Apr 25, 2016 at 5:21 PM, Alexander Bezzubov <b...@apache.org> > wrote: > >> Hi Onkar, >> please note the subj change (to keep the discussion focused). >> >> > How to dive into a large codebase ? >> >> Answering your question - there are many ways to approach large codebase >> and you have to learn your's own - nobody except yourself can teach you >> this. And Zeppelin is quite small, so now should be a good time to start >> doing that. >> >> You should begin with checkout the project from VCS, build it locally, >> import it in IDE of your choice (mvn eclipse:eclipse and then "Import >> existing project" would do for Eclipse) and then start exploring the >> top-level sub-modules\folders from there. >> >> Through this summer project you will be mostly interested in NotebookRepo, >> so a good start for it will be you, tinkering with existing >> implementations >> at first. >> >> An interesting week-long pilot project to help you getting started, as >> well >> as a decent blog post subject, can be - implement a new XmlNotebookRepo. >> The goal would be: to have .xml representation of the notebook persisted >> in >> local filesystem along with existing .json one. Could be just note.xml in >> the same folder, or could be `./notebook-xml/<noteId>/note.xml` - it's up >> to you. >> It should save the same notebook, but in XML format, just in the local >> filesystem. Then it as any other storage [1], can be be configured to use >> together with existing VFSNotebookRepo thought NotebookRepoSync. >> >> >> Please, working on this first project do follow Zeppelin's guideline [3], >> and as this is a pilot one - please feel free to crete final PR to [4] >> instead of apache github mirror to showcase your work. Feel free to look >> around how other PRs looks like, and what usually a reviewers ask for >> (documentation, simple tests, etc) >> >> What do you think? Would you be willing to accept? >> As I mentioined before - opensource is a place where self-learning is the >> king and nobody will teach in the university sense of this word. Hope it >> all makes sense and I'm looking forward your first PR soon! :) >> >> >> P.S in this book [2] you have a chance to see how this task is approached >> by other experienced engineers in other projects. Nice read. >> >> 1. http://zeppelin.incubator.apache >> .org/docs/0.6.0-incubating-SNAPSHOT/storage/storage.html >> 2. http://www.codersatwork.com >> 3. >> https://github.com/apache/incubator-zeppelin/blob/master/CONTRIBUTING.md >> 4. https://github.com/bzz/incubator-zeppelin >> >> On Mon, Apr 25, 2016 at 7:09 PM, onkar shedge <shedge31on...@gmail.com> >> wrote: >> >> > Sorry forgot to mention this in previous email. >> > >> > I wanted to ask one thing. >> > How to dive into a large codebase ?. I read some answers suggesting see >> old >> > commits how the project was developed and debug to see control flow. >> > >> > On Mon, Apr 25, 2016 at 3:26 PM, onkar shedge <shedge31on...@gmail.com> >> > wrote: >> > >> > > Hello Alex, >> > > >> > > Here is a link to my blog[1]. >> > > I have added google calendar with three weeks events. I haven't >> written >> > > any posts about my progress, Will do soon. >> > > >> > > 1] http://gsoc2016onkar.blogspot.in/ >> > > >> > > On Sun, Apr 24, 2016 at 4:34 PM, Alexander Bezzubov <b...@apache.org> >> > > wrote: >> > > >> > >> Thank you Onkar, >> > >> >> > >> Its great to have you on board and looking forward experiments with >> P2P >> > >> notebook storage! >> > >> >> > >> It's great idea to keep list posted on your progress as well as >> having >> > >> deeper writeups on a personal blog, please feel free to share links, >> > etc. >> > >> >> > >> It would be good if in emails you could not only list things that you >> > did >> > >> last week, but also to include a brief plan for the next week. This >> way >> > it >> > >> should be easier for me as a mentor to align efforts in the same >> > >> direction. >> > >> >> > >> It is going to be an exciting project! >> > >> >> > >> -- >> > >> Alex >> > >> >> > >> On Sat, Apr 23, 2016, 22:55 onkar shedge <shedge31on...@gmail.com> >> > wrote: >> > >> >> > >> > Hello zeppelin community, >> > >> > >> > >> > Thanks for giving me the opportunity. I will get myself more >> familiar >> > >> with >> > >> > the codebase and ask questions on mailing list about my doubts . >> Also >> > I >> > >> > will post updates weekly/(4 day interval) what I understood,what I >> > >> > worked/read, on my blog. >> > >> > >> > >> > >> > >> > Regards, >> > >> > Onkar Shedge. >> > >> > >> > >> > On Fri, Mar 25, 2016 at 10:59 AM, Alexander Bezzubov < >> b...@apache.org> >> > >> > wrote: >> > >> > >> > >> > > Hi Onkar, >> > >> > > >> > >> > > that sounds great, thank you. >> > >> > > >> > >> > > Looking forward helping with this project though the summer! >> > >> > > >> > >> > > -- >> > >> > > Alex >> > >> > > >> > >> > > On Fri, Mar 25, 2016 at 2:27 PM, onkar shedge < >> > >> shedge31on...@gmail.com> >> > >> > > wrote: >> > >> > > >> > >> > > > Thanks Sir, >> > >> > > > I have made the changes: diagram, Deliverables removed >> research. >> > >> And >> > >> > > > support for one more p2p storage. >> > >> > > > Also I have uploaded the pdf. >> > >> > > > >> > >> > > > Regards, >> > >> > > > Onkar Shedge >> > >> > > > >> > >> > > > On Thu, Mar 24, 2016 at 3:49 PM, Alexander Bezzubov < >> > b...@apache.org >> > >> > >> > >> > > > wrote: >> > >> > > > >> > >> > > > > Hi Onkar, >> > >> > > > > >> > >> > > > > thank you for sharing a blog and even a video of your >> > >> explorations in >> > >> > > > > preparation for the project. >> > >> > > > > Your timeline and proposal looks very strong and it seems >> that >> > you >> > >> > > > relevant >> > >> > > > > experience for this project. >> > >> > > > > >> > >> > > > > On the GSoC scope - it would be a good start with a storage >> > >> > > > > implementation(s) that result in dat://, magnet:// or ipfs:// >> > >> links >> > >> > > for a >> > >> > > > > notebooks, as a first step. From there sharing the link can >> be >> > >> done >> > >> > > > through >> > >> > > > > any communication medium (IM, email, etc) and importing such >> > link >> > >> in >> > >> > > > > Zeppelin instance is a matter of changing "Import" >> > dialog\backend >> > >> > > inside >> > >> > > > > Zeppelin to support it. And then proper note >> > >> versioning\modification >> > >> > > > > support, as well as make sure that it plays nicely with >> multiple >> > >> > > > > NotebookRepo plugged in though NotebookRepoSync [1]. >> > >> > > > > I would expect at least those use-cases to be implemented as >> a >> > >> part >> > >> > of >> > >> > > > the >> > >> > > > > GSoC project. >> > >> > > > > >> > >> > > > > On the deliverables: >> > >> > > > > - "research" is not a deliverable item, may be would be >> better >> > to >> > >> > put >> > >> > > a >> > >> > > > > "Report on results of the research, covering suitability of >> each >> > >> p2p >> > >> > > > > network\stack for the Zeppelin case" >> > >> > > > > >> > >> > > > > Would you be willing to, just as an extra bonus material, >> take >> > >> care >> > >> > of >> > >> > > > more >> > >> > > > > the one p2p NotebookRepo implementation, of course in case if >> > time >> > >> > > > permits? >> > >> > > > > >> > >> > > > > I have also added few comments to the doc itself. >> > >> > > > > >> > >> > > > > Please feel free to incorporate feedback do not forget to >> submit >> > >> the >> > >> > > > final >> > >> > > > > pdf to google before the deadline tomorrow! >> > >> > > > > >> > >> > > > > 1. >> > >> > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepoSync.java#L40 >> > >> > > > > >> > >> > > > > -- >> > >> > > > > Alex >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > On Thu, Mar 24, 2016 at 12:11 PM, onkar shedge < >> > >> > > shedge31on...@gmail.com> >> > >> > > > > wrote: >> > >> > > > > >> > >> > > > > > Hi moon, >> > >> > > > > > Thanks for your idea. You talked about single online >> storage >> > and >> > >> > then >> > >> > > > > > sharing with others like(Google docs). Then handling fault >> > >> > tolerance >> > >> > > as >> > >> > > > > > multiple instances would change the same storage repo. >> > >> > > > > > I was thinking till now that P2P implementation would be >> > >> generating >> > >> > > the >> > >> > > > > > torrent file in case of Bittorrent or dat ://......dat >> link in >> > >> case >> > >> > > of >> > >> > > > > dat >> > >> > > > > > protocol and that file would be changed/versioned as >> changes >> > are >> > >> > made >> > >> > > > new >> > >> > > > > > hashes will be generated. So the question now is how to >> share >> > >> the >> > >> > > > torrent >> > >> > > > > > file or dat link to other peers? >> > >> > > > > > >> > >> > > > > > Is it that there would be a Zeronet site for having a list >> of >> > >> peers >> > >> > > > > online >> > >> > > > > > and each user sharing his notebooks which he has chosen to >> > >> share. >> > >> > > > > > >> > >> > > > > > On Thu, Mar 24, 2016 at 7:59 AM, moon soo Lee < >> > m...@apache.org> >> > >> > > wrote: >> > >> > > > > > >> > >> > > > > > > Hi, >> > >> > > > > > > >> > >> > > > > > > Scope of ZEPPELIN-683 is implementing a Zeppelin >> > NotebookRepo >> > >> [1] >> > >> > > > based >> > >> > > > > > on >> > >> > > > > > > one of P2P technology. I think ZEPPELIN-683 leads to very >> > >> > > interesting >> > >> > > > > > > challenge (as a future work). >> > >> > > > > > > >> > >> > > > > > > I can see characteristics of P2P technology based >> > NotebookRepo >> > >> > as, >> > >> > > > > > > >> > >> > > > > > > * Massively (globally) scalable. >> > >> > > > > > > * Very Elastic. Any peer can join and leave at any time. >> > >> > > > > > > >> > >> > > > > > > Therefore i can see following possibilities and >> challenges. >> > >> > > > > > > >> > >> > > > > > > * Make every zeppelin instance connect to the single >> storage >> > >> > > network. >> > >> > > > > > > * Then it is possible to provide user unlimited online >> > >> notebook >> > >> > > > > storage. >> > >> > > > > > > * And there will be nicer way to share notebook to the >> other >> > >> > > people. >> > >> > > > > > > * Zeppelin currently does handle the case multiple >> zeppelin >> > >> > > instance >> > >> > > > > > share >> > >> > > > > > > single storage. To leverage advantage of P2P technology >> > based >> > >> > > > > > NotebookRepo, >> > >> > > > > > > Zeppelin need to aware that storage can be updated by >> other >> > >> > > Zeppelin >> > >> > > > > > > instances. This could be challenging job. >> > >> > > > > > > * I think it's very much related to support of fault >> > >> tolerance. >> > >> > > > > > > >> > >> > > > > > > I think ZEPPELIN-683 is very wide open to be evolved. >> Please >> > >> feel >> > >> > > > free >> > >> > > > > to >> > >> > > > > > > add your idea. >> > >> > > > > > > >> > >> > > > > > > Thanks, >> > >> > > > > > > moon >> > >> > > > > > > >> > >> > > > > > > [1] >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://github.com/apache/incubator-zeppelin/blob/master/zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/NotebookRepo.java >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > > On Tue, Mar 22, 2016 at 10:34 PM onkar shedge < >> > >> > > > shedge31on...@gmail.com >> > >> > > > > > >> > >> > > > > > > wrote: >> > >> > > > > > > >> > >> > > > > > > > Just wondering isn't this project important as the >> other >> > two >> > >> > > Apache >> > >> > > > > > Beam >> > >> > > > > > > > interpreter and Sample Notebooks ? >> > >> > > > > > > > >> > >> > > > > > > > On Tue, Mar 22, 2016 at 10:59 AM, onkar shedge < >> > >> > > > > > shedge31on...@gmail.com> >> > >> > > > > > > > wrote: >> > >> > > > > > > > >> > >> > > > > > > > > Hello, >> > >> > > > > > > > > Apologies for late reply. >> > >> > > > > > > > > I have spend time understanding the protocol.I >> installed >> > >> all >> > >> > > the >> > >> > > > > > three >> > >> > > > > > > > > techs and tried them. >> > >> > > > > > > > > Also I read about the docs, whitepapers[1].I read >> the >> > >> > > bittorent >> > >> > > > > > > protocol >> > >> > > > > > > > > and wrote a blog[2] about it using jekyll. Please do >> > watch >> > >> > the >> > >> > > > > video. >> > >> > > > > > > > > >> > >> > > > > > > > > I have written the proposal[3].Your feedback are >> > >> welcomed. I >> > >> > am >> > >> > > > > > > confused >> > >> > > > > > > > , >> > >> > > > > > > > > exactly what to write in implementation part.right >> now I >> > >> have >> > >> > > > > written >> > >> > > > > > > > about >> > >> > > > > > > > > the current implementation. >> > >> > > > > > > > > >> > >> > > > > > > > > The available clients are >> > >> > > > > > > > > IPFS - [4] Java >> > >> > > > > > > > > dat : browserify [5], desktop[6], [7] python . If not >> > >> > > available, >> > >> > > > > > > should I >> > >> > > > > > > > > consider writing own Java client? is it doable? >> > >> > > > > > > > > Zeronet : I didn't understand how notebooks can be >> > shared >> > >> > with >> > >> > > > > > zeronet >> > >> > > > > > > > > which serves sites?. >> > >> > > > > > > > > >> > >> > > > > > > > > Please comment. I guess there are many mistakes. >> Thank >> > >> You. >> > >> > > > > > > > > >> > >> > > > > > > > > 1] >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://github.com/ipfs/papers/raw/master/ipfs-cap2pfs/ipfs-p2p-file-system.pdf >> > >> > > > > > > > > 2] >> > >> > > https://onkarshedge.github.io/2016/03/16/peeking-in-p2p.html >> > >> > > > > > > > > https://www.youtube.com/watch?v=WxX0AjqQ28g >> > >> > > > > > > > > 3] >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://docs.google.com/document/d/1GVu_LEi8o6wnnoj9vrt07j8ByiDCLemYh9F9ERLyni8/edit?usp=sharing >> > >> > > > > > > > > 4] https://github.com/ipfs/java-ipfs-api >> > >> > > > > > > > > 5] https://github.com/karissa/dat-browserify >> > >> > > > > > > > > 6] https://github.com/karissa/dat-desk >> > >> > > > > > > > > 7]https://github.com/karissa/datpy >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > > On Thu, Mar 10, 2016 at 6:28 PM, Alexander Bezzubov < >> > >> > > > > b...@apache.org> >> > >> > > > > > > > > wrote: >> > >> > > > > > > > > >> > >> > > > > > > > >> Hi Onkar, >> > >> > > > > > > > >> >> > >> > > > > > > > >> great to hear the you are interested and thank you >> for >> > >> > sharing >> > >> > > > the >> > >> > > > > > > > example >> > >> > > > > > > > >> notebook that you'v built, preview [0] looks great. >> > >> > > > > > > > >> >> > >> > > > > > > > >> I encourage you review this mailing list archives >> very >> > >> > > > carefully, >> > >> > > > > > > > looking >> > >> > > > > > > > >> for the advices to other students on how to get >> started >> > >> with >> > >> > > > > > zeppelin >> > >> > > > > > > > and >> > >> > > > > > > > >> proceed with proposal draft [1] [2] [3]. >> > >> > > > > > > > >> >> > >> > > > > > > > >> Research, as well as publishing the results of such >> in >> > >> > > > wiki\blogs >> > >> > > > > > > should >> > >> > > > > > > > >> be >> > >> > > > > > > > >> substantial part of this project. The expectations >> are >> > >> > though >> > >> > > > that >> > >> > > > > > you >> > >> > > > > > > > >> will >> > >> > > > > > > > >> be able to familiarize yourself with the p2p >> protocols >> > at >> > >> > > least >> > >> > > > a >> > >> > > > > > bit >> > >> > > > > > > > >> before starting actual gsoc project. Engaging and >> > >> bridging >> > >> > > > > multiple >> > >> > > > > > > > >> project >> > >> > > > > > > > >> communities is very welcome as well. Next steps >> could >> > >> build >> > >> > > > > building >> > >> > > > > > > > >> low-fi >> > >> > > > > > > > >> PoC using JVM tools, and then a hi-fi one, using >> > >> pluggable >> > >> > > > > > Repository >> > >> > > > > > > > >> abstraction [4] >> > >> > > > > > > > >> >> > >> > > > > > > > >> Hope this helps and looking forward your proposal >> > draft: >> > >> > > > plaintext >> > >> > > > > > in >> > >> > > > > > > > wiki >> > >> > > > > > > > >> [5] or a link to a google doc will work nicely to >> > gather >> > >> the >> > >> > > > > > feedback >> > >> > > > > > > > and >> > >> > > > > > > > >> engage with potential mentors. >> > >> > > > > > > > >> >> > >> > > > > > > > >> -- >> > >> > > > > > > > >> Alex >> > >> > > > > > > > >> >> > >> > > > > > > > >> 0. https://www.zeppelinhub >> > >> > > > > > > > >> >> > >> > > > > > > > >> >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> .com/viewer/notebooks/aHR0cHM6Ly9yYXcuZ2l0aHVidXNlcmNvbnRlbnQuY29tL29ua2Fyc2hlZGdlL3NhbXBsZS1ub3RlYm9va3MvbWFzdGVyLzJCRllGVVpDUC9ub3RlLmpzb24 >> > >> > > > > > > > >> 1. http://markmail.org/thread/abw6hoayuvi54ghk >> > >> > > > > > > > >> 2. http://markmail.org/thread/j53j7d4rsiisewfb >> > >> > > > > > > > >> 3. http://markmail.org/message/naocktanol5iuot3 >> > >> > > > > > > > >> 4. http://zeppelin.incubator.apache >> > >> > > > > > > > >> >> > .org/docs/0.6.0-incubating-SNAPSHOT/storage/storage.html >> > >> > > > > > > > >> 5. https://cwiki.apache >> > >> > > > > > > > >> >> > >> .org/confluence/display/ZEPPELIN/Google+Summer+Of+Code+2016 >> > >> > > > > > > > >> >> > >> > > > > > > > >> >> > >> > > > > > > > >> On Wed, Mar 9, 2016 at 11:56 PM, onkar shedge < >> > >> > > > > > > shedge31on...@gmail.com> >> > >> > > > > > > > >> wrote: >> > >> > > > > > > > >> >> > >> > > > > > > > >> > Hello Alexander, >> > >> > > > > > > > >> > I am Onkar from PICT, Pune India. I am interested >> in >> > >> the >> > >> > > > project >> > >> > > > > > > idea >> > >> > > > > > > > >> > regarding Notebook distributed Storage using P2P >> > >> > protocols. >> > >> > > > > > > > >> > In order to contribute and aid in this project, I >> > have >> > >> > been >> > >> > > > > > working >> > >> > > > > > > > with >> > >> > > > > > > > >> > Zeppelin Notebooks.This is a link to one of my >> sample >> > >> > > notebook >> > >> > > > > > which >> > >> > > > > > > > >> uses a >> > >> > > > > > > > >> > dataset about Indian school data from data.gov.in >> : >> > >> > > > github-repo >> > >> > > > > > > > >> > < >> > >> > > > > > > > >> > >> > >> > > > > > > > >> >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > >> https://github.com/onkarshedge/sample-notebooks/blob/master/2BFYFUZCP/note.json >> > >> > > > > > > > >> > > >> > >> > > > > > > > >> > . >> > >> > > > > > > > >> > >> > >> > > > > > > > >> > I am familiar with IPython it also uses similar >> > >> > json(.ipynb) >> > >> > > > way >> > >> > > > > > to >> > >> > > > > > > > >> > represent notebook. So as per my understanding we >> > have >> > >> to >> > >> > > > divide >> > >> > > > > > the >> > >> > > > > > > > >> json >> > >> > > > > > > > >> > file into chunks and store in a distributed manner >> > >> > according >> > >> > > > to >> > >> > > > > > > > >> protocol. >> > >> > > > > > > > >> > While I am familiar with the basics of the product >> > and >> > >> > have >> > >> > > a >> > >> > > > > > clear >> > >> > > > > > > > >> idea of >> > >> > > > > > > > >> > what is required by the problem statement, I am >> not >> > >> quite >> > >> > > sure >> > >> > > > > how >> > >> > > > > > > to >> > >> > > > > > > > >> > proceed about it. I would appreciate your guidance >> > >> > regarding >> > >> > > > the >> > >> > > > > > > > same. I >> > >> > > > > > > > >> > was thinking about starting with a brief >> comparative >> > >> study >> > >> > > of >> > >> > > > > the >> > >> > > > > > > > >> protocols >> > >> > > > > > > > >> > suggested( dat, ipfs, zeronet). I hope to hear >> your >> > >> views >> > >> > > > about >> > >> > > > > > > this. >> > >> > > > > > > > >> > >> > >> > > > > > > > >> > Thanking you, >> > >> > > > > > > > >> > Onkar Shedge >> > >> > > > > > > > >> > >> > >> > > > > > > > >> >> > >> > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > >> > >> > > >> > >> > >> > >> >> > > >> > > >> > >> > >
