Hello, To get the merging process going, I have submitted my first patch. It is available for review at https://reviews.apache.org/r/12798/.
I have created a local branch called 'hadoop-eclipse-merge', which has been forked off from the following commit: ----------------------------------------------------------- commit 0a0744f645bad8b646bafb4f1261bdbf2ddce70a Author: Adam Berry <[email protected]> Date: Tue Jan 15 10:48:07 2013 -0600 markdown changes for navigation bar, and contributing page all with hadoop branding. ----------------------------------------------------------- I have attached the patch to https://issues.apache.org/jira/browse/HDT-32also. The patch creates the following projects: - org.apache.hdt.feature: HDT feature - org.apache.hdt.core: HDT Core/Headless functionality - org.apache.hdt.ui: HDT UI functionality - org.apache.hdt.hadoop.release: Eclipse Fragment per Hadoop release. - org.apache.hdt.updateSite: HDT update sites - org.apache.hdt.ui.test: Test plugin containing all core/UI tests With the new plugins/fragments, I was able to goto Hadoop Perspective, create new HDFS/ZooKeeper connections, view data hierarchy in both, view property sheets, etc. Please let me know if anyone is having any major issues. Any minor issues, please open JIRAs and tag them as 'hadoop-eclipse-merge'. Hoping to get to a combined, stable state for this stream. Best regards, Srimanth On Thu, Jul 11, 2013 at 2:51 PM, Adam Berry <[email protected]> wrote: > Hey guys, > > so for background, the intent for this project was always to get to where > we can connect to multiple versions of Hadoop clusters from one dev tools > install. > http://wiki.apache.org/incubator/HadoopDevelopmentToolsProposalfor > a little reference. > > The code as it stands out is basically a first pass at splitting the code > from Hadoop contrib, there it resided as a single plug-in, with all the > same client library limitations. And yes, you cannot connect to multiple > versions of Hadoop etc, and the logic is pretty coupled in some spots. > > Personally, I'm all for pulling in this new work to for HDFS and zookeeper > work, and then bringing the MR side up to the same point with the same > approach, as Srimanth suggested. > > I also vote for doing this in a feature branch in our git repo, and if no > one objects I'll get that setup so we can get to work. > > Cheers, > Adam > > > On Tue, Jul 9, 2013 at 2:35 AM, Srimanth Gunturi <[email protected]> > wrote: > > > Hi Rahul, > > Hadoop-Eclipse UI is decoupled from models and controllers. > > EMF models automatically provide model and controller separation. > > That work is already done. If we reused the available code as is, we do > not > > have to rewrite a major bulk of the functionality. > > > > IMHO, the least amount of work in getting both projects merged, is > putting > > effort into moving MR core/UI functionality onto the above plugins > > following the same paradigms. If we did this, we wouldnt have to rewrite > > HDFS, and the underlying internals/UI, as they are already working. > > Best regards, > > Srimanth > > > > > > > > > > On Mon, Jul 8, 2013 at 11:02 PM, Rahul Sharma <[email protected]> > wrote: > > > > > +1 to the idea to have some abstraction to access HDFS/Zookeeper. We > > could > > > make different implementations for different versions. As for complete > UI > > > decoupling , I think we would like to achieve it. My only concern here > > is > > > that this looks like complete overwrite as the current version UI is > > > tightly coupled with logic. We should try to distribute this in > multiple > > > releases. I will dive into hadoop-eclipse some time this week and share > > my > > > thoughts on the same J > > > > > > > > > regards, > > > Rahul > > > > > > > > > On Mon, Jul 8, 2013 at 11:49 AM, Srimanth Gunturi <[email protected] > > > >wrote: > > > > > > > Hello, > > > > With regards to contributing HDFS/ZooKeeper functionalities, I was > > going > > > > through HDT code and noted some design issues/thoughts that I wanted > to > > > > discuss. > > > > > > > > 1) Client cannot connect with multiple versions of HDFS/MR servers > > > > org.apache.hdt.core.cluster.HadoopCluster which represents a cluster, > > > > provides direct access to the HDFS and MR java API. This implies that > > at > > > > any time, only 1 version of HDFS and MR client libraries can be used > > > > (typically, whichever version gets loaded first by the classloader). > So > > > if > > > > there was any use case where interactions with multiple HDFS/MR > > versions > > > is > > > > required, we would hit runtime issues. The client would be at the > mercy > > > of > > > > backward/forward compatibility capabilities of the > > HDFS/MR/[any-service] > > > > clients. > > > > > > > > In the Hadoop-Eclipse project, to get around this issue, I have > created > > > an > > > > extension-point based abstraction, where the Eclipse functionality > > itself > > > > would never directly use HDFS/ZooKeeper/[service] classes. Rather, > from > > > > multiple versions of extension point implementations, the right one > > would > > > > be used to talk to the server. This allows the UI/core(headless) > > > > functionalities to be free from the ever changing versions of > > > > clients/servers. > > > > > > > > > > > > 2) No clean seperation of UI, non-UI capabilities. > > > > In HDFS, almost all functionality is non-UI (create, read, write, > > delete > > > of > > > > files/folders). However, currently all HDT plugins are dependent on > UI > > > > plugins (starting with org.apache.hdt.core). This goes against the > > > > model-view-controller (MVC) paradigm, where the Eclipse UI (view) is > > > mixed > > > > in with the models and controllers. There is no reason why someone > > could > > > > not leverage or extend the core/headless/non-UI capabilities of > various > > > > Hadoop services in Eclipse without the UI. > > > > > > > > In the Hadoop-Eclipse project, plugins are categorized into core > > > > (representing non-UI capabilities) and UI plugins. You can create > > > > connections, create/read/write/delete HDFS/ZooKeeper contents, etc., > > > > without even having UI plugins. This is helpful in nightly JUnit > tests > > to > > > > start. But it also allows others to provide their own UI interactions > > on > > > > top of us. The models that are persisted (HDFS/ZooKeeper connections, > > > > metadata, etc.) are Eclipse Modeling Framework > > > > (EMF)<http://www.eclipse.org/modeling/emf/>models, which have a > > > > built-in notification mechanism. They help in a clean > > > > separation of Models and Controllers in MVC. > > > > > > > > > > > > The above were some of the major ones which came to mind. > > > > I encourage the community to go through the > > > > Hadoop-Eclipse<http://people.apache.org/~srimanth/hadoop-eclipse/ > > > >project > > > > codebase, and discuss any issues/concerns you have. > > > > > > > > I am thinking of the best way to merge the functionalities of both > > > > projects, and would like to put forward a proposal. > > > > HDFS is the only functionality common between both projects, along > with > > > > underlying framework. If we can come to a consensus on which parts we > > > want > > > > from where, it will be a smoother effort merging the code. From my > end > > of > > > > the spectrum, I was thinking it might be easier if the MR > functionality > > > > could be merged into the HDFS/ZooKeeper functionalities, thus > > providing a > > > > union of both projects. > > > > > > > > I just wanted to get the merging process started, and look forward to > > > > discussing more about it. > > > > Best regards, > > > > Srimanth > > > > > > > > > >
