Jesse,
The only difference between a bundle and a .jar is the inclusion of additional fields in the MANIFEST.MF file. I think there's a way to allow the maven-bundle-plugin to create this file, and then to include it into a .jar file created by the maven-jar-plugin. Basically, after the code is compiled, the maven-bundle-plugin goes through all of the compiled .class files, and creates a set of Import-Package statements. Then, you can either have the bundle plugin export everything automatically, or specificy the packages you want to export. If you'd like, tonight I can implement this in the parent pom.xml file and submit a patch for your review. Also, did anyone take a look at the Jira ticket I created last night referencing exec plugin attempting to find /bin/bash? I'd really like to be able to compile accumulo and get it working in a stand-alone mode with a single hadoop and zookeeper instance to allow me to test the camel-accumulo component once it is complete. Lastly, do you folks envision creating a "sandbox" area in your svn repository? This would allow me to commit the camel component along with examples and tests to allow you folks to review it. ----- Original Message ----- From: "Jesse Yates" <[email protected]> To: [email protected] Sent: Wednesday, October 26, 2011 10:37:16 AM Subject: Re: Camel Accumulo Comments inline. ------------------- Jesse Yates 240-888-2200 @jesse_yates On Wed, Oct 26, 2011 at 6:39 AM, <[email protected]> wrote: > > > Eric, > > > > I'm confused about the need for a username/password. Does Accumulo validate > against that, or does it accept a set of priveledges for a user? It checks that user and password against an internal db of users, giving the user ability to read from a subset of the permissions on the system ('Authorizations' in Accumulo parlance). > Also, in your discussion of zookeeper, what does the "zookeepers" attribute > represent? > The address of the zookeeper servers keeping track of the accumulo instance. This is the 'root' truth of where things are for clients. > > > > For my first stab at this, I'm using the ReadandWrite example to show me > how to connect to Accumulo and perform reads/writes . Is there a better one > that I should be using? This example class seems pretty heavy-weight... > > It might be a little bit heavy, but I would rather have a complete example, than a bunch of scattered sub-pieces. I think once you have the connection setup, the actual read/write shouldn't be too bad. > > > For the table name, I'd like to keep the camel component as generic as > possible. So, to allow for this, I'm going to require folks to pass in the > table name on their accumulate url. Is this ok? > > > > Full Ack on the "accumulate" versus "accum" uri. The only reason I > attempted to shorten it is that looks like the trend for other > camel-components. That said, I can live with a longer name if it makes it > easier to use. Along these lines, I've thought that having a different uri > for reads, writes, and mutates would be an easy way to handle interactions > through the camel component. To help with this, I'm thinking that he > following uri's would be use ful (developed in this order) : > > accumulate-read > > accumulate-write > > accumulate-mutate > > accumulate-batch-read > > accumulate-batch-write > > > > How does that sound? > shouldn't this be 'accumulo' or is this for implying accumulation of information via Camel? > > > > Also, how do you feel about changing the build so that it creates bundles > instead of .jar files? All a bundle is, is a .jar file with a MANIFEST.MF > file that has some extra attributes. The maven-bundle-plugin can take care > of that pretty easily, but it will change the packaging type from jar to > bundle. This would really only impact folks who are looking for a packaging > type in thier .pom file dependencies, which is not common. > > I would argue vehemently against changing the core build for an external process (no offense mike). I think it would be okay if we have a profile in the build that builds a bundle, but since that is a pretty uncommon distro, people will easily get confused, leading to lower adoption, lots of questions on user@, etc. Yes, its a low likelyhood, but I would rather make it is as easy as possible to 'do the right thing'. Camel (and other systems that need a bundle) are a special case, so already people are going to be be a little more advanced and expect slightly 'special' versions. > > > Mike Van > > > > ----- Original Message ----- > > > From: "Eric Newton" <[email protected]> > To: [email protected] > Sent: Wednesday, October 26, 2011 9:06:19 AM > Subject: Re: Camel Accumulo > > Hi Mike, > > Scan auths are needed for reading, not for writing. > > Looking at constructors and factory methods looks like we'll need: > > ZooKeeperInstance: > zookeepers > instance name > > getConnector(): > username > password > > getMultiTableBatchWriter(): > memory > latency > write threads > > This would allow you to stream in tuples of (table, row, cf, cq, > visibility, > timestamp). If you already have a connector for HBase, we would probably > want something compatible with that, and that is probably table oriented > (just guessing, being unfamiliar with Camel). So, perhaps if you specify a > table you would accept tuples of (row, cf, cq, visibility, timestamp). You > could make visibility and timestamp optional, too. > > A minimal uri would look something like this? You could provide reasonable > defaults for the other arguments. > > > accumulo://instance/write?zookeeper=zoohost&userName=root&password=secret > > Unfortunately, "accumulo" doesn't have a nice short abbreviation. I would > lean towards using the whole word, but that's just a personal preference. > > -Eric > > On Wed, Oct 26, 2011 at 12:17 AM, <[email protected]> wrote: > > > > > > > Hi there. > > > > > > > > I recently read about your project and like the direction it is taking. > > Currently, I am a committer to another incubator project, Kalumet, and > have > > been a long-time contributor to the Karaf and Camel projects. After > talking > > with a few of the Accumulo project members, it looks like the most > immediate > > hurdle is growing the user community. I believe I can help with that. > > > > > > > > Making associations between incubator projects and top-level-projects has > > been a proven mechanism to pique developer interest and garner more > > dedicated contributors and commiters. Because of the wide integration of > > Camel and NOSQL databases, creating a Camel component for interaction > with > > Accumulo seems like a no-brainer. > > > > > > > > In order to help grow the Accumulo community, tonight I began writing a > > camel-accumulo component. This will allow Camel users to route files to > > Accumulo in the same manner as they currently route files to HDFS . > > > > > > > > > > > > For some background, Camel is the open-source implementation of > Enterprise > > Integration Patterns. Most modern ESB's use OSGI and Camel to perform > > routing and orchestration of data through endpoints. Camel has been > written > > to allow various technologies to create Camel Components that folks can > use > > when they define the route that a given file or data will be processed > > through. In this model, users would define a "route" in Camel that > contains > > various Accumulo endpoints for reading, writing, or mutating data > persisted > > through Accumulo. To make this work, I need to define a URI that folks > will > > use. Would you folks be able to help me define the URI and URL > parameters? > > > > > > > > Right now, I'm using the URI "accum". For the first iteration of this > > component, I'm thinking it would be simplest to create an endpoint folks > > could write a single re cord to. Then, follow it up with a scan, and > mutate > > components. Once these are done, I'd like to do the batch-versions of > these > > operations. > > > > > > > > In Camel, an endpoint usually looks similar to a web-service endpoint: > > URI://location/service ?[arg1=value1][&arg<x>=value<x>] > > > > > > > > With this in mind, I'm thinking the following would be the minimally > > acceptable Camel-Accumulo endpoint for simplistic write operations: > > > > accum://location /write ?zookeeperName=value&\ > > > > tableName=value& \ > > > > userName=value&\ > > > > password=value&\ > > > > userPrivs=value&\ > > > > scanAuths=value&\ > > > > debug=value > > > > > > > > Does this contain a c omplete listing of the properties? A m I missing > > anything, did I put something in that's not needed, or are there other > > options a user should be able to pass in? Also, is the URI of "accum" ok > for > > this camel component? > > > > > > > > Because Camel is written to play nicely inside of OSGI (like > Karaf/Felix), > > the .jar files this camel component relies on should be bundle-ized. This > > shouldn't be too hard to do, and as a Karaf contributor, I've done this > with > > hundreds of third party .jar files. Basically, we would replace the > > maven-jar-plugin with a light implementation of the maven-bundle-plugin > > along with some fairly generic attributes. If you folks would like, I > can > > do this for you on a seperate branch so that you can test it. > > > > > > > > Mike Van > > > > Committer - ASF Kalumet > > > > Contributor - ASF Karaf, Camel >
