[ https://issues.apache.org/jira/browse/HBASE-4047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499401#comment-13499401 ]
Asaf Mesika commented on HBASE-4047: ------------------------------------ This truly sounds like a great feature. I was wondering: * Did you find any performance penalties for shifting data back and forth between the processes? Did you this using the loopback interface? * What method did you choose to communicate between those processes? TCP? Output stream piping? > [Coprocessors] Generic external process host > -------------------------------------------- > > Key: HBASE-4047 > URL: https://issues.apache.org/jira/browse/HBASE-4047 > Project: HBase > Issue Type: New Feature > Components: Coprocessors > Reporter: Andrew Purtell > Assignee: Andrew Purtell > > Where HBase coprocessors deviate substantially from the design (as I > understand it) of Google's BigTable coprocessors is we've reimagined it as a > framework for internal extension. In contrast BigTable coprocessors run as > separate processes colocated with tablet servers. The essential trade off is > between performance, flexibility and possibility, and the ability to control > and enforce resource usage. > Since the initial design of HBase coprocessors some additional considerations > are in play: > - Developing computational frameworks sitting directly on top of HBase hosted > in coprocessor(s); > - Introduction of the map reduce next generation (mrng) resource management > model, and the probability that limits will be enforced via cgroups at the OS > level after this is generally available, e.g. when RHEL 6 deployments are > common; > - The possibility of deployment of HBase onto mrng-enabled Hadoop clusters > via the mrng resource manager and a HBase-specific application controller. > Therefore we should consider developing a coprocessor that is a generic host > for another coprocessor, but one that forks a child process, loads the target > coprocessor into the child, establishes a bidirectional pipe and uses an > eventing model and umbilical protocol to provide for the coprocessor loaded > into the child the same semantics as if it was loaded internally to the > parent, and (eventually) use available resource management capabilities on > the platform -- perhaps via the mrng resource controller or directly with > cgroups -- to limit the child as desired by system administrators or the > application designer. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira