[
https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14000121#comment-14000121
]
wolfgang hoschek commented on HBASE-10504:
------------------------------------------
+1 to what [~greid] already said.
To help illustrate some of the benefits of process isolation take a look at the
large and complex dependency tree of hbase-indexer below. I can't imagine
having something like this run inside of the primary region server process
without causing ongoing stability, scalability, reliability, flexibility and
resource management issues for HBase.
In addition, being able to decouple the lifecycle of the hbase-indexer
processes and their configuration from the primary region server processes
further contributes to reliability and flexibility and ease of management. It
also means we can scale out the Indexer tier more independently from the HBase
tier without violating resource management SLAs. For example, indexing can suck
quite some CPU.
When this JIRA was originally filed we were hoping for it to help ease some of
the hbase-indexer pain, but at the moment it looks like it's going in a
different direction, a direction that doesn't apply to our use case even though
it may well be great for other use cases, and, at least from our perspective,
I'm afraid it would be more like a step backwards.
Thoughts?
{code}
[INFO] --- maven-dependency-plugin:2.1:tree (default-cli) @
hbase-indexer-server ---
[INFO] com.ngdata:hbase-indexer-server:jar:1.5-SNAPSHOT
[INFO] +- com.ngdata:hbase-indexer-engine:jar:1.5-SNAPSHOT:compile
[INFO] | +- com.ngdata:hbase-indexer-common:jar:1.5-SNAPSHOT:compile
[INFO] | | +- com.ngdata:hbase-sep-api:jar:1.5-SNAPSHOT:compile
[INFO] | | +- com.ngdata:hbase-sep-impl:jar:1.5-hbase0.98-SNAPSHOT:compile
[INFO] | | | \- com.ngdata:hbase-sep-impl-common:jar:1.5-SNAPSHOT:compile
[INFO] | | \- org.apache.solr:solr:war:4.4.0:compile
[INFO] | +- com.ngdata:hbase-indexer-model:jar:1.5-SNAPSHOT:compile
[INFO] | | \- net.iharder:base64:jar:2.3.8:compile
[INFO] | +- org.apache.solr:solr-core:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-core:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-codecs:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-analyzers-common:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-analyzers-kuromoji:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-analyzers-phonetic:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-highlighter:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-memory:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-misc:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-queryparser:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-spatial:jar:4.4.0:compile
[INFO] | | | \- com.spatial4j:spatial4j:jar:0.3:compile
[INFO] | | +- org.apache.lucene:lucene-suggest:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-grouping:jar:4.4.0:compile
[INFO] | | +- org.apache.lucene:lucene-queries:jar:4.4.0:compile
[INFO] | | +-
com.googlecode.concurrentlinkedhashmap:concurrentlinkedhashmap-lru:jar:1.2:compile
[INFO] | | +- commons-fileupload:commons-fileupload:jar:1.2.1:compile
[INFO] | | +- joda-time:joda-time:jar:1.6:compile (version managed from 2.2)
[INFO] | | +- org.restlet.jee:org.restlet:jar:2.1.1:compile
[INFO] | | +- org.restlet.jee:org.restlet.ext.servlet:jar:2.1.1:compile
[INFO] | | +- org.codehaus.woodstox:wstx-asl:jar:3.2.7:runtime
[INFO] | | +- org.apache.httpcomponents:httpclient:jar:4.2.5:compile
[INFO] | | | \- org.apache.httpcomponents:httpcore:jar:4.2.5:compile
(version managed from 4.2.4)
[INFO] | | \- org.apache.httpcomponents:httpmime:jar:4.2.5:compile
[INFO] | +- org.apache.solr:solr-solrj:jar:4.4.0:compile
[INFO] | | \- org.noggit:noggit:jar:0.5:compile
[INFO] | +- org.apache.solr:solr-cell:jar:4.4.0:compile
[INFO] | | \- org.apache.tika:tika-parsers:jar:1.4:compile
[INFO] | | +- org.apache.tika:tika-core:jar:1.4:compile
[INFO] | | +- org.gagravarr:vorbis-java-tika:jar:0.1:compile
[INFO] | | +- edu.ucar:netcdf:jar:4.2-min:compile
[INFO] | | +- org.apache.james:apache-mime4j-core:jar:0.7.2:compile
[INFO] | | +- org.apache.james:apache-mime4j-dom:jar:0.7.2:compile
[INFO] | | +- org.apache.pdfbox:pdfbox:jar:1.8.1:compile
[INFO] | | | +- org.apache.pdfbox:fontbox:jar:1.8.1:compile
[INFO] | | | \- org.apache.pdfbox:jempbox:jar:1.8.1:compile
[INFO] | | +- org.bouncycastle:bcmail-jdk15:jar:1.45:compile
[INFO] | | +- org.bouncycastle:bcprov-jdk15:jar:1.45:compile
[INFO] | | +- org.apache.poi:poi:jar:3.9:compile
[INFO] | | +- org.apache.poi:poi-scratchpad:jar:3.9:compile
[INFO] | | +- org.apache.poi:poi-ooxml:jar:3.9:compile
[INFO] | | | +- org.apache.poi:poi-ooxml-schemas:jar:3.9:compile
[INFO] | | | | \- org.apache.xmlbeans:xmlbeans:jar:2.3.0:compile
[INFO] | | | \- dom4j:dom4j:jar:1.6.1:compile
[INFO] | | +- org.ccil.cowan.tagsoup:tagsoup:jar:1.2.1:compile
[INFO] | | +- com.googlecode.mp4parser:isoparser:jar:1.0-RC-1:compile
[INFO] | | +- com.drewnoakes:metadata-extractor:jar:2.6.2:compile
[INFO] | | +- de.l3s.boilerpipe:boilerpipe:jar:1.1.0:compile
[INFO] | | +- rome:rome:jar:0.9:compile
[INFO] | | | \- jdom:jdom:jar:1.0:compile
[INFO] | | \-
com.googlecode.juniversalchardet:juniversalchardet:jar:1.0.3:compile
[INFO] | +- com.google.guava:guava:jar:11.0.2:compile
[INFO] | | \- com.google.code.findbugs:jsr305:jar:1.3.9:compile
[INFO] | \- com.yammer.metrics:metrics-core:jar:2.2.0:compile
[INFO] +- com.ngdata:hbase-indexer-mr:jar:1.5-SNAPSHOT:compile
[INFO] | +- com.cloudera.search:search-mr:jar:1.0.0-cdh5.0.0:compile
[INFO] | | +- org.apache.tika:tika-xmp:jar:1.4:compile
[INFO] | | | \- com.adobe.xmp:xmpcore:jar:5.1.2:compile
[INFO] | | \- net.sourceforge.argparse4j:argparse4j:jar:0.4.3:compile
[INFO] | \- org.apache.zookeeper:zookeeper:jar:3.4.5-cdh4.4.0:compile
[INFO] | \- jline:jline:jar:0.9.94:compile
[INFO] +- com.ngdata:hbase-indexer-morphlines:jar:1.5-SNAPSHOT:compile
[INFO] | \- org.kitesdk:kite-morphlines-all:pom:0.13.0:compile (version
managed from 0.10.0-cdh5.0.0)
[INFO] | +- org.kitesdk:kite-morphlines-core:jar:0.13.0:compile
[INFO] | | +- com.typesafe:config:jar:1.0.2:compile
[INFO] | | +- com.codahale.metrics:metrics-core:jar:3.0.1:compile
[INFO] | | \- com.codahale.metrics:metrics-healthchecks:jar:3.0.1:compile
[INFO] | +- org.kitesdk:kite-morphlines-avro:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-json:jar:0.13.0:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-databind:jar:2.3.1:compile
[INFO] | | +-
com.fasterxml.jackson.core:jackson-annotations:jar:2.3.0:compile
[INFO] | | \- com.fasterxml.jackson.core:jackson-core:jar:2.3.1:compile
[INFO] | +- org.kitesdk:kite-morphlines-saxon:jar:0.13.0:compile
[INFO] | | \- net.sf.saxon:Saxon-HE:jar:9.5.1-5:compile
[INFO] | +- org.kitesdk:kite-morphlines-hadoop-core:jar:0.13.0:compile
[INFO] | | \- org.kitesdk:kite-hadoop-compatibility:jar:0.13.0:compile
[INFO] | +-
org.kitesdk:kite-morphlines-hadoop-parquet-avro:jar:0.13.0:compile
[INFO] | | \- com.twitter:parquet-avro:jar:1.2.5:compile
[INFO] | | +- com.twitter:parquet-column:jar:1.2.5:compile
[INFO] | | | +- com.twitter:parquet-common:jar:1.2.5:compile
[INFO] | | | \- com.twitter:parquet-encoding:jar:1.2.5:compile
[INFO] | | | \- com.twitter:parquet-generator:jar:1.2.5:compile
[INFO] | | +- com.twitter:parquet-hadoop:jar:1.2.5:compile
[INFO] | | \- com.twitter:parquet-format:jar:1.0.0:compile
[INFO] | +-
org.kitesdk:kite-morphlines-hadoop-sequencefile:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-hadoop-rcfile:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-tika-core:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-tika-decompress:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-solr-core:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-solr-cell:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-twitter:jar:0.13.0:compile
[INFO] | +- org.kitesdk:kite-morphlines-maxmind:jar:0.13.0:compile
[INFO] | | \- com.maxmind.db:maxmind-db:jar:0.3.1:compile
[INFO] | +- org.kitesdk:kite-morphlines-metrics-servlets:jar:0.13.0:compile
[INFO] | | +- com.codahale.metrics:metrics-servlets:jar:3.0.1:compile
[INFO] | | | +- com.codahale.metrics:metrics-json:jar:3.0.1:compile
[INFO] | | | \- com.codahale.metrics:metrics-jvm:jar:3.0.1:compile
[INFO] | | \- org.eclipse.jetty:jetty-servlet:jar:8.1.14.v20131031:compile
[INFO] | | \-
org.eclipse.jetty:jetty-security:jar:8.1.14.v20131031:compile
[INFO] | | \-
org.eclipse.jetty:jetty-server:jar:8.1.14.v20131031:compile
[INFO] | | +-
org.eclipse.jetty.orbit:javax.servlet:jar:3.0.0.v201112011016:compile
[INFO] | | +-
org.eclipse.jetty:jetty-continuation:jar:8.1.14.v20131031:compile
[INFO] | | \-
org.eclipse.jetty:jetty-http:jar:8.1.14.v20131031:compile
[INFO] | | \-
org.eclipse.jetty:jetty-io:jar:8.1.14.v20131031:compile
[INFO] | | \-
org.eclipse.jetty:jetty-util:jar:8.1.14.v20131031:compile
[INFO] | \- org.kitesdk:kite-morphlines-useragent:jar:0.13.0:compile
[INFO] | \- ua_parser:ua-parser:jar:1.3.0:compile
[INFO] | \- org.yaml:snakeyaml:jar:1.10:compile
[INFO] +- org.apache.hadoop:hadoop-core:jar:2.0.0-mr1-cdh4.4.0:compile
[INFO] | +- commons-cli:commons-cli:jar:1.2:compile
[INFO] | +- xmlenc:xmlenc:jar:0.52:compile
[INFO] | +- commons-httpclient:commons-httpclient:jar:3.1:compile
[INFO] | +- commons-codec:commons-codec:jar:1.4:compile
[INFO] | +- commons-net:commons-net:jar:3.1:compile
[INFO] | +- org.mortbay.jetty:jetty:jar:6.1.26.cloudera.2:compile
[INFO] | +- org.mortbay.jetty:jetty-util:jar:6.1.26.cloudera.2:compile
[INFO] | +- tomcat:jasper-runtime:jar:5.5.23:compile
[INFO] | +- tomcat:jasper-compiler:jar:5.5.23:compile
[INFO] | +- org.codehaus.jackson:jackson-core-asl:jar:1.8.8:compile
[INFO] | +- org.codehaus.jackson:jackson-mapper-asl:jar:1.8.8:compile
[INFO] | +- javax.servlet.jsp:jsp-api:jar:2.1:compile
[INFO] | +- commons-el:commons-el:jar:1.0:compile
[INFO] | +- net.java.dev.jets3t:jets3t:jar:0.6.1:compile
[INFO] | +- javax.servlet:servlet-api:jar:2.5:compile
[INFO] | +- hsqldb:hsqldb:jar:1.8.0.10:compile
[INFO] | \- org.eclipse.jdt:core:jar:3.1.1:compile
[INFO] +- org.slf4j:slf4j-log4j12:jar:1.7.5:compile
[INFO] | +- org.slf4j:slf4j-api:jar:1.7.5:compile
[INFO] | \- log4j:log4j:jar:1.2.17:compile
[INFO] +- org.slf4j:jcl-over-slf4j:jar:1.7.5:compile
[INFO] +- org.slf4j:jul-to-slf4j:jar:1.7.5:compile
[INFO] +- com.yammer.metrics:metrics-ganglia:jar:2.2.0:compile
[INFO] +- junit:junit:jar:4.8.1:test
[INFO] +- org.apache.hadoop:hadoop-test:jar:2.3.0-mr1-cdh5.1.0-SNAPSHOT:test
[INFO] | +- org.apache.ftpserver:ftplet-api:jar:1.0.0:test
[INFO] | +- org.apache.mina:mina-core:jar:2.0.0-M5:test
[INFO] | +- org.apache.ftpserver:ftpserver-core:jar:1.0.0:test
[INFO] | \- org.apache.ftpserver:ftpserver-deprecated:jar:1.0.0-M2:test
[INFO] +- org.mockito:mockito-core:jar:1.9.0:test
[INFO] | +- org.hamcrest:hamcrest-core:jar:1.1:test
[INFO] | \- org.objenesis:objenesis:jar:1.0:test
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:2.3.0-cdh5.1.0-SNAPSHOT:compile
[INFO] | +- commons-io:commons-io:jar:2.4:compile
[INFO] | +- commons-lang:commons-lang:jar:2.6:compile
[INFO] | +- commons-logging:commons-logging:jar:1.1.3:compile
[INFO] | +- commons-daemon:commons-daemon:jar:1.0.13:compile
[INFO] | \- com.google.protobuf:protobuf-java:jar:2.5.0:compile
[INFO] +- com.sun.jersey:jersey-server:jar:1.17:compile
[INFO] | \- asm:asm:jar:3.1:compile
[INFO] +- com.sun.jersey:jersey-servlet:jar:1.17:compile
[INFO] +- com.sun.jersey:jersey-core:jar:1.17:compile
[INFO] +- javax.ws.rs:jsr311-api:jar:1.1.1:compile
[INFO] +- org.kohsuke:akuma:jar:1.9:compile
[INFO] | \- net.java.dev.jna:jna:jar:3.4.0:compile
[INFO] +- org.apache.hadoop:hadoop-hdfs:jar:tests:2.3.0-cdh5.1.0-SNAPSHOT:test
[INFO] +- org.apache.hadoop:hadoop-common:jar:tests:2.3.0-cdh5.1.0-SNAPSHOT:test
[INFO] +-
org.apache.hbase:hbase-server:test-jar:tests:0.98.1-cdh5.1.0-SNAPSHOT:test
[INFO] | \-
org.apache.hadoop:hadoop-hdfs:test-jar:tests:2.3.0-cdh5.1.0-SNAPSHOT:test
[INFO] +-
org.apache.hbase:hbase-hadoop2-compat:test-jar:tests:0.98.1-cdh5.1.0-SNAPSHOT:test
[INFO] +-
org.apache.hbase:hbase-hadoop-compat:test-jar:tests:0.98.1-cdh5.1.0-SNAPSHOT:test
[INFO] +- org.apache.hbase:hbase-server:jar:0.98.1-cdh5.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.hbase:hbase-common:jar:0.98.1-cdh5.1.0-SNAPSHOT:compile
[INFO] | +-
org.apache.hbase:hbase-protocol:jar:0.98.1-cdh5.1.0-SNAPSHOT:compile
[INFO] | +- org.apache.hbase:hbase-client:jar:0.98.1-cdh5.1.0-SNAPSHOT:compile
[INFO] | | \- io.netty:netty:jar:3.6.6.Final:compile
[INFO] | +-
org.apache.hbase:hbase-prefix-tree:jar:0.98.1-cdh5.1.0-SNAPSHOT:runtime
[INFO] | +- commons-collections:commons-collections:jar:3.2.1:compile
[INFO] | +-
org.apache.hbase:hbase-hadoop-compat:jar:0.98.1-cdh5.1.0-SNAPSHOT:compile
[INFO] | +-
org.apache.hbase:hbase-hadoop2-compat:jar:0.98.1-cdh5.1.0-SNAPSHOT:compile
[INFO] | +- com.github.stephenc.high-scale-lib:high-scale-lib:jar:1.1.1:compile
[INFO] | +- org.apache.commons:commons-math:jar:2.1:compile
[INFO] | +- org.mortbay.jetty:jetty-sslengine:jar:6.1.26.cloudera.2:compile
[INFO] | +- org.mortbay.jetty:jsp-2.1:jar:6.1.14:compile
[INFO] | +- org.mortbay.jetty:jsp-api-2.1:jar:6.1.14:compile
[INFO] | +- org.mortbay.jetty:servlet-api-2.5:jar:6.1.14:compile
[INFO] | +- org.codehaus.jackson:jackson-jaxrs:jar:1.8.8:compile
[INFO] | +- org.jamon:jamon-runtime:jar:2.3.1:compile
[INFO] | +- com.sun.jersey:jersey-json:jar:1.17:compile (version managed from
1.8)
[INFO] | | +- org.codehaus.jettison:jettison:jar:1.1:compile
[INFO] | | +- com.sun.xml.bind:jaxb-impl:jar:2.2.3-1:compile
[INFO] | | \- org.codehaus.jackson:jackson-xc:jar:1.9.2:compile
[INFO] | +- javax.xml.bind:jaxb-api:jar:2.1:compile
[INFO] | | \- javax.activation:activation:jar:1.1:compile
[INFO] | +- org.cloudera.htrace:htrace-core:jar:2.04:compile
[INFO] | +- org.apache.hadoop:hadoop-common:jar:2.3.0-cdh5.1.0-SNAPSHOT:compile
[INFO] | | +- org.apache.commons:commons-math3:jar:3.1.1:compile
[INFO] | | +- commons-configuration:commons-configuration:jar:1.6:compile
[INFO] | | | +- commons-digester:commons-digester:jar:1.8:compile
[INFO] | | | | \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
[INFO] | | | \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
[INFO] | | +- org.apache.avro:avro:jar:1.7.5-cdh5.1.0-SNAPSHOT:compile
[INFO] | | | +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
[INFO] | | | \- org.xerial.snappy:snappy-java:jar:1.0.5:compile
[INFO] | | +- com.jcraft:jsch:jar:0.1.42:compile
[INFO] | | \- org.apache.commons:commons-compress:jar:1.4.1:compile
[INFO] | | \- org.tukaani:xz:jar:1.0:compile
[INFO] | +- org.apache.hadoop:hadoop-auth:jar:2.3.0-cdh5.1.0-SNAPSHOT:compile
[INFO] | +-
org.apache.hadoop:hadoop-annotations:jar:2.3.0-cdh5.1.0-SNAPSHOT:compile
[INFO] | | \- jdk.tools:jdk.tools:jar:1.7:system
[INFO] | \-
com.github.stephenc.findbugs:findbugs-annotations:jar:1.3.9-1:compile
[INFO] +-
org.apache.hbase:hbase-client:test-jar:tests:0.98.1-cdh5.1.0-SNAPSHOT:test
[INFO] \-
org.apache.hbase:hbase-common:test-jar:tests:0.98.1-cdh5.1.0-SNAPSHOT:test
{code}
> Define Replication Interface
> ----------------------------
>
> Key: HBASE-10504
> URL: https://issues.apache.org/jira/browse/HBASE-10504
> Project: HBase
> Issue Type: Task
> Reporter: stack
> Assignee: stack
> Priority: Blocker
> Fix For: 0.99.0
>
> Attachments: hbase-10504_wip1.patch
>
>
> HBase has replication. Fellas have been hijacking the replication apis to do
> all kinds of perverse stuff like indexing hbase content (hbase-indexer
> https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/
> overrides that replicate via an alternate channel (over a secure thrift
> channel between dcs over on HBASE-9360). This issue is about surfacing these
> APIs as public with guarantees to downstreamers similar to those we have on
> our public client-facing APIs (and so we don't break them for downstreamers).
> Any input [~phunt] or [~gabriel.reid] or [~toffer]?
> Thanks.
>
--
This message was sent by Atlassian JIRA
(v6.2#6252)
