This is an automated email from the ASF dual-hosted git repository.
cwylie pushed a commit to branch 0.16.0-incubating
in repository https://gitbox.apache.org/repos/asf/incubator-druid.git
The following commit(s) were added to refs/heads/0.16.0-incubating by this push:
new 33163aa [Backport] Move router/indexer config/API docs to main pages
(#8519)
33163aa is described below
commit 33163aa5162ca5d2aaf8522369ef5b42ac6b10cb
Author: Clint Wylie <cwy...@apache.org>
AuthorDate: Wed Sep 11 21:44:13 2019 -0700
[Backport] Move router/indexer config/API docs to main pages (#8519)
* [maven-release-plugin] prepare release druid-0.16.0-incubating-rc2
* [maven-release-plugin] prepare for next development iteration
* Move router/indexer config/API docs to main pages (#8510)
* Move router/indexer config/API docs to main pages
* Restore missing properties, fix typo
* Use sentence casing
* Fix broken link
---
docs/configuration/index.md | 115 +++++++++++++++++++++++-
docs/design/indexer.md | 97 ++++----------------
docs/design/router.md | 188 ++++++++++++++++-----------------------
docs/operations/api-reference.md | 19 ++++
4 files changed, 223 insertions(+), 196 deletions(-)
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index aa1cd93..43d1a14 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -841,7 +841,7 @@ These Overlord static configurations can be defined in the
`overlord/runtime.pro
|Property|Description|Default|
|--------|-----------|-------|
-|`druid.indexer.runner.type`|Choices "local" or "remote". Indicates whether
tasks should be run locally or in a distributed environment. Experimental task
runner "httpRemote" is also available which is same as "remote" but uses HTTP
to interact with Middle Manaters instead of Zookeeper.|local|
+|`druid.indexer.runner.type`|Choices "local" or "remote". Indicates whether
tasks should be run locally or in a distributed environment. Experimental task
runner "httpRemote" is also available which is same as "remote" but uses HTTP
to interact with Middle Managers instead of Zookeeper.|local|
|`druid.indexer.storage.type`|Choices are "local" or "metadata". Indicates
whether incoming tasks should be stored locally (in heap) or in metadata
storage. Storing incoming tasks in metadata storage allows for tasks to be
resumed if the Overlord should fail.|local|
|`druid.indexer.storage.recentlyFinishedThreshold`|A duration of time to store
task results.|PT24H|
|`druid.indexer.tasklock.forceTimeChunkLock`|_**Setting this to false is still
experimental**_<br/> If set, all tasks are enforced to use time chunk lock. If
not set, each task automatically chooses a lock type to use. This configuration
can be overwritten by setting `forceTimeChunkLock` in the [task
context](../ingestion/tasks.md#context). See [Task Locking &
Priority](../ingestion/tasks.md#context) for more details about locking in
tasks.|true|
@@ -1049,6 +1049,8 @@ Amazon's EC2 is currently the only supported autoscaler.
This section contains the configuration options for the processes that reside
on Data servers (MiddleManagers/Peons and Historicals) in the suggested
[three-server configuration](../design/processes.html#server-types).
+Configuration options for the experimental [Indexer
process](../design/indexer.md) are also provided here.
+
### MiddleManager and Peons
These MiddleManager and Peon configurations can be defined in the
`middleManager/runtime.properties` file.
@@ -1173,6 +1175,89 @@ then the value from the configuration below is used:
|--------|-----------|-------|
|`druid.peon.defaultSegmentWriteOutMediumFactory.type`|`tmpFile` or
`offHeapMemory`, see explanation above|`tmpFile`|
+### Indexer
+
+#### Indexer Process Configuration
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.host`|The host for the current process. This is used to advertise the
current processes location as reachable from another process and should
generally be specified such that `http://${druid.host}/` could actually talk to
this process|InetAddress.getLocalHost().getCanonicalHostName()|
+|`druid.bindOnHost`|Indicating whether the process's internal jetty server
bind on `druid.host`. Default is false, which means binding to all
interfaces.|false|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port
mapping is used, this will be the same port as is on `druid.host`|8091|
+|`druid.tlsPort`|TLS port for HTTPS connector, if
[druid.enableTlsPort](../operations/tls-support.md) is set then this config
will be used. If `druid.host` contains port then that port will be ignored.
This should be a non-negative Integer.|8283|
+|`druid.service`|The name of the service. This is used as a dimension when
emitting metrics and alerts to differentiate between the various
services|druid/indexer|
+
+#### Indexer General Configuration
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.worker.version`|Version identifier for the Indexer.|0|
+|`druid.worker.capacity`|Maximum number of tasks the Indexer can
accept.|Number of available processors - 1|
+|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
+|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
+|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
+|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
+|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with
HadoopIndexTasks that do not request a particular
version.|org.apache.hadoop:hadoop-client:2.8.3|
+|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on Indexer
restart for restorable tasks to gracefully exit.|PT5M|
+|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop
tasks.|`/tmp/druid-indexing`|
+|`druid.indexer.task.restoreTasksOnRestart`|If true, the Indexer will attempt
to stop tasks gracefully on shutdown and restore them on restart.|false|
+|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to
communicate with Overlord.|PT5S|
+|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to
communicate with Overlord.|PT1M|
+|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of
retries to communicate with Overlord.|60|
+
+#### Indexer Concurrent Requests
+
+Druid uses Jetty to serve HTTP requests.
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.server.http.numThreads`|Number of threads for HTTP requests. Please
see the [Indexer Server HTTP
threads](../design/indexer.html#server-http-threads) documentation for more
details on how the Indexer uses this configuration.|max(10, (Number of cores *
17) / 16 + 2) + 30|
+|`druid.server.http.queueSize`|Size of the worker queue used by Jetty server
to temporarily store incoming client connections. If this value is set and a
request is rejected by jetty because queue is full then client would observe
request failure with TCP connection being closed immediately with a completely
empty response from server.|Unbounded|
+|`druid.server.http.maxIdleTime`|The Jetty max idle time for a
connection.|PT5M|
+|`druid.server.http.enableRequestLimit`|If enabled, no requests would be
queued in jetty queue and "HTTP 429 Too Many Requests" error response would be
sent. |false|
+|`druid.server.http.defaultQueryTimeout`|Query timeout in millis, beyond which
unfinished queries will be cancelled|300000|
+|`druid.server.http.gracefulShutdownTimeout`|The maximum amount of time Jetty
waits after receiving shutdown signal. After this timeout the threads will be
forcefully shutdown. This allows any queries that are executing to
complete.|`PT0S` (do not wait)|
+|`druid.server.http.unannouncePropagationDelay`|How long to wait for zookeeper
unannouncements to propagate before shutting down Jetty. This is a minimum and
`druid.server.http.gracefulShutdownTimeout` does not start counting down until
after this period elapses.|`PT0S` (do not wait)|
+|`druid.server.http.maxQueryTimeout`|Maximum allowed value (in milliseconds)
for `timeout` parameter. See [query-context](../querying/query-context.html) to
know more about `timeout`. Query is rejected if the query context `timeout` is
greater than this value. |Long.MAX_VALUE|
+|`druid.server.http.maxRequestHeaderSize`|Maximum size of a request header in
bytes. Larger headers consume more memory and can make a server more vulnerable
to denial of service attacks.|8 * 1024|
+
+#### Indexer Processing Resources
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.processing.buffer.sizeBytes`|This specifies a buffer size for the
storage of intermediate results. The computation engine in the Indexer
processes will use a scratch buffer of this size to do all of their
intermediate computations off-heap. Larger values allow for more aggregations
in a single pass over the data while smaller values can require more passes
depending on the query that is being executed.|auto (max 1GB)|
+|`druid.processing.buffer.poolCacheMaxCount`|processing buffer pool caches the
buffers for later use, this is the maximum count cache will grow to. note that
pool can create more buffers than it can cache if necessary.|Integer.MAX_VALUE|
+|`druid.processing.formatString`|Indexer processes use this format string to
name their processing threads.|processing-%s|
+|`druid.processing.numMergeBuffers`|The number of direct memory buffers
available for merging query results. The buffers are sized by
`druid.processing.buffer.sizeBytes`. This property is effectively a concurrency
limit for queries that require merging buffers. If you are using any queries
that require merge buffers (currently, just groupBy v2) then you should have at
least two of these.|`max(2, druid.processing.numThreads / 4)`|
+|`druid.processing.numThreads`|The number of processing threads to have
available for parallel processing of segments. Our rule of thumb is `num_cores
- 1`, which means that even under heavy load there will still be one core
available to do background tasks like talking with ZooKeeper and pulling down
segments. If only one core is available, this property defaults to the value
`1`.|Number of cores - 1 (or 1)|
+|`druid.processing.columnCache.sizeBytes`|Maximum size in bytes for the
dimension value lookup cache. Any value greater than `0` enables the cache. It
is currently disabled by default. Enabling the lookup cache can significantly
improve the performance of aggregators operating on dimension values, such as
the JavaScript aggregator, or cardinality aggregator, but can slow things down
if the cache hit rate is low (i.e. dimensions with few repeating values).
Enabling it may also require add [...]
+|`druid.processing.fifo`|If the processing queue should treat tasks of equal
priority in a FIFO manner|`false`|
+|`druid.processing.tmpDir`|Path where temporary files created while processing
a query should be stored. If specified, this configuration takes priority over
the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`|
+
+The amount of direct memory needed by Druid is at least
+`druid.processing.buffer.sizeBytes * (druid.processing.numMergeBuffers +
druid.processing.numThreads + 1)`. You can
+ensure at least this amount of direct memory is available by providing
`-XX:MaxDirectMemorySize=<VALUE>` at the command
+line.
+
+#### Query Configurations
+
+See [general query configuration](#general-query-configuration).
+
+#### Indexer Caching
+
+You can optionally configure caching to be enabled on the Indexer by setting
caching configs here.
+
+|Property|Possible Values|Description|Default|
+|--------|---------------|-----------|-------|
+|`druid.realtime.cache.useCache`|true, false|Enable the cache on the
realtime.|false|
+|`druid.realtime.cache.populateCache`|true, false|Populate the cache on the
realtime.|false|
+|`druid.realtime.cache.unCacheable`|All druid query types|All query types to
not cache.|`["groupBy", "select"]`|
+|`druid.realtime.cache.maxEntrySize`|Maximum cache entry size in
bytes.|1_000_000|
+
+See [cache configuration](#cache-configuration) for how to configure cache
settings.
+
+Note that only local caches such as the `local`-type cache and `caffeine`
cache are supported. If a remote cache such as `memcached` is used, it will be
ignored.
+
### Historical
For general Historical Process information, see
[here](../design/historical.md).
@@ -1188,7 +1273,6 @@ These Historical configurations can be defined in the
`historical/runtime.proper
|`druid.tlsPort`|TLS port for HTTPS connector, if
[druid.enableTlsPort](../operations/tls-support.md) is set then this config
will be used. If `druid.host` contains port then that port will be ignored.
This should be a non-negative Integer.|8283|
|`druid.service`|The name of the service. This is used as a dimension when
emitting metrics and alerts to differentiate between the various
services|druid/historical|
-
#### Historical General Configuration
|Property|Description|Default|
@@ -1269,6 +1353,8 @@ See [cache configuration](#cache-configuration) for how
to configure cache setti
This section contains the configuration options for the processes that reside
on Query servers (Brokers) in the suggested [three-server
configuration](../design/processes.html#server-types).
+Configuration options for the experimental [Router
process](../design/router.md) are also provided here.
+
### Broker
For general Broker process information, see [here](../design/broker.md).
@@ -1613,3 +1699,28 @@ Supported query contexts:
|`maxIntermediateRows`|Can be used to lower the value of
`druid.query.groupBy.maxIntermediateRows` for this query.|None|
|`maxResults`|Can be used to lower the value of
`druid.query.groupBy.maxResults` for this query.|None|
|`useOffheap`|Set to true to store aggregations off-heap when merging
results.|false|
+
+### Router
+
+#### Router Process Configs
+
+|Property|Description|Default|
+|--------|-----------|-------|
+|`druid.host`|The host for the current process. This is used to advertise the
current processes location as reachable from another process and should
generally be specified such that `http://${druid.host}/` could actually talk to
this process|InetAddress.getLocalHost().getCanonicalHostName()|
+|`druid.bindOnHost`|Indicating whether the process's internal jetty server
bind on `druid.host`. Default is false, which means binding to all
interfaces.|false|
+|`druid.plaintextPort`|This is the port to actually listen on; unless port
mapping is used, this will be the same port as is on `druid.host`|8888|
+|`druid.tlsPort`|TLS port for HTTPS connector, if
[druid.enableTlsPort](../operations/tls-support.md) is set then this config
will be used. If `druid.host` contains port then that port will be ignored.
This should be a non-negative Integer.|9088|
+|`druid.service`|The name of the service. This is used as a dimension when
emitting metrics and alerts to differentiate between the various
services|druid/router|
+
+#### Runtime Configuration
+
+|Property|Possible Values|Description|Default|
+|--------|---------------|-----------|-------|
+|`druid.router.defaultBrokerServiceName`|Any string.|The default Broker to
connect to in case service discovery fails.|druid/broker|
+|`druid.router.tierToBrokerMap`|An ordered JSON map of tiers to Broker names.
The priority of Brokers is based on the ordering.|Queries for a certain tier of
data are routed to their appropriate Broker.|{"_default_tier":
"<defaultBrokerServiceName>"}|
+|`druid.router.defaultRule`|Any string.|The default rule for all
datasources.|"_default"|
+|`druid.router.pollPeriod`|Any ISO8601 duration.|How often to poll for new
rules.|PT1M|
+|`druid.router.strategies`|An ordered JSON array of objects.|Please see
[Router Strategies](../design/router.html#router-strategies) for
details.|[{"type":"timeBoundary"},{"type":"priority"}]|
+|`druid.router.avatica.balancer.type`|String representing an
AvaticaConnectionBalancer name. Please see [Avatica Query
Balancing](../design/router.html#avatica-query-balancing)|Class to use for
balancing Avatica queries across Brokers|rendezvousHash|
+|`druid.router.http.maxRequestBufferSize`|Maximum size of the buffer used to
write requests when forwarding them to the Broker. This should be set to
atleast the maxHeaderSize allowed on the Broker|8 * 1024|
+|`druid.router.managementProxy.enabled`|Enables the Router's [management
proxy](../design/router.html#router-as-management-proxy) functionality.|false|
diff --git a/docs/design/indexer.md b/docs/design/indexer.md
index 4221b4e..93871d4 100644
--- a/docs/design/indexer.md
+++ b/docs/design/indexer.md
@@ -29,23 +29,31 @@ The Apache Druid (incubating) Indexer process is an
alternative to the MiddleMan
The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system and to better enable resource sharing across tasks.
-## Running
+### Configuration
+
+For Apache Druid (incubating) Indexer Process Configuration, see [Indexer
Configuration](../configuration/index.html#indexer).
+
+### HTTP endpoints
+
+The Indexer process shares the same HTTP endpoints as the
[MiddleManager](../operations/api-reference.html#middlemanager).
+
+### Running
```
org.apache.druid.cli.Main server indexer
```
-## Task Resource Sharing
+### Task resource sharing
The following resources are shared across all tasks running inside an Indexer
process.
-### Query resources
+#### Query resources
The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
-If [query caching](#indexer-caching) is enabled, the query cache is also
shared across all tasks.
+If [query caching](../configuration/index.html#indexer-caching) is enabled,
the query cache is also shared across all tasks.
-### Server HTTP threads
+#### Server HTTP threads
The Indexer maintains two equally sized pools of HTTP threads.
@@ -55,7 +63,7 @@ The size of the pools are configured by the
`druid.server.http.numThreads` confi
In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
-### Memory Sharing
+#### Memory sharing
The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
@@ -71,86 +79,13 @@ This means that the peak in-heap usage for row data can be
up to approximately `
The remaining portion of the heap is reserved for query processing and segment
persist/merge operations, and miscellaneous heap usage.
-#### Concurrent Segment Persist/Merge Limits
+#### Concurrent segment persist/merge limits
To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
-## Runtime Configuration
-
-In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.worker.version`|Version identifier for the Indexer.|0|
-|`druid.worker.capacity`|Maximum number of tasks the Indexer can
accept.|Number of available processors - 1|
-|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
-|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
-|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
-|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
-|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with
HadoopIndexTasks that do not request a particular
version.|org.apache.hadoop:hadoop-client:2.8.3|
-|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on Indexer
restart for restorable tasks to gracefully exit.|PT5M|
-|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop
tasks.|`/tmp/druid-indexing`|
-|`druid.indexer.task.restoreTasksOnRestart`|If true, the Indexer will attempt
to stop tasks gracefully on shutdown and restore them on restart.|false|
-|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to
communicate with Overlord.|PT5S|
-|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to
communicate with Overlord.|PT1M|
-|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of
retries to communicate with Overlord.|60|
-
-### Concurrent Requests
-
-Druid uses Jetty to serve HTTP requests.
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.server.http.numThreads`|Number of threads for HTTP requests. Please
see the [Server HTTP threads](#server-http-threads) section for more details on
how the Indexer uses this configuration.|max(10, (Number of cores * 17) / 16 +
2) + 30|
-|`druid.server.http.queueSize`|Size of the worker queue used by Jetty server
to temporarily store incoming client connections. If this value is set and a
request is rejected by jetty because queue is full then client would observe
request failure with TCP connection being closed immediately with a completely
empty response from server.|Unbounded|
-|`druid.server.http.maxIdleTime`|The Jetty max idle time for a
connection.|PT5M|
-|`druid.server.http.enableRequestLimit`|If enabled, no requests would be
queued in jetty queue and "HTTP 429 Too Many Requests" error response would be
sent. |false|
-|`druid.server.http.defaultQueryTimeout`|Query timeout in millis, beyond which
unfinished queries will be cancelled|300000|
-|`druid.server.http.gracefulShutdownTimeout`|The maximum amount of time Jetty
waits after receiving shutdown signal. After this timeout the threads will be
forcefully shutdown. This allows any queries that are executing to
complete.|`PT0S` (do not wait)|
-|`druid.server.http.unannouncePropagationDelay`|How long to wait for zookeeper
unannouncements to propagate before shutting down Jetty. This is a minimum and
`druid.server.http.gracefulShutdownTimeout` does not start counting down until
after this period elapses.|`PT0S` (do not wait)|
-|`druid.server.http.maxQueryTimeout`|Maximum allowed value (in milliseconds)
for `timeout` parameter. See [query-context](../querying/query-context.html) to
know more about `timeout`. Query is rejected if the query context `timeout` is
greater than this value. |Long.MAX_VALUE|
-|`druid.server.http.maxRequestHeaderSize`|Maximum size of a request header in
bytes. Larger headers consume more memory and can make a server more vulnerable
to denial of service attacks.|8 * 1024|
-
-### Processing
-
-|Property|Description|Default|
-|--------|-----------|-------|
-|`druid.processing.buffer.sizeBytes`|This specifies a buffer size for the
storage of intermediate results. The computation engine in the Indexer
processes will use a scratch buffer of this size to do all of their
intermediate computations off-heap. Larger values allow for more aggregations
in a single pass over the data while smaller values can require more passes
depending on the query that is being executed.|auto (max 1GB)|
-|`druid.processing.buffer.poolCacheMaxCount`|processing buffer pool caches the
buffers for later use, this is the maximum count cache will grow to. note that
pool can create more buffers than it can cache if necessary.|Integer.MAX_VALUE|
-|`druid.processing.formatString`|Indexer processes use this format string to
name their processing threads.|processing-%s|
-|`druid.processing.numMergeBuffers`|The number of direct memory buffers
available for merging query results. The buffers are sized by
`druid.processing.buffer.sizeBytes`. This property is effectively a concurrency
limit for queries that require merging buffers. If you are using any queries
that require merge buffers (currently, just groupBy v2) then you should have at
least two of these.|`max(2, druid.processing.numThreads / 4)`|
-|`druid.processing.numThreads`|The number of processing threads to have
available for parallel processing of segments. Our rule of thumb is `num_cores
- 1`, which means that even under heavy load there will still be one core
available to do background tasks like talking with ZooKeeper and pulling down
segments. If only one core is available, this property defaults to the value
`1`.|Number of cores - 1 (or 1)|
-|`druid.processing.columnCache.sizeBytes`|Maximum size in bytes for the
dimension value lookup cache. Any value greater than `0` enables the cache. It
is currently disabled by default. Enabling the lookup cache can significantly
improve the performance of aggregators operating on dimension values, such as
the JavaScript aggregator, or cardinality aggregator, but can slow things down
if the cache hit rate is low (i.e. dimensions with few repeating values).
Enabling it may also require add [...]
-|`druid.processing.fifo`|If the processing queue should treat tasks of equal
priority in a FIFO manner|`false`|
-|`druid.processing.tmpDir`|Path where temporary files created while processing
a query should be stored. If specified, this configuration takes priority over
the default `java.io.tmpdir` path.|path represented by `java.io.tmpdir`|
-
-The amount of direct memory needed by Druid is at least
-`druid.processing.buffer.sizeBytes * (druid.processing.numMergeBuffers +
druid.processing.numThreads + 1)`. You can
-ensure at least this amount of direct memory is available by providing
`-XX:MaxDirectMemorySize=<VALUE>` at the command
-line.
-
-### Query Configurations
-
-See [general query
configuration](../configuration/index.html#general-query-configuration).
-
-### Indexer Caching
-
-You can optionally configure caching to be enabled on the Indexer by setting
caching configs here.
-
-|Property|Possible Values|Description|Default|
-|--------|---------------|-----------|-------|
-|`druid.realtime.cache.useCache`|true, false|Enable the cache on the
realtime.|false|
-|`druid.realtime.cache.populateCache`|true, false|Populate the cache on the
realtime.|false|
-|`druid.realtime.cache.unCacheable`|All druid query types|All query types to
not cache.|`["groupBy", "select"]`|
-|`druid.realtime.cache.maxEntrySize`|Maximum cache entry size in
bytes.|1_000_000|
-
-See [cache configuration](../configuration/index.html#cache-configuration) for
how to configure cache settings.
-
-Note that only local caches such as the `local`-type cache and `caffeine`
cache are supported. If a remote cache such as `memcached` is used, it will be
ignored.
-
-## Current Limitations
+### Current limitations
Separate task logs are not currently supported when using the Indexer; all
task log messages will instead be logged in the Indexer process log.
diff --git a/docs/design/router.md b/docs/design/router.md
index 68b348a..b00cf87 100644
--- a/docs/design/router.md
+++ b/docs/design/router.md
@@ -32,82 +32,62 @@ For query routing purposes, you should only ever need the
Router process if you
In addition to query routing, the Router also runs the [Druid
Console](../operations/management-uis.html#druid-console), a management UI for
datasources, segments, tasks, data processes (Historicals and MiddleManagers),
and coordinator dynamic configuration. The user can also run SQL and native
Druid queries within the console.
+### Configuration
-Running
--------
+For Apache Druid (incubating) Router Process Configuration, see [Router
Configuration](../configuration/index.html#router).
+
+### HTTP endpoints
+
+For a list of API endpoints supported by the Router, see [Router
API](../operations/api-reference.html#router).
+
+### Running
```
org.apache.druid.cli.Main server router
```
-Example Production Configuration
---------------------------------
+### Router as management proxy
-In this example, we have two tiers in our production cluster: `hot` and
`_default_tier`. Queries for the `hot` tier are routed through the `broker-hot`
set of Brokers, and queries for the `_default_tier` are routed through the
`broker-cold` set of Brokers. If any exceptions or network problems occur,
queries are routed to the `broker-cold` set of brokers. In our example, we are
running with a c3.2xlarge EC2 instance. We assume a `common.runtime.properties`
already exists.
+The Router can be configured to forward requests to the active Coordinator or
Overlord process. This may be useful for
+setting up a highly available cluster in situations where the HTTP redirect
mechanism of the inactive -> active
+Coordinator/Overlord does not function correctly (servers are behind a load
balancer, the hostname used in the redirect
+is only resolvable internally, etc.).
-JVM settings:
+#### Enabling the management proxy
-```
--server
--Xmx13g
--Xms13g
--XX:NewSize=256m
--XX:MaxNewSize=256m
--XX:+UseConcMarkSweepGC
--XX:+PrintGCDetails
--XX:+PrintGCTimeStamps
--XX:+UseLargePages
--XX:+HeapDumpOnOutOfMemoryError
--XX:HeapDumpPath=/mnt/galaxy/deploy/current/
--Duser.timezone=UTC
--Dfile.encoding=UTF-8
--Djava.io.tmpdir=/mnt/tmp
+To enable this functionality, set the following in the Router's
runtime.properties:
--Dcom.sun.management.jmxremote.port=17071
--Dcom.sun.management.jmxremote.authenticate=false
--Dcom.sun.management.jmxremote.ssl=false
```
-
-Runtime.properties:
-
+druid.router.managementProxy.enabled=true
```
-druid.host=#{IP_ADDR}:8080
-druid.plaintextPort=8080
-druid.service=druid/router
-
-druid.router.defaultBrokerServiceName=druid:broker-cold
-druid.router.coordinatorServiceName=druid:coordinator
-druid.router.tierToBrokerMap={"hot":"druid:broker-hot","_default_tier":"druid:broker-cold"}
-druid.router.http.numConnections=50
-druid.router.http.readTimeout=PT5M
-# Number of threads used by the Router proxy http client
-druid.router.http.numMaxThreads=100
+#### Management proxy routing
-druid.server.http.numThreads=100
-```
+The management proxy supports implicit and explicit routes. Implicit routes
are those where the destination can be
+determined from the original request path based on Druid API path conventions.
For the Coordinator the convention is
+`/druid/coordinator/*` and for the Overlord the convention is
`/druid/indexer/*`. These are convenient because they mean
+that using the management proxy does not require modifying the API request
other than issuing the request to the Router
+instead of the Coordinator or Overlord. Most Druid API requests can be routed
implicitly.
-Runtime Configuration
----------------------
+Explicit routes are those where the request to the Router contains a path
prefix indicating which process the request
+should be routed to. For the Coordinator this prefix is `/proxy/coordinator`
and for the Overlord it is `/proxy/overlord`.
+This is required for API calls with an ambiguous destination. For example, the
`/status` API is present on all Druid
+processes, so explicit routing needs to be used to indicate the proxy
destination.
-The Router module uses several of the default modules in
[Configuration](../configuration/index.md) and has the following set of
configurations as well:
+This is summarized in the table below:
-|Property|Possible Values|Description|Default|
-|--------|---------------|-----------|-------|
-|`druid.router.defaultBrokerServiceName`|Any string.|The default Broker to
connect to in case service discovery fails.|druid/broker|
-|`druid.router.tierToBrokerMap`|An ordered JSON map of tiers to Broker names.
The priority of Brokers is based on the ordering.|Queries for a certain tier of
data are routed to their appropriate Broker.|{"_default_tier":
"<defaultBrokerServiceName>"}|
-|`druid.router.defaultRule`|Any string.|The default rule for all
datasources.|"_default"|
-|`druid.router.pollPeriod`|Any ISO8601 duration.|How often to poll for new
rules.|PT1M|
-|`druid.router.strategies`|An ordered JSON array of objects.|All custom
strategies to use for routing.|[{"type":"timeBoundary"},{"type":"priority"}]|
-|`druid.router.avatica.balancer.type`|String representing an
AvaticaConnectionBalancer name|Class to use for balancing Avatica queries
across Brokers|rendezvousHash|
-|`druid.router.http.maxRequestBufferSize`|Maximum size of the buffer used to
write requests when forwarding them to the Broker. This should be set to
atleast the maxHeaderSize allowed on the Broker|8 * 1024|
+|Request Route|Destination|Rewritten Route|Example|
+|-------------|-----------|---------------|-------|
+|`/druid/coordinator/*`|Coordinator|`/druid/coordinator/*`|`router:8888/druid/coordinator/v1/datasources`
-> `coordinator:8081/druid/coordinator/v1/datasources`|
+|`/druid/indexer/*`|Overlord|`/druid/indexer/*`|`router:8888/druid/indexer/v1/task`
-> `overlord:8090/druid/indexer/v1/task`|
+|`/proxy/coordinator/*`|Coordinator|`/*`|`router:8888/proxy/coordinator/status`
-> `coordinator:8081/status`|
+|`/proxy/overlord/*`|Overlord|`/*`|`router:8888/proxy/overlord/druid/indexer/v1/isLeader`
-> `overlord:8090/druid/indexer/v1/isLeader`|
+### Router strategies
-Router Strategies
------------------
The Router has a configurable list of strategies for how it selects which
Brokers to route queries to. The order of the strategies matter because as soon
as a strategy condition is matched, a Broker is selected.
-### timeBoundary
+#### timeBoundary
```json
{
@@ -117,7 +97,7 @@ The Router has a configurable list of strategies for how it
selects which Broker
Including this strategy means all timeBoundary queries are always routed to
the highest priority Broker.
-### priority
+#### priority
```json
{
@@ -129,7 +109,7 @@ Including this strategy means all timeBoundary queries are
always routed to the
Queries with a priority set to less than minPriority are routed to the lowest
priority Broker. Queries with priority set to greater than maxPriority are
routed to the highest priority Broker. By default, minPriority is 0 and
maxPriority is 1. Using these default values, if a query with priority 0 (the
default query priority is 0) is sent, the query skips the priority selection
logic.
-### JavaScript
+#### JavaScript
Allows defining arbitrary routing rules using a JavaScript function. The
function is passed the configuration and the query to be executed, and returns
the tier it should be routed to, or null for the default tier.
@@ -145,8 +125,7 @@ Allows defining arbitrary routing rules using a JavaScript
function. The functio
> JavaScript-based functionality is disabled by default. Please refer to the
> Druid [JavaScript programming guide](../development/javascript.md) for
> guidelines about using Druid's JavaScript functionality, including
> instructions on how to enable it.
-Avatica Query Balancing
---------------
+### Avatica query balancing
All Avatica JDBC requests with a given connection ID must be routed to the
same Broker, since Druid Brokers do not share connection state with each other.
@@ -154,7 +133,7 @@ To accomplish this, Druid provides two built-in balancers
that use rendezvous ha
Note that when multiple Routers are used, all Routers should have identical
balancer configuration to ensure that they make the same routing decisions.
-### Rendezvous hash balancer
+#### Rendezvous hash balancer
This balancer uses [Rendezvous
Hashing](https://en.wikipedia.org/wiki/Rendezvous_hashing) on an Avatica
request's connection ID to assign the request to a Broker.
@@ -166,7 +145,7 @@ druid.router.avatica.balancer.type=rendezvousHash
If no `druid.router.avatica.balancer` property is set, the Router will also
default to using the Rendezvous Hash Balancer.
-### Consistent hash balancer
+#### Consistent hash balancer
This balancer uses [Consistent
Hashing](https://en.wikipedia.org/wiki/Consistent_hashing) on an Avatica
request's connection ID to assign the request to a Broker.
@@ -178,67 +157,50 @@ druid.router.avatica.balancer.type=consistentHash
This is a non-default implementation that is provided for experimentation
purposes. The consistent hasher has longer setup times on initialization and
when the set of Brokers changes, but has a faster Broker assignment time than
the rendezous hasher when tested with 5 Brokers. Benchmarks for both
implementations have been provided in `ConsistentHasherBenchmark` and
`RendezvousHasherBenchmark`. The consistent hasher also requires locking, while
the rendezvous hasher does not.
-HTTP endpoints
---------------
-
-The Router process exposes several HTTP endpoints for interactions.
-
-### GET
-
-* `/status`
-
-Returns the Druid version, loaded extensions, memory used, total memory and
other useful information about the process.
-* `/druid/v2/datasources`
+### Example production configuration
-Returns a list of queryable datasources.
-
-* `/druid/v2/datasources/{dataSourceName}`
-
-Returns the dimensions and metrics of the datasource.
-
-* `/druid/v2/datasources/{dataSourceName}/dimensions`
-
-Returns the dimensions of the datasource.
-
-* `/druid/v2/datasources/{dataSourceName}/metrics`
-
-Returns the metrics of the datasource.
+In this example, we have two tiers in our production cluster: `hot` and
`_default_tier`. Queries for the `hot` tier are routed through the `broker-hot`
set of Brokers, and queries for the `_default_tier` are routed through the
`broker-cold` set of Brokers. If any exceptions or network problems occur,
queries are routed to the `broker-cold` set of brokers. In our example, we are
running with a c3.2xlarge EC2 instance. We assume a `common.runtime.properties`
already exists.
-Router as Management Proxy
---------------------------
+JVM settings:
-The Router can be configured to forward requests to the active Coordinator or
Overlord process. This may be useful for
-setting up a highly available cluster in situations where the HTTP redirect
mechanism of the inactive -> active
-Coordinator/Overlord does not function correctly (servers are behind a load
balancer, the hostname used in the redirect
-is only resolvable internally, etc.).
+```
+-server
+-Xmx13g
+-Xms13g
+-XX:NewSize=256m
+-XX:MaxNewSize=256m
+-XX:+UseConcMarkSweepGC
+-XX:+PrintGCDetails
+-XX:+PrintGCTimeStamps
+-XX:+UseLargePages
+-XX:+HeapDumpOnOutOfMemoryError
+-XX:HeapDumpPath=/mnt/galaxy/deploy/current/
+-Duser.timezone=UTC
+-Dfile.encoding=UTF-8
+-Djava.io.tmpdir=/mnt/tmp
-### Enabling the Management Proxy
+-Dcom.sun.management.jmxremote.port=17071
+-Dcom.sun.management.jmxremote.authenticate=false
+-Dcom.sun.management.jmxremote.ssl=false
+```
-To enable this functionality, set the following in the Router's
runtime.properties:
+Runtime.properties:
```
-druid.router.managementProxy.enabled=true
-```
-
-### Routing
+druid.host=#{IP_ADDR}:8080
+druid.plaintextPort=8080
+druid.service=druid/router
-The management proxy supports implicit and explicit routes. Implicit routes
are those where the destination can be
-determined from the original request path based on Druid API path conventions.
For the Coordinator the convention is
-`/druid/coordinator/*` and for the Overlord the convention is
`/druid/indexer/*`. These are convenient because they mean
-that using the management proxy does not require modifying the API request
other than issuing the request to the Router
-instead of the Coordinator or Overlord. Most Druid API requests can be routed
implicitly.
+druid.router.defaultBrokerServiceName=druid:broker-cold
+druid.router.coordinatorServiceName=druid:coordinator
+druid.router.tierToBrokerMap={"hot":"druid:broker-hot","_default_tier":"druid:broker-cold"}
+druid.router.http.numConnections=50
+druid.router.http.readTimeout=PT5M
-Explicit routes are those where the request to the Router contains a path
prefix indicating which process the request
-should be routed to. For the Coordinator this prefix is `/proxy/coordinator`
and for the Overlord it is `/proxy/overlord`.
-This is required for API calls with an ambiguous destination. For example, the
`/status` API is present on all Druid
-processes, so explicit routing needs to be used to indicate the proxy
destination.
+# Number of threads used by the Router proxy http client
+druid.router.http.numMaxThreads=100
-This is summarized in the table below:
+druid.server.http.numThreads=100
+```
-|Request Route|Destination|Rewritten Route|Example|
-|-------------|-----------|---------------|-------|
-|`/druid/coordinator/*`|Coordinator|`/druid/coordinator/*`|`router:8888/druid/coordinator/v1/datasources`
-> `coordinator:8081/druid/coordinator/v1/datasources`|
-|`/druid/indexer/*`|Overlord|`/druid/indexer/*`|`router:8888/druid/indexer/v1/task`
-> `overlord:8090/druid/indexer/v1/task`|
-|`/proxy/coordinator/*`|Coordinator|`/*`|`router:8888/proxy/coordinator/status`
-> `coordinator:8081/status`|
-|`/proxy/overlord/*`|Overlord|`/*`|`router:8888/proxy/overlord/druid/indexer/v1/isLeader`
-> `overlord:8090/druid/indexer/v1/isLeader`|
diff --git a/docs/operations/api-reference.md b/docs/operations/api-reference.md
index d271a95..9b06eee 100644
--- a/docs/operations/api-reference.md
+++ b/docs/operations/api-reference.md
@@ -765,3 +765,22 @@ The endpoint for submitting queries. Accepts an option
`?pretty` that pretty pri
Returns segment information lists including server locations for the given
query..
+### Router
+
+#### GET
+
+* `/druid/v2/datasources`
+
+Returns a list of queryable datasources.
+
+* `/druid/v2/datasources/{dataSourceName}`
+
+Returns the dimensions and metrics of the datasource.
+
+* `/druid/v2/datasources/{dataSourceName}/dimensions`
+
+Returns the dimensions of the datasource.
+
+* `/druid/v2/datasources/{dataSourceName}/metrics`
+
+Returns the metrics of the datasource.
\ No newline at end of file
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
