[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888120
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
+
+In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
+
+|Property|Description|Default|
+||---|---|
+|`druid.worker.version`|Version identifier for the MiddleManager.|0|
Review comment:
This line, and a few other places, say "MiddleManager" when they should say
"Indexer".
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888497
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
+
+In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
+
+|Property|Description|Default|
+||---|---|
+|`druid.worker.version`|Version identifier for the MiddleManager.|0|
+|`druid.worker.capacity`|Maximum number of tasks the MiddleManager can
accept.|Number of available processors - 1|
+|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
+|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
+|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
+|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
+|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with
HadoopIndexTasks that do not request a particular
version.|org.apache.hadoop:hadoop-client:2.8.3|
+|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager
restart for restorable tasks to gracefully exit.|PT5M|
+|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop
tasks.|`/tmp/druid-indexing`|
+|`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will
attempt to stop tasks gracefully on shutdown and restore them on restart.|false|
+|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to
communicate with Overlord.|PT5S|
+|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to
communicate with Overlord.|PT1M|
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310887454 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. Review comment: "and better enable resource sharing between tasks"? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310885372
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
+
+In addition to the [common
configurations](../configuration/index.html#common-configurations), the Indexer
accepts the following configurations:
+
+|Property|Description|Default|
+||---|---|
+|`druid.worker.version`|Version identifier for the MiddleManager.|0|
+|`druid.worker.capacity`|Maximum number of tasks the MiddleManager can
accept.|Number of available processors - 1|
+|`druid.worker.globalIngestionHeapLimitBytes`|Total amount of heap available
for ingestion processing. This is applied by automatically setting the
`maxBytesInMemory` property on tasks.|60% of configured JVM heap|
+|`druid.worker.numConcurrentMerges`|Maximum number of segment persist or merge
operations that can run concurrently across all tasks.|`druid.worker.capacity`
/ 2, rounded down|
+|`druid.indexer.task.baseDir`|Base temporary working
directory.|`System.getProperty("java.io.tmpdir")`|
+|`druid.indexer.task.baseTaskDir`|Base temporary working directory for
tasks.|`${druid.indexer.task.baseDir}/persistent/tasks`|
+|`druid.indexer.task.defaultHadoopCoordinates`|Hadoop version to use with
HadoopIndexTasks that do not request a particular
version.|org.apache.hadoop:hadoop-client:2.8.3|
+|`druid.indexer.task.gracefulShutdownTimeout`|Wait this long on middleManager
restart for restorable tasks to gracefully exit.|PT5M|
+|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop
tasks.|`/tmp/druid-indexing`|
+|`druid.indexer.task.restoreTasksOnRestart`|If true, middleManagers will
attempt to stop tasks gracefully on shutdown and restore them on restart.|false|
+|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to
communicate with Overlord.|PT5S|
+|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to
communicate with Overlord.|PT1M|
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as
an experimental feature
URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310888041
##
File path: docs/content/development/indexer.md
##
@@ -0,0 +1,152 @@
+---
+layout: doc_page
+title: "Indexer Process"
+---
+
+
+
+# Indexer Process
+
+
+The Indexer is an optional and experimental feature. Its memory
management system is still under development and will be significantly enhanced
in later releases.
+
+
+The Apache Druid (incubating) Indexer process is an alternative to the
MiddleManager + Peon task execution system. Instead of forking a separate JVM
process per-task, the Indexer runs tasks as separate threads within a single
JVM process.
+
+The Indexer is designed to be easier to configure and deploy compared to the
MiddleManager + Peon system.
+
+## Running
+
+```
+org.apache.druid.cli.Main server indexer
+```
+
+## Task Resource Sharing
+
+The following resources are shared across all tasks running inside an Indexer
process:
+
+### Query resources
+
+The query processing threads and buffers are shared across all tasks. The
Indexer will serve queries from a single endpoint shared by all tasks.
+
+### Server HTTP threads
+
+The Indexer maintains two equally sized pools of HTTP threads.
+
+One pool is exclusively used for task control messages between the Overlord
and the Indexer ("chat handler threads"). The other pool is used for handling
all other HTTP requests.
+
+The size of the pools are configured by the `druid.server.http.numThreads`
configuration (e.g., if this is set to 10, there will be 10 chat handler
threads and 10 non-chat handler threads).
+
+In addition to these two pools, 2 separate threads are allocated for lookup
handling. If lookups are not used, these threads will not be used.
+
+### Memory Sharing
+
+The Indexer uses the `druid.worker.globalIngestionHeapLimitBytes`
configuration to impose a global heap limit across all of the tasks it is
running.
+
+This global limit is evenly divided across the number of task slots configured
by `druid.worker.capacity`.
+
+To apply the per-task heap limit, the Indexer will override `maxBytesInMemory`
in task tuning configs (i.e., ignoring the default value or any user configured
value). `maxRowsInMemory` will also be overridden to an essentially unlimited
value: the Indexer does not support row limits.
+
+By default, `druid.worker.globalIngestionHeapLimitBytes` is set to 60% of the
available JVM heap. The remaining portion of the heap is reserved for query
processing and segment persist/merge operations, and miscellaneous heap usage.
+
+ Concurrent Segment Persist/Merge Limits
+
+To help reduce peak memory usage, the Indexer imposes a limit on the number of
concurrent segment persist/merge operations across all running tasks.
+
+By default, the number of concurrent persist/merge operations is limited to
(`druid.worker.capacity` / 2), rounded down. This limit can be configured with
the `druid.worker.numConcurrentMerges` property.
+
+## Runtime Configuration
Review comment:
How about adding these to `configuration/index.md`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
-
To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org
For additional commands, e-mail: commits-h...@druid.apache.org
[GitHub] [incubator-druid] gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature
gianm commented on a change in pull request #8245: Add docs for CliIndexer as an experimental feature URL: https://github.com/apache/incubator-druid/pull/8245#discussion_r310887504 ## File path: docs/content/development/indexer.md ## @@ -0,0 +1,152 @@ +--- +layout: doc_page +title: "Indexer Process" +--- + + + +# Indexer Process + + +The Indexer is an optional and experimental feature. Its memory management system is still under development and will be significantly enhanced in later releases. + + +The Apache Druid (incubating) Indexer process is an alternative to the MiddleManager + Peon task execution system. Instead of forking a separate JVM process per-task, the Indexer runs tasks as separate threads within a single JVM process. + +The Indexer is designed to be easier to configure and deploy compared to the MiddleManager + Peon system. + +## Running + +``` +org.apache.druid.cli.Main server indexer +``` + +## Task Resource Sharing + +The following resources are shared across all tasks running inside an Indexer process: Review comment: `.` would work better than `:` since the following sections are subheads (not a list). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services - To unsubscribe, e-mail: commits-unsubscr...@druid.apache.org For additional commands, e-mail: commits-h...@druid.apache.org
