This is an automated email from the ASF dual-hosted git repository.
techdocsmith pushed a commit to branch 30.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/30.0.0 by this push:
new e68f9173653 [backport] multiple docs prs (#16546)
e68f9173653 is described below
commit e68f91736533c35686e006742d347c07b2e63783
Author: 317brian <[email protected]>
AuthorDate: Tue Jun 4 13:20:24 2024 -0700
[backport] multiple docs prs (#16546)
* docs: Migration guide for subquery limit (#16519)
Adds a migration guide for Druid 30 to help users understand the new
byte-based subquery limit property maxSubqueryBytes
(cherry picked from commit 3c72ec8413fc066dbb56e9f02d6fbc6e3b28bbe1)
* Adds a migration guide overview page to the release-info section (#16506)
Co-authored-by: 317brian <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
(cherry picked from commit 92e565e3b814ac945aa6c9f48670e6c1576eaec4)
* Update front coding text (#16491)
Co-authored-by: 317brian <[email protected]>
Co-authored-by: Victoria Lim <[email protected]>
(cherry picked from commit f70ef1f434f2e0092b704653f2f506a01b2f7b88)
* docs: Adds a redirect for flatten-json which was removed (#16263)
(cherry picked from commit b1568fb95be26ab4c78db836315c14779d163a22)
* docs: add lookups to the sidebar (#16530)
Co-authored-by: 317brian <[email protected]>
(cherry picked from commit 8f78c901e7d70e6908904cd0c3dde305e0448c0a)
* Added new use cases and description of the use case - 5/14/24 (#16451)
Thanks for your contribution @amit-git-account
* Added new use cases and description of the use case - 5/14/24
The use case listing is not changed in a long time. While speaking with
users, I came across several other use cases not listed here in the index. So I
added new use cases and also added description against the use cases.
* Apply suggestions from code review
Co-authored-by: Katya Macedo <[email protected]>
* Apply suggestions from code review
Co-authored-by: Katya Macedo <[email protected]>
* update spelling file
* Update docs/design/index.md
---------
Co-authored-by: Charles Smith <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: 317brian <[email protected]>
Co-authored-by: Benedict Jin <[email protected]>
(cherry picked from commit 540d3e6af5fa289a768dd0fa8da1674e7a665653)
* docs: add subquery limit migration guide to overview page
* fix typo
* update migration guide blurb for subquerybytes
* caps
---------
Co-authored-by: Jill Osborne <[email protected]>
Co-authored-by: Charles Smith <[email protected]>
Co-authored-by: Katya Macedo <[email protected]>
Co-authored-by: Amit <[email protected]>
---
docs/configuration/extensions.md | 4 +-
docs/configuration/index.md | 2 +-
docs/design/index.md | 23 ++++--
docs/ingestion/flatten-json.md | 28 +++++++
docs/ingestion/ingestion-spec.md | 84 +++++++++++++-------
.../kafka-extraction-namespace.md | 6 +-
.../lookups-cached-global.md | 16 ++--
docs/querying/lookups.md | 10 +--
docs/release-info/migr-front-coded-dict.md | 89 ++++++++++++++++++++++
docs/release-info/migr-subquery-limit.md | 60 +++++++++++++++
docs/release-info/migration-guide.md | 43 +++++++++++
website/.spelling | 4 +
website/redirects.js | 7 +-
website/sidebars.json | 27 ++++++-
14 files changed, 345 insertions(+), 58 deletions(-)
diff --git a/docs/configuration/extensions.md b/docs/configuration/extensions.md
index 575ad25d655..29356ebc05c 100644
--- a/docs/configuration/extensions.md
+++ b/docs/configuration/extensions.md
@@ -44,11 +44,11 @@ Core extensions are maintained by Druid committers.
|druid-google-extensions|Google Cloud Storage deep
storage.|[link](../development/extensions-core/google.md)|
|druid-hdfs-storage|HDFS deep
storage.|[link](../development/extensions-core/hdfs.md)|
|druid-histogram|Approximate histograms and quantiles aggregator. Deprecated,
please use the [DataSketches quantiles
aggregator](../development/extensions-core/datasketches-quantiles.md) from the
`druid-datasketches` extension
instead.|[link](../development/extensions-core/approximate-histograms.md)|
-|druid-kafka-extraction-namespace|Apache Kafka-based namespaced lookup.
Requires namespace lookup
extension.|[link](../development/extensions-core/kafka-extraction-namespace.md)|
+|druid-kafka-extraction-namespace|Apache Kafka-based namespaced lookup.
Requires namespace lookup
extension.|[link](../querying/kafka-extraction-namespace.md)|
|druid-kafka-indexing-service|Supervised exactly-once Apache Kafka ingestion
for the indexing service.|[link](../ingestion/kafka-ingestion.md)|
|druid-kinesis-indexing-service|Supervised exactly-once Kinesis ingestion for
the indexing service.|[link](../ingestion/kinesis-ingestion.md)|
|druid-kerberos|Kerberos authentication for druid
processes.|[link](../development/extensions-core/druid-kerberos.md)|
-|druid-lookups-cached-global|A module for [lookups](../querying/lookups.md)
providing a jvm-global eager caching for lookups. It provides JDBC and URI
implementations for fetching lookup
data.|[link](../development/extensions-core/lookups-cached-global.md)|
+|druid-lookups-cached-global|A module for [lookups](../querying/lookups.md)
providing a jvm-global eager caching for lookups. It provides JDBC and URI
implementations for fetching lookup
data.|[link](../querying/lookups-cached-global.md)|
|druid-lookups-cached-single| Per lookup caching module to support the use
cases where a lookup need to be isolated from the global pool of lookups
|[link](../development/extensions-core/druid-lookups.md)|
|druid-multi-stage-query| Support for the multi-stage query architecture for
Apache Druid and the multi-stage query task
engine.|[link](../multi-stage-query/index.md)|
|druid-orc-extensions|Support for data in Apache ORC data
format.|[link](../development/extensions-core/orc.md)|
diff --git a/docs/configuration/index.md b/docs/configuration/index.md
index 857e76611b1..d204c24cf90 100644
--- a/docs/configuration/index.md
+++ b/docs/configuration/index.md
@@ -627,7 +627,7 @@ the [HTTP input
source](../ingestion/input-sources.md#http-input-source).
You can use the following properties to specify permissible JDBC options for:
- [SQL input source](../ingestion/input-sources.md#sql-input-source)
-- [globally cached JDBC
lookups](../development/extensions-core/lookups-cached-global.md#jdbc-lookup)
+- [globally cached JDBC
lookups](../querying/lookups-cached-global.md#jdbc-lookup)
- [JDBC Data Fetcher for per-lookup
caching](../development/extensions-core/druid-lookups.md#data-fetcher-layer).
These properties do not apply to metadata storage connections.
diff --git a/docs/design/index.md b/docs/design/index.md
index e49e158e8b5..4d4655a9b1b 100644
--- a/docs/design/index.md
+++ b/docs/design/index.md
@@ -28,13 +28,20 @@ Druid is commonly used as the database backend for GUIs of
analytical applicatio
Common application areas for Druid include:
-- Clickstream analytics including web and mobile analytics
-- Network telemetry analytics including network performance monitoring
-- Server metrics storage
-- Supply chain analytics including manufacturing metrics
-- Application performance metrics
-- Digital marketing/advertising analytics
-- Business intelligence/OLAP
+|Use Case|Description|
+|-----------------|-------------------|
+|Clickstream analytics|Analyze user behavior on websites and mobile
applications to understand navigation patterns, popular content, and user
engagement|
+|Network telemetry analytics|Monitor and analyze network traffic and
performance metrics to optimize network efficiency, identify bottlenecks, and
ensure quality of service|
+|Server metrics storage|Collect and store performance metrics such as CPU
usage, memory usage, disk I/O, and network activity to monitor server health
and optimize resource allocation|
+|Supply chain analytics|Use data from various stages of the supply chain to
optimize inventory management, streamline logistics, forecast demand, and
improve overall operational efficiency|
+|Application performance metrics|Monitor and analyze the performance of
software applications to identify areas for improvement, troubleshoot issues,
and ensure optimal user experience|
+|Digital marketing/advertising analytics|Track and analyze the effectiveness
of digital marketing campaigns and advertising efforts across various channels,
such as social media, search engines, and display ads|
+|Business intelligence (BI)/OLAP (Online Analytical Processing)|Use data
analysis tools and techniques to gather insights from large datasets, generate
reports, and make data-driven decisions to improve business operations and
strategy|
+|Customer analytics|Analyze customer data to understand preferences, behavior,
and purchasing patterns, enabling personalized marketing strategies, improved
customer service, and customer retention efforts|
+|IoT (Internet of Things) analytics|Process and analyze data generated by IoT
devices to gain insights into device performance, user behavior, and
environmental conditions, facilitating automation, optimization, and predictive
maintenance|
+|Financial analytics| Evaluate finance data to gauge financial performance,
manage risk, detect fraud, and make informed investment decisions|
+|Healthcare analytics|Analyze healthcare data to improve patient outcomes,
optimize healthcare delivery, reduce costs, and identify trends and patterns in
diseases and treatments|
+|Social media analytics|Monitor and analyze social media activity, such as
likes, shares, comments, and mentions, to understand audience sentiment, track
brand perception, and identify influencers|
If you are experimenting with a new use case for Druid or have questions about
Druid's capabilities and features, join the [Apache Druid
Slack](http://apachedruidworkspace.slack.com/) channel. There, you can connect
with Druid experts, ask questions, and get help in real time.
@@ -94,4 +101,4 @@ taking a long time to complete.
## Learn more
- Try the Druid [Quickstart](../tutorials/index.md).
- Learn more about Druid components in [Design](../design/architecture.md).
-- Read about new features and other details of [Druid
Releases](https://github.com/apache/druid/releases).
\ No newline at end of file
+- Read about new features and improvements in [Druid
Releases](https://github.com/apache/druid/releases).
diff --git a/docs/ingestion/flatten-json.md b/docs/ingestion/flatten-json.md
new file mode 100644
index 00000000000..2ac2f023071
--- /dev/null
+++ b/docs/ingestion/flatten-json.md
@@ -0,0 +1,28 @@
+---
+id: flatten-json
+title: "Redirecting"
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+ <head>
+<script>window.location.replace("/docs/latest/ingestion/data-formats#flattenspec")</script>
+</head>
+<a href="/docs/latest/ingestion/data-formats#flattenspec">Click here if you
are not redirected.</a>
\ No newline at end of file
diff --git a/docs/ingestion/ingestion-spec.md b/docs/ingestion/ingestion-spec.md
index 6d1f9609d85..0d9a82d6b3c 100644
--- a/docs/ingestion/ingestion-spec.md
+++ b/docs/ingestion/ingestion-spec.md
@@ -471,14 +471,27 @@ An example `ioConfig` to read JSON data is:
...
}
```
-For more details, see the documentation provided by each [ingestion
method](./index.md#ingestion-methods).
+
+For details, see the documentation provided by each [ingestion
method](./index.md#ingestion-methods).
## `tuningConfig`
-Tuning properties are specified in a `tuningConfig`, which goes at the top
level of an ingestion spec. Some
-properties apply to all [ingestion methods](./index.md#ingestion-methods), but
most are specific to each individual
-ingestion method. An example `tuningConfig` that sets all of the shared,
common properties to their defaults
-is:
+You specify tuning properties in a `tuningConfig` object, which goes at the
top level of an ingestion spec.
+Some properties apply to all [ingestion
methods](./index.md#ingestion-methods), but most are specific to each
individual ingestion method.
+
+The following table lists the common tuning properties shared among ingestion
methods:
+
+|Field|Description|Default|
+|-----|-----------|-------|
+|type|Each ingestion method has its own tuning type code. You must specify the
type code that matches your ingestion method. Common options are `index`,
`hadoop`, `kafka`, and `kinesis`.||
+|maxRowsInMemory|The maximum number of records to store in memory before
persisting to disk. Note that this is the number of rows post-rollup, and so it
may not be equal to the number of input records. Ingested records will be
persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are
reached (whichever happens first).|`1000000`|
+|maxBytesInMemory|The maximum aggregate size of records, in bytes, to store in
the JVM heap before persisting. This is based on a rough estimate of memory
usage. Ingested records will be persisted to disk when either `maxRowsInMemory`
or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory`
also includes heap usage of artifacts created from intermediary persists. This
means that after every persist, the amount of `maxBytesInMemory` until the next
persist will decre [...]
+|skipBytesInMemoryOverheadCheck|The calculation of maxBytesInMemory takes into
account overhead objects created during ingestion and each intermediate
persist. Setting this to true can exclude the bytes of these overhead objects
from maxBytesInMemory check.|false|
+|indexSpec|Defines segment storage format options to use at indexing time.|See
[`indexSpec`](#indexspec) for more information.|
+|indexSpecForIntermediatePersists|Defines segment storage format options to
use at indexing time for intermediate persisted temporary segments.|See
[`indexSpec`](#indexspec) for more information.|
+|Other properties|Each ingestion method has its own list of additional tuning
properties. See the documentation for each method for a full list: [Kafka
indexing service](../ingestion/kafka-ingestion.md#tuning-configuration),
[Kinesis indexing
service](../ingestion/kinesis-ingestion.md#tuning-configuration), [Native
batch](native-batch.md#tuningconfig), and
[Hadoop-based](hadoop.md#tuningconfig).||
+
+The following example shows a `tuningConfig` object that sets all of the
shared common properties to their defaults:
```plaintext
"tuningConfig": {
@@ -495,38 +508,57 @@ is:
}
```
-|Field|Description|Default|
-|-----|-----------|-------|
-|type|Each ingestion method has its own tuning type code. You must specify the
type code that matches your ingestion method. Common options are `index`,
`hadoop`, `kafka`, and `kinesis`.||
-|maxRowsInMemory|The maximum number of records to store in memory before
persisting to disk. Note that this is the number of rows post-rollup, and so it
may not be equal to the number of input records. Ingested records will be
persisted to disk when either `maxRowsInMemory` or `maxBytesInMemory` are
reached (whichever happens first).|`1000000`|
-|maxBytesInMemory|The maximum aggregate size of records, in bytes, to store in
the JVM heap before persisting. This is based on a rough estimate of memory
usage. Ingested records will be persisted to disk when either `maxRowsInMemory`
or `maxBytesInMemory` are reached (whichever happens first). `maxBytesInMemory`
also includes heap usage of artifacts created from intermediary persists. This
means that after every persist, the amount of `maxBytesInMemory` until the next
persist will decre [...]
-|skipBytesInMemoryOverheadCheck|The calculation of maxBytesInMemory takes into
account overhead objects created during ingestion and each intermediate
persist. Setting this to true can exclude the bytes of these overhead objects
from maxBytesInMemory check.|false|
-|indexSpec|Defines segment storage format options to use at indexing time.|See
[`indexSpec`](#indexspec) for more information.|
-|indexSpecForIntermediatePersists|Defines segment storage format options to
use at indexing time for intermediate persisted temporary segments.|See
[`indexSpec`](#indexspec) for more information.|
-|Other properties|Each ingestion method has its own list of additional tuning
properties. See the documentation for each method for a full list: [Kafka
indexing service](../ingestion/kafka-ingestion.md#tuning-configuration),
[Kinesis indexing
service](../ingestion/kinesis-ingestion.md#tuning-configuration), [Native
batch](native-batch.md#tuningconfig), and
[Hadoop-based](hadoop.md#tuningconfig).||
-
### `indexSpec`
-The `indexSpec` object can include the following properties:
+The `indexSpec` object can include the following properties.
+For information on defining an `indexSpec` in a query context, see [SQL-based
ingestion reference](../multi-stage-query/reference.md#context-parameters).
|Field|Description|Default|
|-----|-----------|-------|
|bitmap|Compression format for bitmap indexes. Should be a JSON object with
`type` set to `roaring` or `concise`.|`{"type": "roaring"}`|
-|dimensionCompression|Compression format for dimension columns. Options are
`lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-|stringDictionaryEncoding|Encoding format for STRING value dictionaries used
by STRING and COMPLEX<json> columns. <br /><br />Example to enable front
coding: `{"type":"frontCoded", "bucketSize": 4}`<br />`bucketSize` is the
number of values to place in a bucket to perform delta encoding. Must be a
power of 2, maximum is 128. Defaults to 4.<br /> `formatVersion` can specify
older versions for backwards compatibility during rolling upgrades, valid
options are `0` and `1`. Defaults to [...]
+|dimensionCompression|Compression format for dimension columns. One of `lz4`,
`lzf`, `zstd`, or `uncompressed`.|`lz4`|
+|stringDictionaryEncoding|Encoding format for string value dictionaries used
by STRING and [COMPLEX<json>](../querying/nested-columns.md) columns. To
enable front coding, set `stringDictionaryEncoding.type` to `frontCoded`.
Optionally, you can specify the `bucketSize` and `formatVersion` properties.
See [Front coding](#front-coding) for more information.|`{"type":"utf8"}`|
|metricCompression|Compression format for primitive type metric columns.
Options are `lz4`, `lzf`, `zstd`, `uncompressed`, or `none` (which is more
efficient than `uncompressed`, but not supported by older versions of
Druid).|`lz4`|
|longEncoding|Encoding format for long-typed columns. Applies regardless of
whether they are dimensions or metrics. Options are `auto` or `longs`. `auto`
encodes the values using offset or lookup table depending on column
cardinality, and store them with variable size. `longs` stores the value as-is
with 8 bytes each.|`longs`|
|jsonCompression|Compression format to use for nested column raw data. Options
are `lz4`, `lzf`, `zstd`, or `uncompressed`.|`lz4`|
-##### Front coding
-
-Front coding is an experimental feature starting in version 25.0. Front coding
is an incremental encoding strategy that Druid can use to store STRING and
[COMPLEX<json>](../querying/nested-columns.md) columns. It allows Druid
to create smaller UTF-8 encoded segments with very little performance cost.
-
-You can enable front coding with all types of ingestion. For information on
defining an `indexSpec` in a query context, see [SQL-based ingestion
reference](../multi-stage-query/reference.md#context-parameters).
+#### Front coding
:::info
- Front coding was originally introduced in Druid 25.0, and an improved
'version 1' was introduced in Druid 26.0, with typically faster read speed and
smaller storage size. The current recommendation is to enable it in a staging
environment and fully test your use case before using in production. By
default, segments created with front coding enabled in Druid 26.0 are backwards
compatible with Druid 25.0, but those created with Druid 26.0 or 25.0 are not
compatible with Druid versions old [...]
+Front coding is an [experimental feature](../development/experimental.md).
:::
-Beyond these properties, each ingestion method has its own specific tuning
properties. See the documentation for each
-[ingestion method](./index.md#ingestion-methods) for details.
+Druid encodes string columns into dictionaries for better compression.
+Front coding is an incremental encoding strategy that lets you store STRING
and [COMPLEX<json>](../querying/nested-columns.md) columns in Druid with
minimal performance impact.
+Front-coded dictionaries reduce storage and improve performance by optimizing
for strings where the front part looks similar.
+For example, if you are tracking website visits, most URLs start with
`https://domain.xyz/`, and front coding is able to exploit this pattern for
more optimal compression when storing such datasets.
+Druid performs the optimization automatically, which means that the
performance of string columns is generally not affected when they don't match
the front-coded pattern.
+Consequently, you can enable this feature universally without having to know
the underlying data shapes of the columns.
+
+You can use front coding with all types of ingestion.
+
+##### Enable front coding
+
+Before you enable front coding for your cluster, review the [Migration guide
for front-coded dictionaries](../release-info/migr-front-coded-dict.md).
+It contains important information about compatibility with Druid versions
preceding 25.0.0.
+
+To enable front coding, set `indexSpec.stringDictionaryEncoding.type` to
`frontCoded` in the `tuningConfig` object of your [ingestion
spec](../ingestion/ingestion-spec.md).
+
+You can specify the following optional properties:
+
+* `bucketSize`: Number of values to place in a bucket to perform delta
encoding. Setting this property instructs indexing tasks to write segments
using compressed dictionaries of the specified bucket size. You can set it to
any power of 2 less than or equal to 128. `bucketSize` defaults to 4.
+* `formatVersion`: Specifies which front coding version to use. Options are 0
and 1 (supported for Druid versions 26.0.0 and higher). `formatVersion`
defaults to 0. For faster speeds and smaller storage sizes, set `formatVersion`
to 1. After setting `formatVersion` to 1, you can no longer downgrade to Druid
25.0.0 seamlessly. To downgrade to Druid 25.0.0, you must re-ingest your data
with the `formatVersion` property set to 0.
+
+For example:
+
+```
+"tuningConfig": {
+ "indexSpec": {
+ "stringDictionaryEncoding": {
+ "type":"frontCoded",
+ "bucketSize": 4,
+ "formatVersion": 0
+ }
+ }
+}
+```
\ No newline at end of file
diff --git a/docs/development/extensions-core/kafka-extraction-namespace.md
b/docs/querying/kafka-extraction-namespace.md
similarity index 92%
rename from docs/development/extensions-core/kafka-extraction-namespace.md
rename to docs/querying/kafka-extraction-namespace.md
index 2d841dfc943..1cfa91aac55 100644
--- a/docs/development/extensions-core/kafka-extraction-namespace.md
+++ b/docs/querying/kafka-extraction-namespace.md
@@ -22,7 +22,7 @@ title: "Apache Kafka Lookups"
~ under the License.
-->
-To use this Apache Druid extension,
[include](../../configuration/extensions.md#loading-extensions)
`druid-lookups-cached-global` and `druid-kafka-extraction-namespace` in the
extensions load list.
+To use this Apache Druid extension,
[include](../configuration/extensions.md#loading-extensions)
`druid-lookups-cached-global` and `druid-kafka-extraction-namespace` in the
extensions load list.
If you need updates to populate as promptly as possible, it is possible to
plug into a Kafka topic whose key is the old value and message is the desired
new value (both in UTF-8) as a LookupExtractorFactory.
@@ -41,13 +41,13 @@ If you need updates to populate as promptly as possible, it
is possible to plug
| `kafkaTopic` | The Kafka topic to read the data from
| Yes ||
| `kafkaProperties` | Kafka consumer properties (`bootstrap.servers` must be
specified) | Yes ||
| `connectTimeout` | How long to wait for an initial connection
| No | `0` (do not wait) |
-| `isOneToOne` | The map is a one-to-one (see [Lookup
DimensionSpecs](../../querying/dimensionspecs.md)) | No | `false`
|
+| `isOneToOne` | The map is a one-to-one (see [Lookup
DimensionSpecs](./dimensionspecs.md)) | No | `false` |
The extension `kafka-extraction-namespace` enables reading from an [Apache
Kafka](https://kafka.apache.org/) topic which has name/key pairs to allow
renaming of dimension values. An example use case would be to rename an ID to a
human-readable format.
## How it Works
-The extractor works by consuming the configured Kafka topic from the
beginning, and appending every record to an internal map. The key of the Kafka
record is used as they key of the map, and the payload of the record is used as
the value. At query time, a lookup can be used to transform the key into the
associated value. See [lookups](../../querying/lookups.md) for how to configure
and use lookups in a query. Keys and values are both stored as strings by the
lookup extractor.
+The extractor works by consuming the configured Kafka topic from the
beginning, and appending every record to an internal map. The key of the Kafka
record is used as they key of the map, and the payload of the record is used as
the value. At query time, a lookup can be used to transform the key into the
associated value. See [lookups](./lookups.md) for how to configure and use
lookups in a query. Keys and values are both stored as strings by the lookup
extractor.
The extractor remains subscribed to the topic, so new records are added to the
lookup map as they appear. This allows for lookup values to be updated in
near-realtime. If two records are added to the topic with the same key, the
record with the larger offset will replace the previous record in the lookup
map. A record with a `null` payload will be treated as a tombstone record, and
the associated key will be removed from the lookup map.
diff --git a/docs/development/extensions-core/lookups-cached-global.md
b/docs/querying/lookups-cached-global.md
similarity index 93%
rename from docs/development/extensions-core/lookups-cached-global.md
rename to docs/querying/lookups-cached-global.md
index 5cfcbea01c2..72c4189c2da 100644
--- a/docs/development/extensions-core/lookups-cached-global.md
+++ b/docs/querying/lookups-cached-global.md
@@ -22,12 +22,12 @@ title: "Globally Cached Lookups"
~ under the License.
-->
-To use this Apache Druid extension,
[include](../../configuration/extensions.md#loading-extensions)
`druid-lookups-cached-global` in the extensions load list.
+To use this Apache Druid extension,
[include](../configuration/extensions.md#loading-extensions)
`druid-lookups-cached-global` in the extensions load list.
## Configuration
:::info
Static configuration is no longer supported. Lookups can be configured through
- [dynamic configuration](../../querying/lookups.md#configuration).
+ [dynamic configuration](./lookups.md#configuration).
:::
Globally cached lookups are appropriate for lookups which are not possible to
pass at query time due to their size,
@@ -36,7 +36,7 @@ and are small enough to reasonably populate in-memory. This
usually means tens t
Globally cached lookups all draw from the same cache pool, allowing each
process to have a fixed cache pool that can be used by cached lookups.
-Globally cached lookups can be specified as part of the [cluster wide config
for lookups](../../querying/lookups.md) as a type of `cachedNamespace`
+Globally cached lookups can be specified as part of the [cluster wide config
for lookups](./lookups.md) as a type of `cachedNamespace`
```json
{
@@ -84,7 +84,7 @@ The parameters are as follows
|--------|-----------|--------|-------|
|`extractionNamespace`|Specifies how to populate the local cache. See
below|Yes|-|
|`firstCacheTimeout`|How long to wait (in ms) for the first run of the cache
to populate. 0 indicates to not wait|No|`0` (do not wait)|
-|`injective`|If the underlying map is
[injective](../../querying/lookups.md#query-rewrites) (keys and values are
unique) then optimizations can occur internally by setting this to
`true`|No|`false`|
+|`injective`|If the underlying map is [injective](./lookups.md#query-rewrites)
(keys and values are unique) then optimizations can occur internally by setting
this to `true`|No|`false`|
If `firstCacheTimeout` is set to a non-zero value, it should be less than
`druid.manager.lookups.hostUpdateTimeout`. If `firstCacheTimeout` is NOT set,
then management is essentially asynchronous and does not know if a lookup
succeeded or failed in starting. In such a case logs from the processes using
lookups should be monitored for repeated failures.
@@ -93,7 +93,7 @@ Proper functionality of globally cached lookups requires the
following extension
## Example configuration
-In a simple case where only one
[tier](../../querying/lookups.md#dynamic-configuration) exists
(`realtime_customer2`) with one `cachedNamespace` lookup called `country_code`,
the resulting configuration JSON looks similar to the following:
+In a simple case where only one [tier](./lookups.md#dynamic-configuration)
exists (`realtime_customer2`) with one `cachedNamespace` lookup called
`country_code`, the resulting configuration JSON looks similar to the following:
```json
{
@@ -170,7 +170,7 @@ It's highly recommended that
`druid.lookup.namespace.numBufferedEntries` is set
## Supported lookups
-For additional lookups, please see our [extensions
list](../../configuration/extensions.md).
+For additional lookups, please see our [extensions
list](../configuration/extensions.md).
### URI lookup
@@ -345,7 +345,7 @@ The JDBC lookups will poll a database to populate its local
cache. If the `tsCol
|Parameter|Description|Required|Default|
|---------|-----------|--------|-------|
-|`connectorConfig`|The connector config to use. You can set `connectURI`,
`user` and `password`. You can selectively allow JDBC properties in
`connectURI`. See [JDBC connections security
config](../../configuration/index.md#jdbc-connections-to-external-databases)
for more details.|Yes||
+|`connectorConfig`|The connector config to use. You can set `connectURI`,
`user` and `password`. You can selectively allow JDBC properties in
`connectURI`. See [JDBC connections security
config](../configuration/index.md#jdbc-connections-to-external-databases) for
more details.|Yes||
|`table`|The table which contains the key value pairs|Yes||
|`keyColumn`|The column in `table` which contains the keys|Yes||
|`valueColumn`|The column in `table` which contains the values|Yes||
@@ -377,7 +377,7 @@ The JDBC lookups will poll a database to populate its local
cache. If the `tsCol
:::info
If using JDBC, you will need to add your database's client JAR files to the
extension's directory.
For Postgres, the connector JAR is already included.
- See the MySQL extension documentation for instructions to obtain
[MySQL](./mysql.md#installing-the-mysql-connector-library) or
[MariaDB](./mysql.md#alternative-installing-the-mariadb-connector-library)
connector libraries.
+ See the MySQL extension documentation for instructions to obtain
[MySQL](../development/extensions-core/mysql.md#installing-the-mysql-connector-library)
or
[MariaDB](../development/extensions-core/mysql.md#alternative-installing-the-mariadb-connector-library)
connector libraries.
The connector JAR should reside in the classpath of Druid's main class loader.
To add the connector JAR to the classpath, you can copy the downloaded file
to `lib/` under the distribution root directory. Alternatively, create a
symbolic link to the connector in the `lib` directory.
:::
diff --git a/docs/querying/lookups.md b/docs/querying/lookups.md
index bbc1b03faca..a22fbf03928 100644
--- a/docs/querying/lookups.md
+++ b/docs/querying/lookups.md
@@ -24,7 +24,7 @@ title: "Lookups"
Lookups are a concept in Apache Druid where dimension values are (optionally)
replaced with new values, allowing join-like
functionality. Applying lookups in Druid is similar to joining a dimension
table in a data warehouse. See
-[dimension specs](../querying/dimensionspecs.md) for more information. For the
purpose of these documents, a "key"
+[dimension specs](./dimensionspecs.md) for more information. For the purpose
of these documents, a "key"
refers to a dimension value to match, and a "value" refers to its replacement.
So if you wanted to map
`appid-12345` to `Super Mega Awesome App` then the key would be `appid-12345`
and the value would be
`Super Mega Awesome App`.
@@ -43,12 +43,12 @@ and such data belongs in the raw denormalized data for use
in Druid.
Lookups are generally preloaded in-memory on all servers. But very small
lookups (on the order of a few dozen to a few
hundred entries) can also be passed inline in native queries time using the
"map" lookup type. Refer to the
-[dimension specs](dimensionspecs.md) documentation for details.
+[dimension specs](./dimensionspecs.md) documentation for details.
Other lookup types are available as extensions, including:
-- Globally cached lookups from local files, remote URIs, or JDBC through
[lookups-cached-global](../development/extensions-core/lookups-cached-global.md).
-- Globally cached lookups from a Kafka topic through
[kafka-extraction-namespace](../development/extensions-core/kafka-extraction-namespace.md).
+- Globally cached lookups from local files, remote URIs, or JDBC through
[lookups-cached-global](./lookups-cached-global.md).
+- Globally cached lookups from a Kafka topic through
[kafka-extraction-namespace](./kafka-extraction-namespace.md).
Query Syntax
------------
@@ -213,7 +213,7 @@ Injective lookups are eligible for the largest set of query
rewrites. Injective
function may encounter null input values.
To determine whether a lookup is injective, Druid relies on an `injective`
property that you can set in the
-[lookup definition](../development/extensions-core/lookups-cached-global.md).
In general, you should set
+[lookup definition](./lookups-cached-global.md). In general, you should set
`injective: true` for any lookup that satisfies the required properties, to
allow Druid to run your queries as fast as
possible.
diff --git a/docs/release-info/migr-front-coded-dict.md
b/docs/release-info/migr-front-coded-dict.md
new file mode 100644
index 00000000000..1c825d4f6e3
--- /dev/null
+++ b/docs/release-info/migr-front-coded-dict.md
@@ -0,0 +1,89 @@
+---
+id: migr-front-coded-dict
+title: "Migration guide: front-coded dictionaries"
+sidebar_label: Front-coded dictionaries
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+:::info
+Front coding is an [experimental feature](../development/experimental.md)
introduced in Druid 25.0.0.
+:::
+
+Apache Druid encodes string columns into dictionaries for better compression.
+Front coding is an incremental encoding strategy that lets you store STRING
and [COMPLEX<json>](../querying/nested-columns.md) columns in Druid with
minimal performance impact.
+Front-coded dictionaries reduce storage and improve performance by optimizing
for strings where the front part looks similar.
+For example, if you are tracking website visits, most URLs start with
`https://domain.xyz/`, and front coding is able to exploit this pattern for
more optimal compression when storing such datasets.
+Druid performs the optimization automatically, which means that the
performance of string columns is generally not affected when they don't match
the front-coded pattern.
+Consequently, you can enable this feature universally without having to know
the underlying data shapes of the columns.
+
+You can use front coding with all types of ingestion.
+
+## Enable front coding
+
+To enable front coding, set `indexSpec.stringDictionaryEncoding.type` to
`frontCoded` in the `tuningConfig` object of your [ingestion
spec](../ingestion/ingestion-spec.md).
+
+You can specify the following optional properties:
+
+* `bucketSize`: Number of values to place in a bucket to perform delta
encoding. Setting this property instructs indexing tasks to write segments
using compressed dictionaries of the specified bucket size. You can set it to
any power of 2 less than or equal to 128. `bucketSize` defaults to 4.
+* `formatVersion`: Specifies which front coding version to use. Options are 0
and 1 (supported for Druid versions 26.0.0 and higher). `formatVersion`
defaults to 0.
+
+For example:
+
+```
+"tuningConfig": {
+ "indexSpec": {
+ "stringDictionaryEncoding": {
+ "type":"frontCoded",
+ "bucketSize": 4,
+ "formatVersion": 0
+ }
+ }
+}
+```
+
+## Upgrade from Druid 25.0.0
+
+Druid 26.0.0 introduced a new version of the front-coded dictionary, version
1, offering typically faster read speeds and smaller storage sizes.
+When upgrading to versions Druid 26.0.0 and higher, Druid continues to default
front coding settings to version 0.
+This default enables seamless downgrades to Druid 25.0.0.
+
+To use the newer version, set the `formatVersion` property to 1:
+
+```
+"tuningConfig": {
+ "indexSpec": {
+ "stringDictionaryEncoding": {
+ "type":"frontCoded",
+ "bucketSize": 4,
+ "formatVersion": 1
+ }
+ }
+}
+```
+
+## Downgrade to Druid 25.0.0
+
+After upgrading to version 1, you can no longer downgrade to Druid 25.0.0
seamlessly.
+To downgrade to Druid 25.0.0, re-ingest your data with the
`stringDictionaryEncoding.formatVersion` property set to 0.
+
+## Downgrade to a version preceding Druid 25.0.0
+
+Druid versions preceding 25.0.0 can't read segments with front-coded
dictionaries. To downgrade to an older version, you must either delete the
segments containing front-coded dictionaries or re-ingest them with
`stringDictionaryEncoding.type` set to `utf8`.
diff --git a/docs/release-info/migr-subquery-limit.md
b/docs/release-info/migr-subquery-limit.md
new file mode 100644
index 00000000000..853be50294d
--- /dev/null
+++ b/docs/release-info/migr-subquery-limit.md
@@ -0,0 +1,60 @@
+---
+id: migr-subquery-limit
+title: "Migration guide: Subquery limit"
+sidebar_label: Subquery limit
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+-->
+
+Druid now allows you to set a byte-based limit on subquery size, to prevent
brokers from running out of memory when handling large subqueries.
+Druid uses subqueries as joins as well as in common table expressions, such as
WITH.
+
+The byte-based subquery limit overrides Druid's row-based subquery limit.
+
+:::info
+We recommend that you move towards using byte-based limits starting in Druid
30.0.
+:::
+
+For queries that generate a large number of rows (5 million or more), we
recommend that you don't use `maxSubqueryBytes` from the outset.
+You can increase `maxSubqueryRows` and then configure the byte-based limit if
you find that Druid needs it to process the query.
+
+## Row-based subquery limit
+
+Druid uses the `maxSubqueryRows` property to limit the number of rows Druid
returns in a subquery.
+Because this is a row-based limit, it doesn't restrict the overall size of the
returned data.
+
+The `maxSubqueryRows` property is set to 100,000 by default.
+
+## Enable a byte-based subquery limit
+
+Set the optional property `maxSubqueryBytes` to set a maximum number of
returned bytes.
+This property takes precedence over `maxSubqueryRows`.
+
+## Usage considerations
+
+You can set both `maxSubqueryRows` and `maxSubqueryBytes` at cluster level and
override them in individual queries.
+See [Overriding default query context
values](../configuration#overriding-default-query-context-values) for more
information.
+
+## Learn more
+
+See the following topics for more information:
+
+- [Query context](../querying/query-context.md) for information on setting
query context parameters.
+- [Broker configuration
reference](../configuration#guardrails-for-materialization-of-subqueries) for
more information on `maxSubqueryRows` and `maxSubqueryBytes`.
diff --git a/docs/release-info/migration-guide.md
b/docs/release-info/migration-guide.md
new file mode 100644
index 00000000000..639ccb086b5
--- /dev/null
+++ b/docs/release-info/migration-guide.md
@@ -0,0 +1,43 @@
+---
+id: migration-guide
+title: "Migration guides"
+description: How to migrate from legacy features to get the most from Druid
updates
+---
+
+<!--
+ ~ Licensed to the Apache Software Foundation (ASF) under one
+ ~ or more contributor license agreements. See the NOTICE file
+ ~ distributed with this work for additional information
+ ~ regarding copyright ownership. The ASF licenses this file
+ ~ to you under the Apache License, Version 2.0 (the
+ ~ "License"); you may not use this file except in compliance
+ ~ with the License. You may obtain a copy of the License at
+ ~
+ ~ http://www.apache.org/licenses/LICENSE-2.0
+ ~
+ ~ Unless required by applicable law or agreed to in writing,
+ ~ software distributed under the License is distributed on an
+ ~ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ ~ KIND, either express or implied. See the License for the
+ ~ specific language governing permissions and limitations
+ ~ under the License.
+ -->
+
+In general, when we introduce new features and behaviors into Apache Druid, we
make every effort to avoid breaking existing features when introducing new
behaviors. However, sometimes there are either bugs or performance limitations
with the old behaviors that are not possible to fix in a backward-compatible
way. In these cases, we must introduce breaking changes for the future
maintainability of Druid.
+
+The guides in this section outline breaking changes introduced in Druid 25 and
later. Each guide provides instructions to migrate to new features.
+
+<!--
+
+## Migrate to arrays from multi-value dimensions
+
+Druid now supports SQL-compliant array types. Whenever possible, you should
use the array type over multi-value dimensions. See []()>.
+-->
+
+## Migrate to front-coded dictionary encoding
+
+Druid encodes string columns into dictionaries for better compression.
Front-coded dictionary encoding reduces storage and improves performance by
optimizing for strings that share similar beginning substrings. See [Migration
guide: front-coded dictionaries](migr-front-coded-dict.md) for more information.
+
+## Migrate to `maxSubqueryBytes` from `maxSubqueryRows`
+
+Druid allows you to set a byte-based limit on subquery size to prevent Brokers
from running out of memory when handling large subqueries. The byte-based
subquery limit overrides Druid's row-based subquery limit. We recommend that
you move towards using byte-based limits starting in Druid 30.0. See [Migration
guide: subquery limit](migr-subquery-limit.md) for more information.
\ No newline at end of file
diff --git a/website/.spelling b/website/.spelling
index 7098eb1b5f3..bf32856574e 100644
--- a/website/.spelling
+++ b/website/.spelling
@@ -125,6 +125,7 @@ IANA
IcebergFilter
IcebergInputSource
IETF
+IoT
IP
IPv4
IPv6
@@ -363,6 +364,8 @@ https
idempotency
i.e.
influxdb
+influencer
+influencers
ingestions
ingestionSpec
injective
@@ -542,6 +545,7 @@ subqueries
subquery
subsecond
substring
+substrings
subtask
subtasks
supervisorTaskId
diff --git a/website/redirects.js b/website/redirects.js
index bcbe53cb51e..e2bb4da7b68 100644
--- a/website/redirects.js
+++ b/website/redirects.js
@@ -148,8 +148,11 @@ const Redirects=[
"to": "/docs/latest/querying/scan-query"
},
{
- "from": "/docs/latest/development/extensions-core/namespaced-lookup.html",
- "to": "/docs/latest/development/extensions-core/lookups-cached-global"
+ "from": [
+ "/docs/latest/development/extensions-core/namespaced-lookup.html",
+ "/docs/latest/development/extensions-core/lookups-cached-global"
+ ],
+ "to": "/docs/latest/querying/lookups-cached-global"
},
{
"from": "/docs/latest/development/indexer.html",
diff --git a/website/sidebars.json b/website/sidebars.json
index 45631da72a6..1f2ea873fcb 100644
--- a/website/sidebars.json
+++ b/website/sidebars.json
@@ -185,7 +185,16 @@
"items": [
"querying/datasource",
"querying/joins",
- "querying/lookups",
+ {"type": "category",
+ "label": "Lookups",
+ "link": {
+ "type": "doc",
+ "id": "querying/lookups"
+ },
+ "items":[
+ "querying/lookups-cached-global",
+ "querying/kafka-extraction-namespace"
+ ]},
"querying/multi-value-dimensions",
"querying/arrays",
"querying/nested-columns",
@@ -354,8 +363,20 @@
"items":
[
"release-info/release-notes",
- "release-info/upgrade-notes"
+ "release-info/upgrade-notes",
+ {
+ "type": "category",
+ "label": "Migration guides",
+ "link": {
+ "type": "doc",
+ "id": "release-info/migration-guide"
+ },
+ "items": [
+ "release-info/migr-front-coded-dict",
+ "release-info/migr-subquery-limit"
+ ]
+ }
]},
"misc/papers-and-talks"
]
-}
\ No newline at end of file
+}
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
