This is an automated email from the ASF dual-hosted git repository.
lakshsingla pushed a commit to branch 29.0.0
in repository https://gitbox.apache.org/repos/asf/druid.git
The following commit(s) were added to refs/heads/29.0.0 by this push:
new b45752c4645 Update `groupId` for delta-lake and iceberg extensions
(#15843) (#15868)
b45752c4645 is described below
commit b45752c4645deab65f577f5e508fead6b9bd8edf
Author: Abhishek Radhakrishnan <[email protected]>
AuthorDate: Thu Feb 8 08:16:47 2024 -0800
Update `groupId` for delta-lake and iceberg extensions (#15843) (#15868)
* Update the group id to org.apache.druid.extensions.contrib for contrib
exts.
* Note iceberg and delta lake extensions in extensions.md
* properties and shell backticks
* Update groupId in distribution/pom.xml
* remove delta-lake from dist.
* Add note on downloading extension.
---
distribution/pom.xml | 6 ++----
docs/configuration/extensions.md | 6 ++++--
docs/development/extensions-contrib/delta-lake.md | 17 +++++++++++++++++
docs/development/extensions-contrib/iceberg.md | 17 +++++++++++++++++
extensions-contrib/druid-deltalake-extensions/pom.xml | 2 +-
extensions-contrib/druid-iceberg-extensions/pom.xml | 2 +-
6 files changed, 42 insertions(+), 8 deletions(-)
diff --git a/distribution/pom.xml b/distribution/pom.xml
index 0c14b913e3f..14013a7a5f5 100644
--- a/distribution/pom.xml
+++ b/distribution/pom.xml
@@ -257,8 +257,6 @@
<argument>org.apache.druid.extensions:druid-kubernetes-extensions</argument>
<argument>-c</argument>
<argument>org.apache.druid.extensions:druid-catalog</argument>
- <argument>-c</argument>
-
<argument>org.apache.druid.extensions:druid-deltalake-extensions</argument>
<argument>${druid.distribution.pulldeps.opts}</argument>
</arguments>
</configuration>
@@ -453,9 +451,9 @@
<argument>-c</argument>
<argument>org.apache.druid.extensions.contrib:opentelemetry-emitter</argument>
<argument>-c</argument>
-
<argument>org.apache.druid.extensions:druid-iceberg-extensions</argument>
+
<argument>org.apache.druid.extensions.contrib:druid-iceberg-extensions</argument>
<argument>-c</argument>
-
<argument>org.apache.druid.extensions:druid-deltalake-extensions</argument>
+
<argument>org.apache.druid.extensions.contrib:druid-deltalake-extensions</argument>
<argument>-c</argument>
<argument>org.apache.druid.extensions.contrib:druid-spectator-histogram</argument>
</arguments>
diff --git a/docs/configuration/extensions.md b/docs/configuration/extensions.md
index 21c459ed747..0b09fb75ce6 100644
--- a/docs/configuration/extensions.md
+++ b/docs/configuration/extensions.md
@@ -84,7 +84,9 @@ All of these community extensions can be downloaded using
[pull-deps](../operati
|druid-cloudfiles-extensions|Rackspace Cloudfiles deep storage and
firehose.|[link](../development/extensions-contrib/cloudfiles.md)|
|druid-compressed-bigdecimal|Compressed Big Decimal Type |
[link](../development/extensions-contrib/compressed-big-decimal.md)|
|druid-ddsketch|Support for DDSketch approximate quantiles based on
[DDSketch](https://github.com/datadog/sketches-java) |
[link](../development/extensions-contrib/ddsketch-quantiles.md)|
+|druid-deltalake-extensions|Support for ingesting Delta Lake
tables.|[link](../development/extensions-contrib/delta-lake.md)|
|druid-distinctcount|DistinctCount
aggregator|[link](../development/extensions-contrib/distinctcount.md)|
+|druid-iceberg-extensions|Support for ingesting Iceberg
tables.|[link](../development/extensions-contrib/iceberg.md)|
|druid-redis-cache|A cache implementation for Druid based on
Redis.|[link](../development/extensions-contrib/redis-cache.md)|
|druid-time-min-max|Min/Max aggregator for
timestamp.|[link](../development/extensions-contrib/time-min-max.md)|
|sqlserver-metadata-storage|Microsoft SQLServer deep
storage.|[link](../development/extensions-contrib/sqlserver.md)|
@@ -121,7 +123,7 @@ can load bundled extensions by adding their names to your
common.runtime.propert
`druid.extensions.loadList` property. For example, to load the
postgresql-metadata-storage and
druid-hdfs-storage extensions, use the configuration:
-```
+```properties
druid.extensions.loadList=["postgresql-metadata-storage", "druid-hdfs-storage"]
```
@@ -146,7 +148,7 @@ if they are available from Maven, the included
[pull-deps](../operations/pull-de
specify the full Maven coordinate of the extension in the form
`groupId:artifactId:version`. For example,
for the (hypothetical) extension *com.example:druid-example-extension:1.0.0*,
run:
-```
+```shell
java \
-cp "lib/*" \
-Ddruid.extensions.directory="extensions" \
diff --git a/docs/development/extensions-contrib/delta-lake.md
b/docs/development/extensions-contrib/delta-lake.md
index 8b1de484971..bab07ee3dbb 100644
--- a/docs/development/extensions-contrib/delta-lake.md
+++ b/docs/development/extensions-contrib/delta-lake.md
@@ -38,6 +38,23 @@ These Delta Lake files are versioned Parquet files.
The Delta Lake extension uses the Delta Kernel introduced in Delta Lake 3.0.0,
which is compatible with Apache Spark 3.5.x.
Older versions are unsupported, so consider upgrading to Delta Lake 3.0.x or
higher to use this extension.
+## Downloading Delta Lake extension
+
+To download `druid-deltalake-extensions`, run the following command after
replacing `<VERSION>` with the desired
+Druid version:
+
+```shell
+java \
+ -cp "lib/*" \
+ -Ddruid.extensions.directory="extensions" \
+ -Ddruid.extensions.hadoopDependenciesDir="hadoop-dependencies" \
+ org.apache.druid.cli.Main tools pull-deps \
+ --no-default-hadoop \
+ -c "org.apache.druid.extensions.contrib:druid-deltalake-extensions:<VERSION>"
+```
+
+See [Loading community
extensions](../../configuration/extensions.md#loading-community-extensions) for
more information.
+
## Known limitations
- This extension relies on the Delta Kernel API and can only read from the
latest Delta table snapshot.
diff --git a/docs/development/extensions-contrib/iceberg.md
b/docs/development/extensions-contrib/iceberg.md
index 0c02d6bf709..f4a1735ed3b 100644
--- a/docs/development/extensions-contrib/iceberg.md
+++ b/docs/development/extensions-contrib/iceberg.md
@@ -107,6 +107,23 @@ Since the Hadoop AWS connector uses the `s3a` filesystem
client, specify the war
The local catalog type can be used for catalogs configured on the local
filesystem. Set the `icebergCatalog` type to `local`. You can use this catalog
for demos or localized tests. It is not recommended for production use cases.
The `warehouseSource` is set to `local` because this catalog only supports
reading from a local filesystem.
+## Downloading Iceberg extension
+
+To download `druid-iceberg-extensions`, run the following command after
replacing `<VERSION>` with the desired
+Druid version:
+
+```shell
+java \
+ -cp "lib/*" \
+ -Ddruid.extensions.directory="extensions" \
+ -Ddruid.extensions.hadoopDependenciesDir="hadoop-dependencies" \
+ org.apache.druid.cli.Main tools pull-deps \
+ --no-default-hadoop \
+ -c "org.apache.druid.extensions.contrib:druid-iceberg-extensions:<VERSION>"
+```
+
+See [Loading community
extensions](../../configuration/extensions.md#loading-community-extensions) for
more information.
+
## Known limitations
This section lists the known limitations that apply to the Iceberg extension.
diff --git a/extensions-contrib/druid-deltalake-extensions/pom.xml
b/extensions-contrib/druid-deltalake-extensions/pom.xml
index 14020aa5c8a..9f570ddaf31 100644
--- a/extensions-contrib/druid-deltalake-extensions/pom.xml
+++ b/extensions-contrib/druid-deltalake-extensions/pom.xml
@@ -21,7 +21,7 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <groupId>org.apache.druid.extensions</groupId>
+ <groupId>org.apache.druid.extensions.contrib</groupId>
<artifactId>druid-deltalake-extensions</artifactId>
<name>druid-deltalake-extensions</name>
<description>Delta Lake connector for Druid</description>
diff --git a/extensions-contrib/druid-iceberg-extensions/pom.xml
b/extensions-contrib/druid-iceberg-extensions/pom.xml
index d5be8235d4a..4056f0f6261 100644
--- a/extensions-contrib/druid-iceberg-extensions/pom.xml
+++ b/extensions-contrib/druid-iceberg-extensions/pom.xml
@@ -21,7 +21,7 @@
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <groupId>org.apache.druid.extensions</groupId>
+ <groupId>org.apache.druid.extensions.contrib</groupId>
<artifactId>druid-iceberg-extensions</artifactId>
<name>druid-iceberg-extensions</name>
<description>druid-iceberg-extensions</description>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
