This is an automated email from the ASF dual-hosted git repository.
reiabreu pushed a change to branch worker-launcher-oci-mount-sources
in repository https://gitbox.apache.org/repos/asf/storm.git
omit 4a71be092 Resolve OCI bind-mount sources with realpath before the
allowed-directory check
omit acac7d6b5 Document the OCI allowed bind-mount source directories
setting
omit bc02b570d Restrict OCI bind-mount sources to configured directories
and check the launch command username
add 40d14c004 Return only the log files the requesting user may access
from the log file listing (#8974)
add c14bfaf21 Add ui.enable.jsonp to control JSONP callback wrapping in UI
and Logviewer API responses (#8977)
add cb8d29b0b Change the worker log file permission only for download
requests that are served (#8978)
add 364dd7913 Reject image tags that are neither mapped nor valid image
hashes before building the manifest path (#8983)
add ce56aef02 Update logging behavior
add 76a59d2a4 Reject blobstore map local names that resolve outside the
topology and worker directories
add 2d972caa3 Stop echoing the request Origin with allow-credentials in
logviewer JSON responses
add 22bd22a6a Reject submitted jar locations that are not inside the
Nimbus inbox
add 54d2c64b7 Verify the peer identity of the Netty TLS server the worker
client connects to
add c78ab7462 Give each uploaded dependency artifact its own blob key
instead of reusing a pre-existing blob
add 6d36a14d9 Require Kryo class registration in the state serializer
add 405092452 Return a normalized copy of the topology conf from Nimbus
add ea78c5bea Add to instead of replace Jetty's default TLS exclusions on
HTTPS connectors
add 27ae976bf Pass only the required Kafka consumer properties to the
spout lag monitor
add f6b7be390 Restrict topology scheduler strategies to the shipped
strategies when no allowlist is configured
add ee97546fb Decode netty server frames only after the handshake completes
add 059ebea9c Document that an unset scheduler strategy whitelist no
longer allows every strategy class
add ed585b884 Set the CORS filter credentials parameter explicitly instead
of a header name that the filter ignores
add 1d68ef649 Add storm-iceberg module: an Apache Iceberg sink bolt (#8950)
add ce21f7990 BugFix Utils.java (#8972)
add d77982d71 Regenerate license files after dependency changes (#8996)
add b7865e47a Apply the configured log user and group lists to daemon log
requests (#8973)
add c1c3338fd Generate tuple tree ids from a key stream instead of a
recoverable linear congruential generator (#8989)
add 4af262d17 Remove DRPC per-function request queues once they are empty
(#8985)
add 053570037 Validate topology.blobstore.map in rebalance conf overrides
against the calling user (#8999)
add b8c71da61 Evaluate nimbus.groups when nimbus.users is empty (#8998)
add 94b408891 Evaluate authorization for requests that carry no principal
(#9005)
add 0441cfc2b Apply the nimbus ACLs to listBlobs so callers only see blobs
they may read (#9000)
add c8a1474a6 Read and check the docker command file before changing
worker directory ownership
add 1b3dcc12d Authorize createStateInZookeeper requests like the other
blob operations
add 9a6490a0f Filter the topology history by the authenticated caller
rather than the requested user name (#9003)
add 5ebe6e717 Check the calling principal against the topology owner in
uploadNewCredentials (#9001)
add 4a1c21a8b STORM-3871: remove a topology's dependency artifact blobs
when it is cleaned up (#9011)
add c8d41cf60 Reject a topology submission that lists a blob key which is
not a dependency blob key (#9012)
add 6dd5355c5 Validate the container id and invoke runc without a shell in
reap-oci-container (#9008)
add 6aa0066ab Apply worker file ownership and mode via an open descriptor
(#9006)
add 07ac4d553 Restrict OCI bind-mount sources to configured directories
and check the launch command username
add 4cd2ef697 Document the OCI allowed bind-mount source directories
setting
add 25c09eee5 Resolve OCI bind-mount sources with realpath before the
allowed-directory check
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (4a71be092)
\
N -- N -- N refs/heads/worker-launcher-oci-mount-sources
(25c09eee5)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
DEPENDENCY-LICENSES | 28 +
conf/defaults.yaml | 15 +
docs/Resource_Aware_Scheduler_overview.md | 2 +
docs/STORM-UI-REST-API.md | 1 +
docs/State-checkpointing.md | 73 +--
docs/index.md | 1 +
docs/storm-iceberg.md | 319 ++++++++++++
examples/pom.xml | 1 +
.../pom.xml | 6 +-
.../storm/iceberg/examples/BoundedTupleSpout.java | 97 ++++
.../examples/IcebergBoltExampleTopology.java | 118 +++++
.../IcebergPartitionedBoltExampleTopology.java | 129 +++++
.../examples/IcebergSplitSinkExampleTopology.java | 129 +++++
external/pom.xml | 1 +
.../oci/LocalOrHdfsImageTagToManifestPlugin.java | 23 +-
.../LocalOrHdfsImageTagToManifestPluginTest.java | 85 ++++
external/storm-iceberg/README.md | 231 +++++++++
external/{storm-jdbc => storm-iceberg}/pom.xml | 49 +-
.../org/apache/storm/iceberg/bolt/IcebergBolt.java | 195 +++++++
.../storm/iceberg/bolt/IcebergCommitterBolt.java | 209 ++++++++
.../storm/iceberg/bolt/IcebergWriterBolt.java | 193 +++++++
.../org/apache/storm/iceberg/common/CommitWal.java | 174 +++++++
.../iceberg/common/CountingAppenderFactory.java | 109 ++++
.../apache/storm/iceberg/common/DataFileCodec.java | 86 ++++
.../iceberg/common/FieldNameRecordMapper.java | 99 ++++
.../storm/iceberg/common/IcebergCommitter.java | 207 ++++++++
.../storm/iceberg/common/IcebergMetrics.java | 113 ++++
.../storm/iceberg/common/IcebergOptions.java | 300 +++++++++++
.../apache/storm/iceberg/common/IcebergWriter.java | 201 ++++++++
.../iceberg/common/PartitionedRecordWriter.java | 54 ++
.../apache/storm/iceberg/common/RecordMapper.java | 36 +-
.../apache/storm/iceberg/bolt/IcebergBoltTest.java | 253 +++++++++
.../iceberg/bolt/IcebergCommitterBoltTest.java | 269 ++++++++++
.../storm/iceberg/bolt/IcebergSplitSinkTest.java | 256 ++++++++++
.../storm/iceberg/bolt/IcebergWriterBoltTest.java | 218 ++++++++
.../apache/storm/iceberg/common/CommitWalTest.java | 146 ++++++
.../storm/iceberg/common/DataFileCodecTest.java | 110 ++++
.../iceberg/common/FieldNameRecordMapperTest.java | 143 ++++++
.../storm/iceberg/common/IcebergCommitterTest.java | 263 ++++++++++
.../storm/iceberg/common/IcebergMetricsTest.java | 96 ++++
.../storm/iceberg/common/IcebergOptionsTest.java | 133 +++++
.../storm/iceberg/common/IcebergWriterTest.java | 270 ++++++++++
.../iceberg/common/RecordingMetricsContext.java | 109 ++++
pom.xml | 32 ++
storm-client/src/jvm/org/apache/storm/Config.java | 13 +-
.../src/jvm/org/apache/storm/StormSubmitter.java | 14 +-
.../storm/dependency/DependencyBlobStoreUtils.java | 18 +-
.../storm/dependency/DependencyUploader.java | 16 +-
.../jvm/org/apache/storm/executor/Executor.java | 3 +-
.../org/apache/storm/messaging/netty/Client.java | 2 +-
.../storm/messaging/netty/MessageDecoder.java | 34 ++
.../storm/messaging/netty/SaslNettyServer.java | 3 +-
.../messaging/netty/SaslStormClientHandler.java | 3 +-
.../messaging/netty/SaslStormServerHandler.java | 3 +-
.../netty/StormClientPipelineFactory.java | 26 +-
.../netty/StormServerPipelineFactory.java | 7 +-
.../auth/authorizer/SimpleACLAuthorizer.java | 15 +-
.../authorizer/SupervisorSimpleACLAuthorizer.java | 6 +-
.../apache/storm/state/DefaultStateSerializer.java | 65 ++-
.../storm/utils/DisallowedStrategyException.java | 4 +
.../org/apache/storm/utils/KeyStreamRandom.java | 107 ++++
.../org/apache/storm/utils/ReflectionUtils.java | 20 +-
.../src/jvm/org/apache/storm/utils/Utils.java | 2 +-
.../storm/dependency/DependencyUploaderTest.java | 49 +-
.../apache/storm/executor/SpoutExecutorTest.java | 28 +
.../storm/messaging/netty/MessageDecoderTest.java | 168 ++++++
.../netty/StormClientPipelineFactoryTest.java | 84 +++
.../auth/authorizer/SimpleACLAuthorizerTest.java | 104 ++++
.../SupervisorSimpleACLAuthorizerTest.java | 65 +++
.../storm/state/DefaultStateSerializerTest.java | 117 +++++
.../jvm/org/apache/storm/tuple/MessageIdTest.java | 99 ++++
.../apache/storm/utils/ReflectionUtilsTest.java | 75 +++
.../test/jvm/org/apache/storm/utils/UtilsTest.java | 57 +++
.../org/apache/storm/utils/TopologySpoutLag.java | 39 +-
storm-core/src/native/worker-launcher/impl/main.c | 33 +-
.../worker-launcher/impl/oci/oci_launch_cmd.c | 2 +-
.../worker-launcher/impl/oci/oci_launch_cmd.h | 8 +
.../src/native/worker-launcher/impl/oci/oci_reap.c | 35 +-
.../native/worker-launcher/impl/worker-launcher.c | 79 ++-
.../native/worker-launcher/impl/worker-launcher.h | 12 +
.../worker-launcher/test/test-worker-launcher.c | 37 ++
.../apache/storm/utils/TopologySpoutLagTest.java | 89 ++++
.../final-package/src/main/assembly/common.xml | 7 +
.../main/java/org/apache/storm/DaemonConfig.java | 8 +
.../java/org/apache/storm/daemon/drpc/DRPC.java | 48 +-
.../org/apache/storm/daemon/nimbus/Nimbus.java | 308 +++++++++--
.../apache/storm/daemon/supervisor/Container.java | 6 +-
.../org/apache/storm/localizer/AsyncLocalizer.java | 4 +-
.../scheduler/resource/ResourceAwareScheduler.java | 4 +-
.../java/org/apache/storm/utils/ServerUtils.java | 39 ++
.../org/apache/storm/daemon/drpc/DRPCTest.java | 105 ++++
.../storm/daemon/nimbus/NimbusClojurePortTest.java | 4 +-
.../daemon/nimbus/NimbusGetTopologyConfTest.java | 122 +++++
.../org/apache/storm/daemon/nimbus/NimbusTest.java | 566 +++++++++++++++++++++
.../storm/daemon/supervisor/ContainerTest.java | 43 ++
.../apache/storm/localizer/AsyncLocalizerTest.java | 72 +++
.../org/apache/storm/utils/ServerUtilsTest.java | 21 +
.../handler/LogviewerLogDownloadHandler.java | 5 +-
.../logviewer/handler/LogviewerLogPageHandler.java | 21 +-
.../handler/LogviewerLogSearchHandler.java | 3 +-
.../daemon/logviewer/utils/LogFileDownloader.java | 14 +-
.../logviewer/utils/LogviewerResponseBuilder.java | 17 +-
.../daemon/logviewer/utils/ResourceAuthorizer.java | 35 ++
.../java/org/apache/storm/daemon/ui/UIHelpers.java | 26 +-
.../handler/LogviewerLogDownloadHandlerTest.java | 134 ++++-
.../handler/LogviewerLogPageHandlerTest.java | 99 +++-
.../handler/LogviewerLogSearchHandlerTest.java | 51 ++
.../utils/LogviewerResponseBuilderTest.java | 46 ++
.../logviewer/utils/ResourceAuthorizerTest.java | 114 +++++
.../org/apache/storm/daemon/ui/UIHelpersTest.java | 74 ++-
110 files changed, 8889 insertions(+), 230 deletions(-)
create mode 100644 docs/storm-iceberg.md
copy examples/{storm-jdbc-examples => storm-iceberg-examples}/pom.xml (96%)
create mode 100644
examples/storm-iceberg-examples/src/main/java/org/apache/storm/iceberg/examples/BoundedTupleSpout.java
create mode 100644
examples/storm-iceberg-examples/src/main/java/org/apache/storm/iceberg/examples/IcebergBoltExampleTopology.java
create mode 100644
examples/storm-iceberg-examples/src/main/java/org/apache/storm/iceberg/examples/IcebergPartitionedBoltExampleTopology.java
create mode 100644
examples/storm-iceberg-examples/src/main/java/org/apache/storm/iceberg/examples/IcebergSplitSinkExampleTopology.java
create mode 100644
external/storm-hdfs-oci/src/test/java/org/apache/storm/container/oci/LocalOrHdfsImageTagToManifestPluginTest.java
create mode 100644 external/storm-iceberg/README.md
copy external/{storm-jdbc => storm-iceberg}/pom.xml (67%)
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/bolt/IcebergBolt.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/bolt/IcebergCommitterBolt.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/bolt/IcebergWriterBolt.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/CommitWal.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/CountingAppenderFactory.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/DataFileCodec.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/FieldNameRecordMapper.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/IcebergCommitter.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/IcebergMetrics.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/IcebergOptions.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/IcebergWriter.java
create mode 100644
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/PartitionedRecordWriter.java
copy
examples/storm-redis-examples/src/main/java/org/apache/storm/redis/trident/WordCountStoreMapper.java
=>
external/storm-iceberg/src/main/java/org/apache/storm/iceberg/common/RecordMapper.java
(56%)
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/bolt/IcebergBoltTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/bolt/IcebergCommitterBoltTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/bolt/IcebergSplitSinkTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/bolt/IcebergWriterBoltTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/CommitWalTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/DataFileCodecTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/FieldNameRecordMapperTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/IcebergCommitterTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/IcebergMetricsTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/IcebergOptionsTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/IcebergWriterTest.java
create mode 100644
external/storm-iceberg/src/test/java/org/apache/storm/iceberg/common/RecordingMetricsContext.java
create mode 100644
storm-client/src/jvm/org/apache/storm/utils/KeyStreamRandom.java
create mode 100644
storm-client/test/jvm/org/apache/storm/messaging/netty/MessageDecoderTest.java
create mode 100644
storm-client/test/jvm/org/apache/storm/messaging/netty/StormClientPipelineFactoryTest.java
create mode 100644
storm-client/test/jvm/org/apache/storm/security/auth/authorizer/SupervisorSimpleACLAuthorizerTest.java
create mode 100644
storm-client/test/jvm/org/apache/storm/tuple/MessageIdTest.java
create mode 100644
storm-client/test/jvm/org/apache/storm/utils/ReflectionUtilsTest.java
create mode 100644
storm-core/test/jvm/org/apache/storm/utils/TopologySpoutLagTest.java
create mode 100644
storm-server/src/test/java/org/apache/storm/daemon/nimbus/NimbusGetTopologyConfTest.java
create mode 100644
storm-webapp/src/test/java/org/apache/storm/daemon/logviewer/utils/LogviewerResponseBuilderTest.java
