github-actions[bot] commented on code in PR #66297:
URL: https://github.com/apache/doris/pull/66297#discussion_r3699614282
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/metacache/paimon/PaimonLatestSnapshotProjectionLoader.java:
##########
@@ -69,18 +71,70 @@ public PaimonSnapshotCacheValue load(NameMapping
nameMapping, Table paimonTable)
}
}
- private PaimonSnapshot resolveLatestSnapshot(Table paimonTable) {
+ public PaimonSnapshotCacheValue loadFence(NameMapping nameMapping, Table
paimonTable) {
+ try {
+ // A statement fence needs version/schema identity only;
enumerating partitions here
+ // can fail before relation-level options have replaced an unsafe
physical setting.
+ return new PaimonSnapshotCacheValue(
+ PaimonPartitionInfo.EMPTY,
resolveLatestSnapshot(paimonTable, false));
+ } catch (Exception e) {
+ throw new CacheException("failed to load paimon snapshot fence
%s.%s.%s: %s",
+ e, nameMapping.getCtlId(), nameMapping.getLocalDbName(),
nameMapping.getLocalTblName(),
+ e.getMessage());
+ }
+ }
+
+ public PaimonSnapshotCacheValue loadAtFence(NameMapping nameMapping,
PaimonSnapshot fence) {
+ return loadEffectiveAtFence(nameMapping, fence.getTable(), fence);
+ }
+
+ public PaimonSnapshotCacheValue loadEffectiveAtFence(
+ NameMapping nameMapping, Table effectiveTable, PaimonSnapshot
fence) {
+ try {
+ // The fence owns both version and table generation. Reopening the
catalog here can pair
+ // the old snapshot id with a newer schema or branch after
invalidation.
+ FileStoreTable latestSchemaTable = (FileStoreTable) effectiveTable;
+ Table snapshotTable = latestSchemaTable;
+ if (fence.getSnapshotId() != PaimonSnapshot.INVALID_SNAPSHOT_ID) {
+ // Pin data at the statement fence without replaying time
travel, which would
+ // discard a schema-only ALTER that happened after the last
data snapshot.
+ snapshotTable = PaimonReaderOptions.runtimeSafeTable(
+ latestSchemaTable.copyWithoutTimeTravel(
+
PaimonScanParams.isolateSnapshotRead(fence.getSnapshotId())));
+ }
+ List<Column> partitionColumns =
schemaValueLoader.load(nameMapping, fence.getSchemaId())
+ .getPartitionColumns();
+ PaimonPartitionInfo partitionInfo =
+ partitionInfoLoader.load(nameMapping, snapshotTable,
partitionColumns);
+ return new PaimonSnapshotCacheValue(partitionInfo,
+ new PaimonSnapshot(fence.getSnapshotId(),
fence.getSchemaId(), snapshotTable));
+ } catch (Exception e) {
+ throw new CacheException("failed to load paimon snapshot at fence
%s.%s.%s: %s",
+ e, nameMapping.getCtlId(), nameMapping.getLocalDbName(),
nameMapping.getLocalTblName(),
+ e.getMessage());
+ }
+ }
+
+ private PaimonSnapshot resolveLatestSnapshot(Table paimonTable, boolean
normalizeForPartitionLoad) {
FileStoreTable latestSchemaTable = ((FileStoreTable)
paimonTable).copyWithLatestSchema();
Table snapshotTable = latestSchemaTable;
long latestSnapshotId = PaimonSnapshot.INVALID_SNAPSHOT_ID;
Optional<Snapshot> optionalSnapshot =
latestSchemaTable.latestSnapshot();
+ Map<String, String> projectionOptions = Collections.emptyMap();
if (optionalSnapshot.isPresent()) {
latestSnapshotId = optionalSnapshot.get().id();
// Pin the data snapshot for MVCC while retaining the latest table
schema. A normal
// copy applies time travel and falls back to the snapshot's
schema, which can be stale
// immediately after a schema change that has not produced a new
data snapshot.
- snapshotTable = latestSchemaTable.copyWithoutTimeTravel(
-
Collections.singletonMap(CoreOptions.SCAN_SNAPSHOT_ID.key(),
String.valueOf(latestSnapshotId)));
+ projectionOptions =
PaimonScanParams.isolateSnapshotRead(latestSnapshotId);
+ }
+ if (!projectionOptions.isEmpty()) {
+ snapshotTable =
latestSchemaTable.copyWithoutTimeTravel(projectionOptions);
+ }
+ if (normalizeForPartitionLoad) {
+ // Full projection enumerates partitions immediately, so normalize
the complete
+ // planning tree after pinning; the lightweight fence remains
relation-neutral.
+ snapshotTable =
PaimonReaderOptions.runtimeSafeTable(snapshotTable);
}
DataTable dataTable = (DataTable) latestSchemaTable;
long latestSchemaId =
dataTable.schemaManager().latest().map(TableSchema::id).orElse(0L);
Review Comment:
[P1] Derive the schema ID from the table generation captured above. Paimon's
`copyWithLatestSchema()` already reads latest and returns a handle whose
`schema()` is that concrete generation, but this separate live
`schemaManager().latest()` can observe a schema-only ALTER that lands between
the two calls. The fence then pairs table/schema A with schema ID B, so binding
accepts B's added or renamed columns while scan planning serializes A and fails
schema matching. Use `latestSchemaTable.schema().id()` (as the branch path
does) and cover the interleaving.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/StatementContext.java:
##########
@@ -485,11 +485,9 @@ public void registerExternalTableForPreload(TableIf table,
Optional<TableSnapsho
}
ExternalTablePreloadInfo preloadInfo =
externalTablePreloadInfos.computeIfAbsent(table.getId(),
id -> new ExternalTablePreloadInfo((ExternalTable) table));
- boolean selectorFreePaimonOptions = scanParams.isPresent()
- && scanParams.get().isOptions()
- && (table instanceof PaimonExternalTable || table instanceof
PaimonSysExternalTable)
- &&
PaimonScanParams.usesStatementSnapshot(scanParams.get().getMapParams());
- if (tableSnapshot.isPresent() || (scanParams.isPresent() &&
!selectorFreePaimonOptions)) {
+ if (tableSnapshot.isPresent() || scanParams.isPresent()) {
Review Comment:
[P2] Preserve pre-lock fencing for selector-free OPTIONS. This now marks
`t@options('read.batch-size'='4096')` non-latest, so `PreloadExternalMetadata`
does no snapshot/schema warmup; in an internal-table join the planner then
acquires internal read locks before `BindRelation` first calls
`loadLatestSnapshotFence`, whose Paimon metadata reads can hold those locks for
remote latency and delay DDL waiting for the write lock. Explicit historical
selectors should skip latest preload, but OPTIONS for which
`requiresLatestSnapshotFence` is true should preload and reuse the lightweight
fence before locking, with an internal-lock regression test.
##########
fe/be-java-extensions/paimon-connector/src/main/java/org/apache/doris/paimon/PaimonJniScanner.java:
##########
@@ -519,69 +622,271 @@ static Optional<Long> parseDataSizeBytes(String value) {
if (value == null || value.trim().isEmpty()) {
return Optional.empty();
}
- String normalized = value.trim().toLowerCase(Locale.ROOT).replace("_",
"").replace(" ", "");
- int unitStart = 0;
- while (unitStart < normalized.length()
- && (Character.isDigit(normalized.charAt(unitStart)) ||
normalized.charAt(unitStart) == '.')) {
- unitStart++;
- }
- if (unitStart == 0) {
- return Optional.empty();
- }
try {
- double number = Double.parseDouble(normalized.substring(0,
unitStart));
- String unit = normalized.substring(unitStart);
- long multiplier;
- switch (unit) {
- case "":
- case "b":
- case "byte":
- case "bytes":
- multiplier = 1L;
- break;
- case "k":
- case "kb":
- case "kib":
- multiplier = 1024L;
- break;
- case "m":
- case "mb":
- case "mib":
- multiplier = 1024L * 1024L;
- break;
- case "g":
- case "gb":
- case "gib":
- multiplier = 1024L * 1024L * 1024L;
- break;
- case "t":
- case "tb":
- case "tib":
- multiplier = 1024L * 1024L * 1024L * 1024L;
- break;
- default:
- return Optional.empty();
- }
- return Optional.of((long) (number * multiplier));
- } catch (NumberFormatException e) {
+ // Keep the BE guard's accepted grammar identical to the Paimon
option parser that will
+ // consume this value; accepting a superset lets invalid
serialized options reach scans.
+ return Optional.of(MemorySize.parse(value).getBytes());
+ } catch (IllegalArgumentException e) {
return Optional.empty();
}
}
private void initTable() {
Preconditions.checkState(params.containsKey("serialized_table"));
table = PaimonUtils.deserialize(params.get("serialized_table"));
+ String encodedSystemSource = params.get(PAIMON_OPTION_PREFIX +
DORIS_SERIALIZED_SYSTEM_SOURCE);
+ FileStoreTable systemSource = encodedSystemSource == null
+ ? null : PaimonUtils.deserialize(encodedSystemSource);
+ table = applyBackendManifestParallelism(table,
+ params.get(PAIMON_OPTION_PREFIX +
DORIS_MANIFEST_PARALLELISM_CAP),
+ Runtime.getRuntime().availableProcessors(), systemSource,
+ params.get(PAIMON_OPTION_PREFIX + DORIS_SYSTEM_TABLE_TYPE));
+ table = applyDefaultReadBatchSize(table, batchSize);
+ paimonAllFieldNames = PaimonUtils.getFieldNames(this.table.rowType());
+ if (LOG.isDebugEnabled()) {
+ LOG.debug("paimonAllFieldNames:{}", paimonAllFieldNames);
+ }
+ }
+
+ static Table applyDefaultReadBatchSize(Table table, int dorisBatchSize) {
+ validateSerializedReaderOptions(table);
+ if (hasReadBatchSize(table)) {
Review Comment:
[P2] Preserve an explicit batch on hidden fallback branches.
`validateSerializedReaderOptions` descends through `hiddenSystemSource`, but
`hasReadBatchSize` does not, so an `AuditLogTable` wrapping
`FallbackReadFileStoreTable(main-without-batch,
fallback-with-read.batch-size=1)` reaches the default-injection path. Paimon's
system-table `copy` then forwards the Doris batch to the fallback pair and
overwrites that explicit fallback value. Recurse and default per leaf (then
rebuild the wrapper), and cover both absent/explicit branch orders.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/commands/ExecuteCommand.java:
##########
@@ -72,35 +75,51 @@ public void run(ConnectContext ctx, StmtExecutor executor)
throws Exception {
StatementContext statementContext = ctx.getStatementContext();
statementContext.setPrepareStage(false);
statementContext.setIsInsert(false);
+ statementContext.resetMvccSnapshots();
PreparedStatementContext preparedStmtCtx =
ctx.getPreparedStementContext(stmtName);
if (null == preparedStmtCtx) {
throw new AnalysisException(
"prepare statement " + stmtName + " not found, maybe
expired");
}
PrepareCommand prepareCommand = preparedStmtCtx.command;
LogicalPlan logicalPlan = prepareCommand.getLogicalPlan();
- LogicalPlan relationRoot = logicalPlan;
+ List<LogicalPlan> relationRoots = new ArrayList<>();
if (logicalPlan instanceof InsertIntoTableCommand) {
- relationRoot = ((InsertIntoTableCommand)
logicalPlan).getLogicalQuery();
+ relationRoots.add(((InsertIntoTableCommand)
logicalPlan).getLogicalQuery());
} else if (logicalPlan instanceof InsertOverwriteTableCommand) {
- relationRoot = ((InsertOverwriteTableCommand)
logicalPlan).getLogicalQuery();
+ relationRoots.add(((InsertOverwriteTableCommand)
logicalPlan).getLogicalQuery());
} else if (logicalPlan instanceof UpdateCommand) {
- relationRoot = ((UpdateCommand) logicalPlan).getLogicalQuery();
- } else if (logicalPlan instanceof Command) {
- // Non-DML commands deliberately have no traversable children;
they cannot own a
- // relation scan tree whose resolved state needs resetting.
- relationRoot = null;
+ relationRoots.add(((UpdateCommand) logicalPlan).getLogicalQuery());
Review Comment:
[P1] Reset subqueries retained by UPDATE assignments too. This branch
exposes only `logicalQuery`, but `UpdateCommand` separately stores each SET
right-hand expression and injects it later; for `SET v = (SELECT ... FROM
p@options(...))`, the `SubqueryExpr` owns that relation outside both plan
children and expression children, so this reset walk never clears its resolved
parameters. A second EXECUTE can therefore reuse the first execution's pinned
selector after latest advances. Expose/traverse the assignment expressions
(including their subquery plans) and add a repeated prepared-UPDATE regression.
##########
fe/fe-core/src/main/java/org/apache/doris/datasource/paimon/PaimonReaderOptions.java:
##########
@@ -0,0 +1,315 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.datasource.paimon;
+
+import com.google.common.collect.ImmutableSet;
+import org.apache.paimon.CoreOptions;
+import org.apache.paimon.options.ConfigOption;
+import org.apache.paimon.options.MemorySize;
+import org.apache.paimon.options.Options;
+import org.apache.paimon.privilege.PrivilegedFileStoreTable;
+import org.apache.paimon.table.DelegatedFileStoreTable;
+import org.apache.paimon.table.FallbackReadFileStoreTable;
+import org.apache.paimon.table.FileStoreTable;
+import org.apache.paimon.table.Table;
+
+import java.util.Collections;
+import java.util.LinkedHashMap;
+import java.util.Locale;
+import java.util.Map;
+import java.util.OptionalInt;
+import java.util.Set;
+
+/** Validation shared by catalog-scoped and relation-scoped Paimon reader
tuning. */
+public final class PaimonReaderOptions {
+ public static final String TABLE_OPTION_PREFIX = "paimon.table-option.";
+ public static final int MIN_READ_BATCH_SIZE = 1;
+ public static final int MAX_READ_BATCH_SIZE = 65536;
+ // Keep catalog replay deterministic while bounding a single option's
JVM-wide thread impact.
+ public static final int MAX_MANIFEST_PARALLELISM = 256;
+ public static final long MIN_ASYNC_THRESHOLD_BYTES = 1024L * 1024L;
+ public static final long MAX_ASYNC_THRESHOLD_BYTES = 1024L * 1024L * 1024L;
+
+ // Keep this list to batch-read controls consumed by Doris' Paimon scan
path. Context selectors,
+ // streaming-source settings, storage layout, and write options are unsafe
after schema binding.
+ private static final Set<String> SUPPORTED_OPTIONS = ImmutableSet.of(
+ CoreOptions.READ_BATCH_SIZE.key(),
+ CoreOptions.FILE_READER_ASYNC_THRESHOLD.key(),
+ CoreOptions.FILE_INDEX_READ_ENABLED.key(),
+ CoreOptions.SOURCE_SPLIT_TARGET_SIZE.key(),
+ CoreOptions.SOURCE_SPLIT_OPEN_FILE_COST.key(),
+ CoreOptions.SCAN_MANIFEST_PARALLELISM.key(),
+ CoreOptions.SCAN_PLAN_SORT_PARTITION.key());
+
+ // These settings do not alter the selected snapshot or manifest
projection, so relation-local
+ // copies can reuse the memoized partition projection while still planning
splits from the copy.
+ private static final Set<String> METADATA_NEUTRAL_OPTIONS =
ImmutableSet.of(
+ CoreOptions.READ_BATCH_SIZE.key(),
+ CoreOptions.FILE_READER_ASYNC_THRESHOLD.key(),
+ CoreOptions.FILE_INDEX_READ_ENABLED.key(),
+ CoreOptions.SOURCE_SPLIT_TARGET_SIZE.key(),
+ CoreOptions.SOURCE_SPLIT_OPEN_FILE_COST.key());
+
+ private PaimonReaderOptions() {
+ }
+
+ public static Set<String> supportedOptions() {
+ return SUPPORTED_OPTIONS;
+ }
+
+ public static Set<String> metadataNeutralOptions() {
+ return METADATA_NEUTRAL_OPTIONS;
+ }
+
+ public static void validate(String key, String value) {
+ if (!SUPPORTED_OPTIONS.contains(key)) {
+ throw new IllegalArgumentException("Unsupported Paimon dynamic
reader option '" + key
+ + "'. Supported options are " + SUPPORTED_OPTIONS);
+ }
+
+ if (CoreOptions.READ_BATCH_SIZE.key().equals(key)) {
+ int batchSize = parse(key, value, CoreOptions.READ_BATCH_SIZE);
+ // A zero batch can make Paimon's vectorized reader report success
without
+ // advancing input; the upper bound also prevents one relation
from over-allocating.
+ requireRange(key, batchSize, MIN_READ_BATCH_SIZE,
MAX_READ_BATCH_SIZE);
+ } else if (CoreOptions.FILE_READER_ASYNC_THRESHOLD.key().equals(key)) {
+ MemorySize threshold = parse(key, value,
CoreOptions.FILE_READER_ASYNC_THRESHOLD);
+ // Bound the trigger on both sides so a query cannot fan out tiny
async reads or
+ // silently disable asynchronous reading with an effectively
infinite threshold.
+ requireRange(key, threshold.getBytes(),
+ MIN_ASYNC_THRESHOLD_BYTES, MAX_ASYNC_THRESHOLD_BYTES);
+ } else if (CoreOptions.SOURCE_SPLIT_TARGET_SIZE.key().equals(key)) {
+ MemorySize targetSize = parse(key, value,
CoreOptions.SOURCE_SPLIT_TARGET_SIZE);
+ // A split-size option represents byte capacity; non-positive
values silently defeat
+ // Paimon's bin packing and turn every data file into a separate
Doris scan range.
+ requireRange(key, targetSize.getBytes(), 1, Long.MAX_VALUE);
+ } else if (CoreOptions.SOURCE_SPLIT_OPEN_FILE_COST.key().equals(key)) {
+ parse(key, value, CoreOptions.SOURCE_SPLIT_OPEN_FILE_COST);
+ } else if (CoreOptions.SCAN_MANIFEST_PARALLELISM.key().equals(key)) {
+ validateManifestParallelism(value);
+ } else if (CoreOptions.FILE_INDEX_READ_ENABLED.key().equals(key)) {
+ parse(key, value, CoreOptions.FILE_INDEX_READ_ENABLED);
+ } else {
+ parse(key, value, CoreOptions.SCAN_PLAN_SORT_PARTITION);
+ }
+ }
+
+ public static void validateCatalogProperties(Map<String, String>
properties) {
+ properties.forEach((key, value) -> {
+ if (!key.toLowerCase(Locale.ROOT).startsWith(TABLE_OPTION_PREFIX))
{
+ return;
+ }
+ String optionKey = key.substring(TABLE_OPTION_PREFIX.length());
+ if (optionKey.isEmpty()) {
+ throw new IllegalArgumentException(
+ "Paimon table option name must not be empty after
prefix " + TABLE_OPTION_PREFIX);
+ }
+ validate(optionKey, value);
+ });
+ }
+
+ public static Map<String, String> compatibleCatalogOptions(Map<String,
String> properties) {
+ Map<String, String> compatibleOptions = new LinkedHashMap<>();
+ properties.forEach((key, value) -> {
+ if (!key.toLowerCase(Locale.ROOT).startsWith(TABLE_OPTION_PREFIX))
{
+ return;
+ }
+ String optionKey = key.substring(TABLE_OPTION_PREFIX.length());
+ try {
+ validate(optionKey, value);
+ compatibleOptions.put(optionKey, value);
+ } catch (IllegalArgumentException ignored) {
+ // Images written before the reader-only allowlist may contain
arbitrary Paimon
+ // options. Keep the catalog loadable, but never apply an
unsafe legacy option.
+ }
+ });
+ return Collections.unmodifiableMap(compatibleOptions);
+ }
+
+ public static void validateReaderOptions(Map<String, String> options) {
+ SUPPORTED_OPTIONS.stream()
+ .filter(options::containsKey)
+ .forEach(key -> validate(key, options.get(key)));
+ }
+
+ public static void validateEffectiveTableOptions(Map<String, String>
options) {
+ validateReaderOptions(options);
+ validateIfPresentForRuntime(options,
CoreOptions.SCAN_MANIFEST_PARALLELISM.key());
+ }
+
+ public static Table runtimeSafeTable(Table table) {
+ return runtimeSafeTable(table,
Runtime.getRuntime().availableProcessors());
+ }
+
+ public static Table runtimeSafeTable(Table table, Map<String, String>
copyOptions) {
+ // Relation options intentionally apply to both fallback branches.
Runtime safety is a
+ // different operation and must normalize each resulting planner
without flattening them.
+ return runtimeSafeTable(table.copy(copyOptions));
+ }
+
+ static Table runtimeSafeTable(Table table, int localCapacity) {
+ if (localCapacity < 1) {
+ throw new IllegalArgumentException("Paimon planning capacity must
be positive.");
+ }
+ int safeBound = Math.min(localCapacity, MAX_MANIFEST_PARALLELISM);
+ return normalizeManifestParallelism(table, safeBound, localCapacity >
safeBound);
+ }
+
+ public static OptionalInt backendManifestParallelismCap(Table table) {
+ // The transport value is the execution ceiling, not the smallest
branch preference;
+ // the BE preserves each branch's lower value while independently
capping larger siblings.
+ return OptionalInt.of(Math.min(
+ Runtime.getRuntime().availableProcessors(),
MAX_MANIFEST_PARALLELISM));
+ }
+
+ private static Table normalizeManifestParallelism(
+ Table table, int safeBound, boolean materializeAbsent) {
+ if (table instanceof FallbackReadFileStoreTable) {
+ FallbackReadFileStoreTable pair = (FallbackReadFileStoreTable)
table;
+ FileStoreTable main = normalizeManifestParallelism(
+ pair.wrapped(), safeBound, materializeAbsent);
+ FileStoreTable fallback = normalizeManifestParallelism(
+ pair.fallback(), safeBound, materializeAbsent);
+ return main == pair.wrapped() && fallback == pair.fallback()
+ ? table : new FallbackReadFileStoreTable(main, fallback);
+ }
+
+ if (table instanceof DelegatedFileStoreTable) {
+ if (!(table instanceof PrivilegedFileStoreTable)) {
+ throw new IllegalArgumentException("Unsupported Paimon
planning table delegate: "
+ + table.getClass().getName());
+ }
+ FileStoreTable wrapped = ((DelegatedFileStoreTable)
table).wrapped();
+ FileStoreTable normalized = normalizeManifestParallelism(
+ wrapped, safeBound, materializeAbsent);
+ if (normalized == wrapped) {
+ return table;
+ }
+ // A delegate copy broadcasts one value to every fallback branch.
Check authorization
+ // before peeling the privilege-only layer so branch-local limits
remain independent.
+ ((FileStoreTable) table).newScan();
+ return normalized;
+ }
+
+ if (!(table instanceof FileStoreTable)) {
+ // System-table wrappers hide the already-normalized source.
Treating the wrapper as a
+ // planning leaf would broadcast one cap through its private
fallback tree on copy().
+ return table;
+ }
+
+ String key = CoreOptions.SCAN_MANIFEST_PARALLELISM.key();
+ String configured = table.options().get(key);
+ if (configured == null && !materializeAbsent) {
+ return table;
+ }
+ if (configured != null) {
+ validateManifestParallelism(configured);
Review Comment:
[P2] Cap external physical manifest values before applying the catalog-write
ceiling. This is the runtime normalization path, but
`validateManifestParallelism` rejects `scan.manifest.parallelism=300` before
the code can copy it down to `safeBound`, so a table configured outside Doris
that is safe to cap becomes unqueryable on a new FE. The BE rolling-upgrade
backstop already treats the same positive 300 as cappable. Keep the 1..256 rule
for Doris CREATE/ALTER inputs, but parse any positive runtime value and cap it
per leaf before validating the effective result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
