Jackie-Jiang commented on code in PR #20121:
URL: https://github.com/apache/pinot/pull/20121#discussion_r4570832096
##########
pinot-connectors/pinot-flink-connector/README.md:
##########
@@ -92,6 +92,27 @@ execEnv.execute();
For more examples, please see
`src/main/java/org/apache/pinot/connector/flink/FlinkQuickStart.java`
+## Authentication
+If the controller has authentication enabled, pass the auth headers to the
`PinotAdminClient` and the token to
+`PinotSink`. The sink sends the token as the `Authorization` header when
uploading segments.
+```java
+String authToken = "Basic <base64(user:password)>";
+try (PinotAdminClient client =
+ new PinotAdminClient(controllerAddress, properties,
Map.of("Authorization", authToken))) {
+ Schema schema = client.getSchemaClient().getSchemaObject("starbucksStores");
+ TableConfig tableConfig =
+ client.getTableClient().getTableConfigObjectForType("starbucksStores",
TableType.OFFLINE);
+ srcDs.sinkTo(new PinotSink<>(new PinotRowRecordConverter(TEST_TYPE_INFO),
tableConfig, schema,
Review Comment:
Minor: `PinotRowRecordConverter` does not exist in this revision. Use
`FlinkRowGenericRowConverter(TEST_TYPE_INFO)`, matching the new integration
test, so this example compiles.
##########
pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/PinotSink.java:
##########
@@ -137,6 +172,25 @@ static TableConfig prepareTableConfigForSink(TableConfig
tableConfig, String con
return tableConfig;
}
+ /// Builds an [AuthProvider] from the `authToken` in the table's batch
config, or returns null if none is set.
+ ///
+ /// Prefer passing the token to [#PinotSink(PinotGenericRowConverter,
TableConfig, Schema, String, String)]. Never
+ /// store it in the table config saved on the controller: that config is
kept in ZooKeeper and returned by the table
+ /// APIs.
+ @Nullable
+ static AuthProvider createAuthProvider(TableConfig tableConfig) {
+ IngestionConfig ingestionConfig = tableConfig.getIngestionConfig();
+ if (ingestionConfig == null || ingestionConfig.getBatchIngestionConfig()
== null) {
+ return null;
+ }
+ List<Map<String, String>> batchConfigMaps =
ingestionConfig.getBatchIngestionConfig().getBatchConfigMaps();
+ if (batchConfigMaps == null || batchConfigMaps.isEmpty()) {
+ return null;
+ }
+ String authToken =
batchConfigMaps.get(0).get(BatchConfigProperties.AUTH_TOKEN);
+ return StringUtils.isBlank(authToken) ? null :
AuthProviderUtils.makeAuthProvider(authToken);
Review Comment:
Optional simplification: `AuthProviderUtils.makeAuthProvider(String)`
already handles null/blank tokens by returning `NullAuthProvider`. Resolve the
optional token and delegate all cases to that utility, keeping the writer's
provider nonnullable. Tests can assert empty request headers for absent
credentials instead of a null provider.
##########
pinot-connectors/pinot-flink-connector/src/main/java/org/apache/pinot/connector/flink/sink/PinotSink.java:
##########
@@ -83,6 +87,18 @@ public PinotSink(PinotGenericRowConverter<T>
recordConverter, TableConfig tableC
DEFAULT_SEGMENT_FLUSH_MAX_NUM_RECORDS, DEFAULT_EXECUTOR_POOL_SIZE);
}
+ /// Same as [#PinotSink(PinotGenericRowConverter, TableConfig, Schema,
String)], and also adds `authToken` to the
+ /// batch config so segment uploads authenticate with the controller. A
non-blank `authToken` overrides any token
+ /// already in the batch config.
+ ///
+ /// The given `tableConfig` is not modified: the sink works on a deep copy,
so the token never ends up in the
+ /// caller's table config.
+ public PinotSink(PinotGenericRowConverter<T> recordConverter, TableConfig
tableConfig, Schema schema,
+ String controllerBaseUrl, @Nullable String authToken) {
+ this(recordConverter,
prepareTableConfigForSink(copyTableConfig(tableConfig), controllerBaseUrl,
authToken),
Review Comment:
Optional simplification: keep the explicit credential in a nullable `String
_authToken` field on `PinotSink`, and create the provider inside the writer as
today. This separates credentials from table configuration and removes the
auth-specific `prepareTableConfigForSink` overload. Preserve nonblank
explicit-token precedence and fall back to the batch-config token for
null/blank arguments. Passing the token through the full constructor would also
allow custom flush limits, executor size, and segment naming. Keep the config
copy if retaining the promise that the caller's entire config is unchanged:
preparing upload defaults still mutates it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]