hansva commented on code in PR #9389:
URL: https://github.com/apache/hop/pull/9389#discussion_r4537798675
##########
plugins/transforms/languagemodelchat/src/main/java/org/apache/hop/pipeline/transforms/languagemodelchat/internals/LanguageModelFacade.java:
##########
@@ -93,11 +105,41 @@ private String resolve(String param) {
return variables.resolve(trimToNull(param));
}
+ /**
+ * The API key to send, or null to send none.
+ *
+ * <p>A key field defaults to the provider's own variable, such as {@code
${OPENAI_API_KEY}}. That
+ * variable is read from the Hop variables, and from the environment when no
Hop variable defines
+ * it, since the operating system environment is only imported into Hop
variables on request.
+ * Pipelines written before the default became a variable hold the bare
name, which reads the same
+ * way. Any other variable that nothing defines sends no key, rather than
its own text.
+ *
+ * @param field the key field as stored in the transform
+ * @param variableName the provider's environment variable, such as {@code
OPENAI_API_KEY}
+ */
+ String resolveApiKey(String field, String variableName) {
+ String value = trimToNull(field);
+ if (value == null) {
+ return null;
+ }
+ String ownVariable = "${" + variableName + "}";
+ if (value.equals(variableName)) {
+ value = ownVariable;
+ }
+ String resolved = trimToNull(variables.resolve(value));
+ if (resolved != null && !resolved.contains("${")) {
+ return resolved;
+ }
+ if (value.equals(ownVariable)) {
Review Comment:
This falls back to the real `OPENAI_API_KEY` environment variable whatever
the base URL is. Before, the default was the literal text `OPENAI_API_KEY`,
which is what got sent. Now a user who points `openAiBaseUrl` at an
OpenAI-compatible third party or local server (Groq, vLLM, LM Studio...) and
leaves the key at its default sends their real `sk-...` key to that host. The
same happens through a `CustomOpenAiProvider` (`requiresApiKey=false`) where
`AiChatFactory` leaves the key at `${OPENAI_API_KEY}`, and for the Mistral and
Anthropic base URLs. Could the environment fallback only apply when the base
URL is empty or the provider's own endpoint?
##########
plugins/transforms/languagemodelchat/src/main/java/org/apache/hop/pipeline/transforms/languagemodelchat/internals/LanguageModelFacade.java:
##########
@@ -93,11 +105,41 @@ private String resolve(String param) {
return variables.resolve(trimToNull(param));
}
+ /**
+ * The API key to send, or null to send none.
+ *
+ * <p>A key field defaults to the provider's own variable, such as {@code
${OPENAI_API_KEY}}. That
+ * variable is read from the Hop variables, and from the environment when no
Hop variable defines
+ * it, since the operating system environment is only imported into Hop
variables on request.
+ * Pipelines written before the default became a variable hold the bare
name, which reads the same
+ * way. Any other variable that nothing defines sends no key, rather than
its own text.
+ *
+ * @param field the key field as stored in the transform
+ * @param variableName the provider's environment variable, such as {@code
OPENAI_API_KEY}
+ */
+ String resolveApiKey(String field, String variableName) {
+ String value = trimToNull(field);
+ if (value == null) {
+ return null;
+ }
+ String ownVariable = "${" + variableName + "}";
+ if (value.equals(variableName)) {
+ value = ownVariable;
+ }
+ String resolved = trimToNull(variables.resolve(value));
+ if (resolved != null && !resolved.contains("${")) {
Review Comment:
Two edge cases in this check: only `${` counts as unresolved, so an unset
`%%MY_KEY%%` or `$[..]` is sent as the key itself (contradicting 'any other
variable that nothing defines sends no key');
`StringUtil.containsVariableToken()` covers all delimiters. And a real key
whose value contains `${` (e.g. one that was already resolved and gets resolved
again here) is dropped silently: the request goes out without a key and fails
with a 401, with nothing in the log. A log line when a key is discarded would
help.
##########
plugins/transforms/languagemodelchat/src/test/java/org/apache/hop/pipeline/transforms/languagemodelchat/LanguageModelChatMetaSecretsTest.java:
##########
@@ -100,10 +103,57 @@ void plainTextKeysFromOlderPipelinesStillLoad() throws
Exception {
@Test
void openAiApiKeyDefaultHasNoStrayBrace() {
LanguageModelChatMeta meta = new LanguageModelChatMeta();
- assertEquals("OPENAI_API_KEY", meta.getOpenAiApiKey());
+ assertEquals("${OPENAI_API_KEY}", meta.getOpenAiApiKey());
meta.setDefault();
- assertEquals("OPENAI_API_KEY", meta.getOpenAiApiKey());
+ assertEquals("${OPENAI_API_KEY}", meta.getOpenAiApiKey());
+ }
+
+ /**
+ * Every key field defaults to a variable: a plain value would be sent as
the key, and is reported
+ * as a hardcoded secret by lint rule SEC-002 in every pipeline that uses
another provider, see
+ * issue #9385.
+ */
+ @Test
+ void everyKeyFieldDefaultsToAVariable() throws Exception {
+ LanguageModelChatMeta constructed = new LanguageModelChatMeta();
+ LanguageModelChatMeta defaulted = new LanguageModelChatMeta();
+ defaulted.setDefault();
+
+ int keyFields = 0;
+ for (Field field : LanguageModelChatMeta.class.getDeclaredFields()) {
+ HopMetadataProperty property =
field.getAnnotation(HopMetadataProperty.class);
+ if (property == null || !property.password()) {
+ continue;
+ }
+ keyFields++;
+ field.setAccessible(true);
+ for (LanguageModelChatMeta meta : List.of(constructed, defaulted)) {
+ String value = (String) field.get(meta);
+ assertTrue(
+ value.matches("\\$\\{[A-Z_]+}"), field.getName() + " defaults to
'" + value + "'");
+ }
+ }
Review Comment:
`assertEquals(4, keyFields)` plus the assumption that every password field
is a non-null String means a new `password=true` field (e.g. one that defaults
to null) fails this test with an NPE or a count mismatch, unrelated to what it
checks.
##########
integration-tests/languagemodelchat-api-keys/hop-config.json:
##########
@@ -0,0 +1,290 @@
+{
+ "variables": [
+ {
+ "name": "HOP_LENIENT_STRING_TO_NUMBER_CONVERSION",
+ "value": "N",
+ "description": "System wide flag to allow lenient string to number
conversion for backward compatibility. If this setting is set to \"Y\", an
string starting with digits will be converted successfully into a number.
(example: 192.168.1.1 will be converted into 192 or 192.168 or 192168 depending
on the decimal and grouping symbol). The default (N) will be to throw an error
if non-numeric symbols are found in the string."
+ },
+ {
+ "name": "HOP_COMPATIBILITY_DB_IGNORE_TIMEZONE",
+ "value": "N",
+ "description": "System wide flag to ignore timezone while writing
date/timestamp value to the database."
+ },
+ {
+ "name": "HOP_LOG_SIZE_LIMIT",
+ "value": "0",
+ "description": "The log size limit for all pipelines and workflows that
don't have the \"log size limit\" property set in their respective properties."
+ },
+ {
+ "name": "HOP_EMPTY_STRING_DIFFERS_FROM_NULL",
+ "value": "N",
+ "description": "NULL vs Empty String. If this setting is set to Y, an
empty string and null are different. Otherwise they are not."
+ },
+ {
+ "name": "HOP_MAX_LOG_SIZE_IN_LINES",
+ "value": "0",
+ "description": "The maximum number of log lines that are kept internally
by Hop. Set to 0 to keep all rows (default)"
+ },
+ {
+ "name": "HOP_MAX_LOG_TIMEOUT_IN_MINUTES",
+ "value": "1440",
+ "description": "The maximum age (in minutes) of a log line while being
kept internally by Hop. Set to 0 to keep all rows indefinitely (default)"
+ },
+ {
+ "name": "HOP_MAX_WORKFLOW_TRACKER_SIZE",
+ "value": "5000",
+ "description": "The maximum number of workflow trackers kept in memory"
+ },
+ {
+ "name": "HOP_MAX_ACTIONS_LOGGED",
+ "value": "5000",
+ "description": "The maximum number of action results kept in memory for
logging purposes."
+ },
+ {
+ "name": "HOP_MAX_LOGGING_REGISTRY_SIZE",
+ "value": "10000",
+ "description": "The maximum number of logging registry entries kept in
memory for logging purposes."
+ },
+ {
+ "name": "HOP_LOG_TAB_REFRESH_DELAY",
+ "value": "1000",
+ "description": "The hop log tab refresh delay."
+ },
+ {
+ "name": "HOP_LOG_TAB_REFRESH_PERIOD",
+ "value": "1000",
+ "description": "The hop log tab refresh period."
+ },
+ {
+ "name": "HOP_PLUGIN_CLASSES",
+ "value": null,
+ "description": "A comma delimited list of classes to scan for plugin
annotations"
+ },
+ {
+ "name": "HOP_PLUGIN_PACKAGES",
+ "value": null,
+ "description": "A comma delimited list of packages to scan for plugin
annotations (warning: slow!!)"
+ },
+ {
+ "name": "HOP_TRANSFORM_PERFORMANCE_SNAPSHOT_LIMIT",
+ "value": "0",
+ "description": "The maximum number of transform performance snapshots to
keep in memory. Set to 0 to keep all snapshots indefinitely (default)"
+ },
+ {
+ "name": "HOP_ROWSET_GET_TIMEOUT",
+ "value": "50",
+ "description": "The name of the variable that optionally contains an
alternative rowset get timeout (in ms). This only makes a difference for
extremely short lived pipelines."
+ },
+ {
+ "name": "HOP_ROWSET_PUT_TIMEOUT",
+ "value": "50",
+ "description": "The name of the variable that optionally contains an
alternative rowset put timeout (in ms). This only makes a difference for
extremely short lived pipelines."
+ },
+ {
+ "name": "HOP_CORE_TRANSFORMS_FILE",
+ "value": null,
+ "description": "The name of the project variable that will contain the
alternative location of the hop-transforms.xml file. You can use this to
customize the list of available internal transforms outside of the codebase."
+ },
+ {
+ "name": "HOP_CORE_WORKFLOW_ACTIONS_FILE",
+ "value": null,
+ "description": "The name of the project variable that will contain the
alternative location of the hop-workflow-actions.xml file."
+ },
+ {
+ "name": "HOP_SERVER_OBJECT_TIMEOUT_MINUTES",
+ "value": "1440",
+ "description": "This project variable will set a time-out after which
waiting, completed or stopped pipelines and workflows will be automatically
cleaned up. The default value is 1440 (one day)."
+ },
+ {
+ "name": "HOP_PIPELINE_PAN_JVM_EXIT_CODE",
+ "value": null,
+ "description": "Set this variable to an integer that will be returned as
the Pan JVM exit code."
+ },
+ {
+ "name": "HOP_DISABLE_CONSOLE_LOGGING",
+ "value": "N",
+ "description": "Set this variable to Y to disable standard Hop logging
to the console. (stdout)"
+ },
+ {
+ "name": "HOP_REDIRECT_STDERR",
+ "value": "N",
+ "description": "Set this variable to Y to redirect stderr to Hop
logging."
+ },
+ {
+ "name": "HOP_REDIRECT_STDOUT",
+ "value": "N",
+ "description": "Set this variable to Y to redirect stdout to Hop
logging."
+ },
+ {
+ "name": "HOP_DEFAULT_NUMBER_FORMAT",
+ "value": null,
+ "description": "The name of the variable containing an alternative
default number format"
+ },
+ {
+ "name": "HOP_DEFAULT_BIGNUMBER_FORMAT",
+ "value": null,
+ "description": "The name of the variable containing an alternative
default bignumber format"
+ },
+ {
+ "name": "HOP_DEFAULT_INTEGER_FORMAT",
+ "value": null,
+ "description": "The name of the variable containing an alternative
default integer format"
+ },
+ {
+ "name": "HOP_DEFAULT_DATE_FORMAT",
+ "value": null,
+ "description": "The name of the variable containing an alternative
default date format"
+ },
+ {
+ "name": "HOP_DEFAULT_TIMESTAMP_FORMAT",
+ "value": null,
+ "description": "The name of the variable containing an alternative
default timestamp format"
+ },
+ {
+ "name": "HOP_DEFAULT_SERVLET_ENCODING",
+ "value": null,
+ "description": "Defines the default encoding for servlets, leave it
empty to use Java default encoding"
+ },
+ {
+ "name": "HOP_FAIL_ON_LOGGING_ERROR",
+ "value": "N",
+ "description": "Set this variable to Y when you want the
workflow/pipeline fail with an error when the related logging process (e.g. to
a database) fails."
+ },
+ {
+ "name": "HOP_AGGREGATION_MIN_NULL_IS_VALUED",
+ "value": "N",
+ "description": "Set this variable to Y to set the minimum to NULL if
NULL is within an aggregate. Otherwise by default NULL is ignored by the MIN
aggregate and MIN is set to the minimum value that is not NULL. See also the
variable HOP_AGGREGATION_ALL_NULLS_ARE_ZERO."
+ },
+ {
+ "name": "HOP_AGGREGATION_ALL_NULLS_ARE_ZERO",
+ "value": "N",
+ "description": "Set this variable to Y to return 0 when all values
within an aggregate are NULL. Otherwise by default a NULL is returned when all
values are NULL."
+ },
+ {
+ "name": "HOP_COMPATIBILITY_TEXT_FILE_OUTPUT_APPEND_NO_HEADER",
+ "value": "N",
+ "description": "Set this variable to Y for backward compatibility for
the Text File Output transform. Setting this to Ywill add no header row at all
when the append option is enabled, regardless if the file is existing or not."
+ },
+ {
+ "name": "HOP_PASSWORD_ENCODER_PLUGIN",
+ "value": "Hop",
+ "description": "Specifies the password encoder plugin to use by ID (Hop
is the default)."
+ },
+ {
+ "name": "HOP_SYSTEM_HOSTNAME",
+ "value": null,
+ "description": "You can use this variable to speed up hostname lookup.
Hostname lookup is performed by Hop so that it is capable of logging the server
on which a workflow or pipeline is executed."
+ },
+ {
+ "name": "HOP_SERVER_JETTY_ACCEPTORS",
+ "value": null,
+ "description": "A variable to configure jetty option: acceptors for
Carte"
+ },
+ {
+ "name": "HOP_SERVER_JETTY_ACCEPT_QUEUE_SIZE",
+ "value": null,
+ "description": "A variable to configure jetty option: acceptQueueSize
for Carte"
+ },
+ {
+ "name": "HOP_SERVER_JETTY_RES_MAX_IDLE_TIME",
+ "value": null,
+ "description": "A variable to configure jetty option:
lowResourcesMaxIdleTime for Carte"
+ },
+ {
+ "name":
"HOP_COMPATIBILITY_MERGE_ROWS_USE_REFERENCE_STREAM_WHEN_IDENTICAL",
+ "value": "N",
+ "description": "Set this variable to Y for backward compatibility for
the Merge Rows (diff) transform. Setting this to Y will use the data from the
reference stream (instead of the comparison stream) in case the compared rows
are identical."
+ },
+ {
+ "name": "HOP_SPLIT_FIELDS_REMOVE_ENCLOSURE",
+ "value": "false",
+ "description": "Set this variable to false to preserve enclosure symbol
after splitting the string in the Split fields transform. Changing it to true
will remove first and last enclosure symbol from the resulting string chunks."
+ },
+ {
+ "name": "HOP_ALLOW_EMPTY_FIELD_NAMES_AND_TYPES",
+ "value": "false",
+ "description": "Set this variable to TRUE to allow your pipeline to pass
'null' fields and/or empty types."
+ },
+ {
+ "name": "HOP_GLOBAL_LOG_VARIABLES_CLEAR_ON_EXPORT",
+ "value": "false",
+ "description": "Set this variable to false to preserve global log
variables defined in pipeline / workflow Properties -> Log panel. Changing it
to true will clear it when export pipeline / workflow."
+ },
+ {
+ "name": "HOP_FILE_OUTPUT_MAX_STREAM_COUNT",
+ "value": "1024",
+ "description": "This project variable is used by the Text File Output
transform. It defines the max number of simultaneously open files within the
transform. The transform will close/reopen files as necessary to insure the max
is not exceeded"
+ },
+ {
+ "name": "HOP_FILE_OUTPUT_MAX_STREAM_LIFE",
+ "value": "0",
+ "description": "This project variable is used by the Text File Output
transform. It defines the max number of milliseconds between flushes of files
opened by the transform."
+ },
+ {
+ "name": "HOP_USE_NATIVE_FILE_DIALOG",
+ "value": "N",
+ "description": "Set this value to Y if you want to use the system file
open/save dialog when browsing files"
+ },
+ {
+ "name": "HOP_AUTO_CREATE_CONFIG",
+ "value": "Y",
+ "description": "Set this value to N if you don't want to automatically
create a hop configuration file (hop-config.json) when it's missing"
+ }
+ ],
+ "LocaleDefault": "en_BE",
+ "guiProperties": {
Review Comment:
This looks like a personal hop-config: `LocaleDefault` en_BE, DarkMode Y,
macOS `.AppleSystemUIFont` fonts (and no trailing newline). The Belgian locale
changes number/date formatting compared with the other IT projects; copying the
config of an existing IT project would be safer.
##########
plugins/transforms/languagemodelchat/src/test/java/org/apache/hop/pipeline/transforms/languagemodelchat/LanguageModelChatMetaSecretsTest.java:
##########
@@ -100,10 +103,57 @@ void plainTextKeysFromOlderPipelinesStillLoad() throws
Exception {
@Test
void openAiApiKeyDefaultHasNoStrayBrace() {
Review Comment:
The name says 'has no stray brace', but the test now asserts the default
*is* `${OPENAI_API_KEY}` with braces, and it duplicates
`keyDefaultsNameTheProvidersVariable`. A failing 'HasNoStrayBrace' could tempt
someone to revert to the bare name and bring back the SEC-002 lint finding
(#9385). Renaming or dropping it would avoid that.
##########
plugins/transforms/languagemodelchat/src/main/java/org/apache/hop/pipeline/transforms/languagemodelchat/internals/LanguageModelFacade.java:
##########
@@ -93,11 +105,41 @@ private String resolve(String param) {
return variables.resolve(trimToNull(param));
}
+ /**
+ * The API key to send, or null to send none.
+ *
+ * <p>A key field defaults to the provider's own variable, such as {@code
${OPENAI_API_KEY}}. That
+ * variable is read from the Hop variables, and from the environment when no
Hop variable defines
+ * it, since the operating system environment is only imported into Hop
variables on request.
+ * Pipelines written before the default became a variable hold the bare
name, which reads the same
+ * way. Any other variable that nothing defines sends no key, rather than
its own text.
+ *
+ * @param field the key field as stored in the transform
+ * @param variableName the provider's environment variable, such as {@code
OPENAI_API_KEY}
+ */
+ String resolveApiKey(String field, String variableName) {
+ String value = trimToNull(field);
+ if (value == null) {
+ return null;
+ }
+ String ownVariable = "${" + variableName + "}";
+ if (value.equals(variableName)) {
+ value = ownVariable;
+ }
+ String resolved = trimToNull(variables.resolve(value));
+ if (resolved != null && !resolved.contains("${")) {
+ return resolved;
Review Comment:
Small one: `"${" + variableName + "}"` is built here while the meta's
`DEFAULT_*` constants build the same string separately, so each
variable/default pair has to stay in sync by hand. Passing one provider
descriptor, or using `StringUtil.isVariable`/`getVariableName` on the default,
would avoid a provider silently losing its fallback.
##########
plugins/transforms/languagemodelchat/src/main/java/org/apache/hop/pipeline/transforms/languagemodelchat/internals/LanguageModelFacade.java:
##########
@@ -93,11 +105,41 @@ private String resolve(String param) {
return variables.resolve(trimToNull(param));
}
+ /**
+ * The API key to send, or null to send none.
+ *
+ * <p>A key field defaults to the provider's own variable, such as {@code
${OPENAI_API_KEY}}. That
+ * variable is read from the Hop variables, and from the environment when no
Hop variable defines
+ * it, since the operating system environment is only imported into Hop
variables on request.
+ * Pipelines written before the default became a variable hold the bare
name, which reads the same
+ * way. Any other variable that nothing defines sends no key, rather than
its own text.
+ *
+ * @param field the key field as stored in the transform
+ * @param variableName the provider's environment variable, such as {@code
OPENAI_API_KEY}
+ */
+ String resolveApiKey(String field, String variableName) {
+ String value = trimToNull(field);
+ if (value == null) {
+ return null;
+ }
+ String ownVariable = "${" + variableName + "}";
+ if (value.equals(variableName)) {
Review Comment:
The environment fallback is special-cased per provider here, so
`${OPENAI_API_KEY}` reads the OS environment but `${MY_OPENAI_KEY}` (or the
samples' `${HF_TOKEN}`) never does; the same syntax behaves differently
depending on the name, and other transforms holding secrets would need their
own copy. Something to consider for a general mechanism later.
##########
plugins/transforms/languagemodelchat/src/test/java/org/apache/hop/pipeline/transforms/languagemodelchat/internals/LanguageModelFacadeApiKeyTest.java:
##########
@@ -0,0 +1,262 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.pipeline.transforms.languagemodelchat.internals;
+
+import static java.nio.charset.StandardCharsets.UTF_8;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+import com.sun.net.httpserver.HttpServer;
+import dev.langchain4j.data.message.UserMessage;
+import java.io.IOException;
+import java.io.InputStream;
+import java.net.InetSocketAddress;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.concurrent.atomic.AtomicReference;
+import org.apache.hop.core.variables.IVariables;
+import org.apache.hop.core.variables.Variables;
+import org.apache.hop.junit.rules.RestoreHopEngineEnvironmentExtension;
+import
org.apache.hop.pipeline.transforms.languagemodelchat.LanguageModelChatMeta;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.extension.RegisterExtension;
+
+/**
+ * Which API key is sent, see issue #9385. The key fields default to the
provider's variable, such
+ * as {@code ${OPENAI_API_KEY}}, which is read from the Hop variables and then
from the environment.
+ * The environment is handed to the facade, so no real environment variable is
needed.
+ */
+class LanguageModelFacadeApiKeyTest {
+
+ @RegisterExtension
+ static RestoreHopEngineEnvironmentExtension env = new
RestoreHopEngineEnvironmentExtension();
+
+ private final IVariables variables = new Variables();
+ private final Map<String, String> environment = new HashMap<>();
+ private final AtomicReference<String> receivedKey = new AtomicReference<>();
+
+ private HttpServer server;
+ private String baseUrl;
+
+ @BeforeEach
+ void startServer() throws IOException {
+ server = HttpServer.create(new InetSocketAddress("localhost", 0), 0);
Review Comment:
Binding to `localhost` and connecting to `localhost` can end up on different
address families (server on 127.0.0.1, client trying ::1 first), which makes
these tests flaky on some hosts. Using `127.0.0.1` on both sides avoids it (the
compose file works around the same IPv6 issue).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]