This is an automated email from the ASF dual-hosted git repository.
mattcasters pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/hop.git
The following commit(s) were added to refs/heads/main by this push:
new b458586968 additional fields for git input transform. fixes #8347
(#8348)
b458586968 is described below
commit b458586968d6786bd37ad4a3293329e0c9eee8bd
Author: Bart Maertens <[email protected]>
AuthorDate: Sun Sep 13 16:07:21 2026 +0200
additional fields for git input transform. fixes #8347 (#8348)
---
.../ROOT/pages/pipeline/transforms/gitinput.adoc | 64 ++++--
.../0112-git-input-github-pull-requests.hpl | 4 +-
.../git/pipeline/transforms/gitinput/GitInput.java | 2 +-
.../pipeline/transforms/gitinput/GitInputMeta.java | 23 +-
.../hop/git/provider/BitbucketResourceClient.java | 97 ++++----
.../hop/git/provider/GitHubResourceClient.java | 95 ++++----
.../apache/hop/git/provider/GitInputFields.java | 256 ++++++++++++++++-----
.../hop/git/provider/GitIssueActivityMapper.java | 73 +++---
.../org/apache/hop/git/provider/GitJsonLists.java | 81 +++++++
.../hop/git/provider/GitLabResourceClient.java | 83 +++----
.../apache/hop/git/provider/GitResourceRecord.java | 137 +++++------
.../hop/git/provider/GiteaResourceClient.java | 86 +++----
.../hop/git/provider/LocalGitResourceClient.java | 104 +++++----
.../transforms/git-input-github-pull-requests.hpl | 26 ++-
.../transforms/gitinput/GitInputMetaTest.java | 66 +++++-
.../pipeline/transforms/gitinput/GitInputTest.java | 4 +-
.../BitbucketResourceClientWireMockTest.java | 9 +-
.../hop/git/provider/GitHubResourceClientTest.java | 11 +-
.../provider/GitHubResourceClientWireMockTest.java | 86 ++++++-
.../apache/hop/git/provider/GitJsonListsTest.java | 82 +++++++
.../provider/GitLabResourceClientWireMockTest.java | 10 +-
.../provider/GiteaResourceClientWireMockTest.java | 4 +-
.../git/provider/LocalGitResourceClientTest.java | 90 +++++++-
23 files changed, 1070 insertions(+), 423 deletions(-)
diff --git
a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gitinput.adoc
b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gitinput.adoc
index 4b179526aa..061456f420 100644
--- a/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gitinput.adoc
+++ b/docs/hop-user-manual/modules/ROOT/pages/pipeline/transforms/gitinput.adoc
@@ -86,7 +86,13 @@ Only used when the source is `REMOTE`. For a local
repository the owner and name
== Output fields
-Every row has the same fields whatever the provider or resource type.
+The row layout is chosen per **Resource type**, not shared by all of them. A
single layout would have to be the union of everything any type can report, so
a commit row would carry empty label and assignee columns and an issue row
would carry empty branch and merge columns.
+
+Within a type the layout is identical across providers, so a pipeline written
against GitHub keeps working when it is pointed at GitLab or Bitbucket. A field
a given provider does not report arrives as an empty value, not as a missing
column.
+
+Changing **Resource type** changes the fields the transform emits, so re-open
the dialog after changing it to let the downstream transforms pick the new
layout up.
+
+=== Fields every type reports
[options="header"]
|===
@@ -95,19 +101,49 @@ Every row has the same fields whatever the provider or
resource type.
|entity_type|String|The resource type that produced the row, for example
`commits` or `pull_requests`.
|repo_owner, repo_name|String|The repository the row belongs to.
|id|String|Provider identifier: a GitHub node id, a GitLab id, or the commit
SHA.
-|number|Integer|Issue or pull request number. GitLab uses the issue **iid**.
-|title|String|Issue or pull request title, the commit message subject, or the
changed file path for `COMMIT_FILES`.
-|state|String|Issue or pull request state, or the change type for
`COMMIT_FILES`.
-|author|String|Commit author, issue reporter, or event actor.
-|created_at, updated_at, closed_at|Date|Real Date values, so they can be
compared and formatted without a Select Values transform. A timestamp that
cannot be parsed becomes `null` rather than failing the row; the original text
stays in `raw_json`.
|url|String|Link to the item on the provider.
-|body|String|Commit message, issue or comment body, or the previous path for a
renamed file.
-|sha|String|Commit hash.
-|source_branch, target_branch|String|Pull or merge request branches.
-|merged|String|`Y` or `N`.
-|raw_json|String|The full provider payload. Left out when **Include raw JSON
in output** is unchecked.
+|raw_json|String|The full provider payload, when **Include raw JSON in
output** is checked. Always the last field, so unchecking it never shifts the
fields in front of it.
|===
+=== COMMITS
+
+`sha`, `title` (message subject), `body` (full message), `author`,
`author_email`, `author_login`, `committer`, `committer_email`, `created_at`,
`is_merge`.
+
+A commit carries two idents. `author` and `author_email` are who wrote the
change; `committer` and `committer_email` are who applied it, and the two
differ after a rebase, a squash, a cherry-pick or a merge made through a
provider's web UI. `author_login` is the provider account the ident was matched
to, which is a different thing again: `author` is whatever the person put in
their `git config`, `author_login` is who they are on the platform, and it is
the only one of the three that reli [...]
+
+`is_merge` is a Boolean, true when the commit has more than one parent, which
is what you want to filter out of contributor statistics. It is not related to
`merged` on a pull request: `is_merge` says the commit *is* a merge, `merged`
says a pull request *was* merged. The two never appear on the same row.
+
+=== COMMIT_FILES
+
+`sha`, `title` (file path), `state` (change type), `body` (previous path of a
rename or copy), `author`, `author_email`, `author_login`, `created_at`.
+
+=== ISSUES
+
+`number`, `title`, `state`, `body`, `author`, `author_login`, `labels`,
`assignees`, `created_at`, `updated_at`, `closed_at`.
+
+Issues identify people by account rather than by e-mail, so there is no
`author_email`. `labels` and `assignees` are comma-separated lists, flattened
from the provider's array. Bitbucket has no labels and carries a single
assignee rather than a list.
+
+=== PULL_REQUESTS
+
+The `ISSUES` fields, plus `source_branch`, `target_branch`, `merged` and
`merged_at`.
+
+`merged` is a Boolean; `merged_at` is when it happened, which is not the same
as `closed_at` and is what a lead-time or cycle-time calculation needs.
Bitbucket does not report a merge time.
+
+=== ISSUE_COMMENTS, PR_COMMENTS
+
+`number` (the parent issue or pull request), `title`, `state`, `body`,
`author`, `author_login`, `created_at`, `updated_at`.
+
+For these two types `title` and `state` carry the event type and event detail
rather than the issue title and state.
+
+=== ISSUE_EVENTS
+
+`number`, `title` (event type), `state` (event detail), `body`, `author`,
`author_login`, `created_at`.
+
+=== Dates
+
+`created_at`, `updated_at`, `closed_at` and `merged_at` are real Date values,
so they can be compared and formatted without a Select Values transform. A
timestamp that cannot be parsed becomes `null` rather than failing the row; the
original text stays in `raw_json`.
+
+
Providers format timestamps differently: GitHub uses `2026-05-01T12:00:00Z`,
GitLab adds milliseconds and Bitbucket uses microsecond precision. All of them
are parsed to the same instant.
=== Reading every page
@@ -125,6 +161,10 @@ That is the right setting for a full extract, but it hands
control of the reques
Set **Source** to `LOCAL` and point **Local repository path** at the folder
that contains `.git`. No connection or token is needed.
+Commit rows carry both Git idents in full: `author` and `author_email` from
the author ident, `committer` and `committer_email` from the committer ident,
and `is_merge` from the parent count. `author_login` stays empty, because a
clone on disk has no provider accounts to match idents against. All of it is
repeated in `raw_json`.
+
+That is enough for a contributor list: group on `author` and `author_email`,
count the rows, and filter on `is_merge` being false to leave merge commits out
of the totals.
+
`COMMIT_FILES` emits one row per changed file per commit, with the file path
in `title` and the change type (`added`, `modified`, `deleted`, `renamed`,
`copied`) in `state`. Rename and copy detection is on, so moving a file
produces a single `renamed` row carrying its previous path in `body` rather
than a `deleted` row plus an `added` row. Merge commits are compared against
their first parent.
== Issue comments and events
@@ -143,8 +183,6 @@ On GitHub both come from one repository-wide feed and are
told apart by the comm
NOTE: Gitea and Forgejo do not support `ISSUE_COMMENTS`, `PR_COMMENTS` or
`ISSUE_EVENTS` yet; the transform reports that rather than returning an empty
result.
-For these two types `title` and `state` carry the event type and event detail
rather than the issue title and state, and `number` identifies the parent issue.
-
The GitHub comment feed returns comments without their parent issue, so
`title` and `state` are empty for GitHub `ISSUE_COMMENTS` rows. That feed also
ignores **State**; use **Since** to limit it by comment time. GitLab and
Bitbucket walk issues one at a time and do fill both fields.
The row cap applies to the rows that come out, not to the number of issues
scanned.
diff --git
a/integration-tests/transforms/0112-git-input-github-pull-requests.hpl
b/integration-tests/transforms/0112-git-input-github-pull-requests.hpl
index e6ec53161b..3525d8ff43 100644
--- a/integration-tests/transforms/0112-git-input-github-pull-requests.hpl
+++ b/integration-tests/transforms/0112-git-input-github-pull-requests.hpl
@@ -249,11 +249,11 @@ limitations under the License.
<negated>N</negated>
<operator>AND</operator>
<leftvalue>merged</leftvalue>
- <function>IN LIST</function>
+ <function>IS NOT NULL</function>
<value>
<name>constant</name>
<type>String</type>
- <text>Y;N</text>
+ <text/>
<length>-1</length>
<precision>-1</precision>
<isnull>N</isnull>
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInput.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInput.java
index 8de215f531..f1f4ebd585 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInput.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInput.java
@@ -99,7 +99,7 @@ public class GitInput extends BaseTransform<GitInputMeta,
GitInputData> {
if (data.reader != null && data.reader.hasNext()) {
GitResourceRecord record = data.reader.next();
- Object[] outputRow = record.toRow(meta.isIncludeRawJson());
+ Object[] outputRow = record.toRow(data.resourceType,
meta.isIncludeRawJson());
incrementLinesInput();
putRow(data.outputRowMeta, outputRow);
return true;
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMeta.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMeta.java
index 552f75df07..14a1452860 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMeta.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMeta.java
@@ -331,20 +331,27 @@ public class GitInputMeta extends
BaseTransformMeta<GitInput, GitInputData> {
IHopMetadataProvider metadataProvider)
throws HopTransformException {
- for (int i = 0; i < GitInputFields.fieldCount(includeRawJson); i++) {
+ // The layout depends on what is being read: a commit row has no labels or
branches, an issue
+ // row has no commit idents. An unparseable resource type falls back to
COMMITS so the dialog
+ // can still show a field list while the type is a variable that is not
set yet.
+ GitResourceType type;
+ try {
+ type = GitResourceType.fromStored(variables.resolve(resourceType));
+ } catch (IllegalArgumentException e) {
+ type = GitResourceType.COMMITS;
+ }
+
+ for (GitInputFields.Field field : GitInputFields.layout(type,
includeRawJson)) {
IValueMeta valueMeta;
try {
- valueMeta =
- ValueMetaFactory.createValueMeta(
- GitInputFields.FIELD_NAMES[i], GitInputFields.FIELD_TYPES[i]);
+ valueMeta = ValueMetaFactory.createValueMeta(field.getFieldName(),
field.getType());
} catch (HopPluginException e) {
throw new HopTransformException(
- BaseMessages.getString(
- PKG, "GitInputMeta.Error.CreateValueMeta",
GitInputFields.FIELD_NAMES[i]),
+ BaseMessages.getString(PKG, "GitInputMeta.Error.CreateValueMeta",
field.getFieldName()),
e);
}
- if (GitInputFields.FIELD_LENGTHS[i] > 0) {
- valueMeta.setLength(GitInputFields.FIELD_LENGTHS[i]);
+ if (field.getLength() > 0) {
+ valueMeta.setLength(field.getLength());
}
valueMeta.setOrigin(name);
rowMeta.addValueMeta(valueMeta);
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/BitbucketResourceClient.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/BitbucketResourceClient.java
index 2c7b39f638..975bac6f8c 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/BitbucketResourceClient.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/BitbucketResourceClient.java
@@ -21,6 +21,8 @@ import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.ArrayList;
import java.util.List;
+import java.util.regex.Matcher;
+import java.util.regex.Pattern;
import org.apache.hop.core.exception.HopException;
import org.json.simple.JSONArray;
import org.json.simple.JSONObject;
@@ -29,6 +31,9 @@ import org.json.simple.parser.ParseException;
class BitbucketResourceClient implements GitResourceClient {
+ /** Matches the e-mail inside a raw Git ident such as {@code Ada Lovelace
<[email protected]>}. */
+ private static final Pattern RAW_IDENT_EMAIL = Pattern.compile("<([^>]*)>");
+
private static final String ACCEPT = "application/json";
private static final String PROVIDER = "bitbucket";
private static final int BITBUCKET_MAX_PAGE_LEN = 100;
@@ -314,26 +319,23 @@ class BitbucketResourceClient implements
GitResourceClient {
case COMMITS -> {
JSONObject author = (JSONObject) json.get("author");
JSONObject user = author != null ? (JSONObject) author.get("user") :
null;
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "hash"),
- 0L,
- firstLine(renderedText(json, "message")),
- "",
- user != null ? GitApiHttp.getString(user, "display_name") : "",
- GitApiHttp.getString(json, "date"),
- "",
- "",
- extractLink(json, "html"),
- renderedText(json, "message"),
- GitApiHttp.getString(json, "hash"),
- "",
- "",
- "",
- rawJson);
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "hash"))
+ .sha(GitApiHttp.getString(json, "hash"))
+ .title(firstLine(renderedText(json, "message")))
+ .body(renderedText(json, "message"))
+ .author(user != null ? GitApiHttp.getString(user, "display_name")
: "")
+ .authorEmail(author != null ?
rawEmail(GitApiHttp.getString(author, "raw")) : "")
+ .authorLogin(user != null ? GitApiHttp.getString(user, "nickname")
: "")
+ .createdAt(GitApiHttp.getString(json, "date"))
+ .isMerge(GitJsonLists.mergeFlag(json, "parents"))
+ .url(extractLink(json, "html"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUES, PULL_REQUESTS -> {
JSONObject reporter =
@@ -346,26 +348,28 @@ class BitbucketResourceClient implements
GitResourceClient {
resourceType == GitResourceType.PULL_REQUESTS
? (JSONObject) json.get("destination")
: null;
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "id"),
- GitApiHttp.getLong(json, "id"),
- GitApiHttp.getString(json, "title"),
- GitApiHttp.getString(json, "state"),
- reporter != null ? GitApiHttp.getString(reporter, "display_name")
: "",
- GitApiHttp.getString(json, "created_on"),
- GitApiHttp.getString(json, "updated_on"),
- "",
- extractLink(json, "html"),
- renderedText(json, "content"),
- "",
- branchName(source),
- branchName(destination),
- "MERGED".equalsIgnoreCase(GitApiHttp.getString(json, "state")) ?
"Y" : "N",
- rawJson);
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "id"))
+ .number(GitApiHttp.getLong(json, "id"))
+ .title(GitApiHttp.getString(json, "title"))
+ .state(GitApiHttp.getString(json, "state"))
+ .body(renderedText(json, "content"))
+ .author(reporter != null ? GitApiHttp.getString(reporter,
"display_name") : "")
+ .authorLogin(reporter != null ? GitApiHttp.getString(reporter,
"nickname") : "")
+ // Bitbucket has no labels, and carries a single assignee rather
than a list.
+ .assignees(GitJsonLists.names(json, "assignee", "display_name"))
+ .sourceBranch(branchName(source))
+ .targetBranch(branchName(destination))
+ .merged("MERGED".equalsIgnoreCase(GitApiHttp.getString(json,
"state")))
+ .createdAt(GitApiHttp.getString(json, "created_on"))
+ .updatedAt(GitApiHttp.getString(json, "updated_on"))
+ .url(extractLink(json, "html"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUE_COMMENTS, PR_COMMENTS, ISSUE_EVENTS, COMMIT_FILES ->
throw new IllegalStateException();
@@ -452,6 +456,19 @@ class BitbucketResourceClient implements GitResourceClient
{
return query.toString();
}
+ /**
+ * Bitbucket reports a commit author as the raw Git ident, {@code Name
<[email protected]>}, and
+ * only resolves the linked account under {@code user}. The address is
pulled back out of the
+ * ident because an unlinked author has no {@code user} object at all.
+ */
+ private static String rawEmail(String raw) {
+ if (raw == null) {
+ return "";
+ }
+ Matcher matcher = RAW_IDENT_EMAIL.matcher(raw);
+ return matcher.find() ? matcher.group(1).trim() : "";
+ }
+
private static String firstLine(String message) {
if (message == null || message.isBlank()) {
return "";
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitHubResourceClient.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitHubResourceClient.java
index 9c071e2e11..f24e35d7a8 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitHubResourceClient.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitHubResourceClient.java
@@ -659,26 +659,29 @@ class GitHubResourceClient implements GitResourceClient {
case COMMITS -> {
JSONObject commit = (JSONObject) json.get("commit");
JSONObject author = commit != null ? (JSONObject) commit.get("author")
: null;
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "sha"),
- 0L,
- firstLine(GitApiHttp.getString(commit, "message")),
- "",
- author != null ? GitApiHttp.getString(author, "name") : "",
- author != null ? GitApiHttp.getString(author, "date") : "",
- "",
- "",
- GitApiHttp.getString(json, "html_url"),
- commit != null ? GitApiHttp.getString(commit, "message") : "",
- GitApiHttp.getString(json, "sha"),
- "",
- "",
- "",
- rawJson);
+ JSONObject committer = commit != null ? (JSONObject)
commit.get("committer") : null;
+ // The nested commit.author is the Git ident the commit was written
with; the top level
+ // author is the GitHub account it was matched to, which is absent for
an unmatched ident.
+ JSONObject authorAccount = (JSONObject) json.get("author");
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "sha"))
+ .sha(GitApiHttp.getString(json, "sha"))
+ .title(firstLine(GitApiHttp.getString(commit, "message")))
+ .body(commit != null ? GitApiHttp.getString(commit, "message") :
"")
+ .author(author != null ? GitApiHttp.getString(author, "name") : "")
+ .authorEmail(author != null ? GitApiHttp.getString(author,
"email") : "")
+ .authorLogin(authorAccount != null ?
GitApiHttp.getString(authorAccount, "login") : "")
+ .committer(committer != null ? GitApiHttp.getString(committer,
"name") : "")
+ .committerEmail(committer != null ?
GitApiHttp.getString(committer, "email") : "")
+ .createdAt(author != null ? GitApiHttp.getString(author, "date") :
"")
+ .isMerge(GitJsonLists.mergeFlag(json, "parents"))
+ .url(GitApiHttp.getString(json, "html_url"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUES, PULL_REQUESTS -> {
JSONObject user = (JSONObject) json.get("user");
@@ -686,26 +689,31 @@ class GitHubResourceClient implements GitResourceClient {
resourceType == GitResourceType.PULL_REQUESTS ? (JSONObject)
json.get("head") : null;
JSONObject base =
resourceType == GitResourceType.PULL_REQUESTS ? (JSONObject)
json.get("base") : null;
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "node_id"),
- GitApiHttp.getLong(json, "number"),
- GitApiHttp.getString(json, "title"),
- GitApiHttp.getString(json, "state"),
- user != null ? GitApiHttp.getString(user, "login") : "",
- GitApiHttp.getString(json, "created_at"),
- GitApiHttp.getString(json, "updated_at"),
- GitApiHttp.getString(json, "closed_at"),
- GitApiHttp.getString(json, "html_url"),
- GitApiHttp.getString(json, "body"),
- "",
- head != null ? GitApiHttp.getString(head, "ref") : "",
- base != null ? GitApiHttp.getString(base, "ref") : "",
- mapGithubMerged(json),
- rawJson);
+ String login = user != null ? GitApiHttp.getString(user, "login") : "";
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "node_id"))
+ .number(GitApiHttp.getLong(json, "number"))
+ .title(GitApiHttp.getString(json, "title"))
+ .state(GitApiHttp.getString(json, "state"))
+ .body(GitApiHttp.getString(json, "body"))
+ .author(login)
+ .authorLogin(login)
+ .labels(GitJsonLists.names(json, "labels", "name"))
+ .assignees(GitJsonLists.names(json, "assignees", "login"))
+ .sourceBranch(head != null ? GitApiHttp.getString(head, "ref") :
"")
+ .targetBranch(base != null ? GitApiHttp.getString(base, "ref") :
"")
+ .merged(mapGithubMerged(json))
+ .mergedAt(GitApiHttp.getString(json, "merged_at"))
+ .createdAt(GitApiHttp.getString(json, "created_at"))
+ .updatedAt(GitApiHttp.getString(json, "updated_at"))
+ .closedAt(GitApiHttp.getString(json, "closed_at"))
+ .url(GitApiHttp.getString(json, "html_url"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUE_COMMENTS, PR_COMMENTS, ISSUE_EVENTS, COMMIT_FILES ->
throw new IllegalStateException();
@@ -713,13 +721,12 @@ class GitHubResourceClient implements GitResourceClient {
}
/** GitHub list-pulls responses often omit {@code merged}; {@code merged_at}
is reliable. */
- static String mapGithubMerged(org.json.simple.JSONObject json) {
+ static boolean mapGithubMerged(org.json.simple.JSONObject json) {
Object merged = json.get("merged");
if (merged instanceof Boolean booleanValue) {
- return booleanValue ? "Y" : "N";
+ return booleanValue;
}
- String mergedAt = GitApiHttp.getString(json, "merged_at");
- return mergedAt.isBlank() ? "N" : "Y";
+ return !GitApiHttp.getString(json, "merged_at").isBlank();
}
private static JSONArray parseArray(String body) throws HopException {
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitInputFields.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitInputFields.java
index a66805c30f..e9ea4d3f3f 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitInputFields.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitInputFields.java
@@ -17,67 +17,211 @@
package org.apache.hop.git.provider;
+import java.util.EnumMap;
+import java.util.List;
+import java.util.Map;
+import lombok.Getter;
import org.apache.hop.core.row.IValueMeta;
-/** Describes the normalized output row layout shared by every provider. */
+/**
+ * The output row layout, which is chosen per {@link GitResourceType} rather
than shared by all of
+ * them.
+ *
+ * <p>A single row shared by every resource type would have to be the union of
everything any type
+ * can report, so a commit row would carry empty label and assignee columns
and an issue row would
+ * carry empty branch and merge columns. Each type therefore declares only the
fields it can
+ * actually fill.
+ *
+ * <p>Within a type the layout is still identical across providers, so a
pipeline written against
+ * GitHub keeps working when it is pointed at GitLab or Bitbucket. A field a
given provider does not
+ * report is an empty value, not a missing column.
+ */
public final class GitInputFields {
private GitInputFields() {}
- public static final String[] FIELD_NAMES = {
- "provider",
- "entity_type",
- "repo_owner",
- "repo_name",
- "id",
- "number",
- "title",
- "state",
- "author",
- "created_at",
- "updated_at",
- "closed_at",
- "url",
- "body",
- "sha",
- "source_branch",
- "target_branch",
- "merged",
- "raw_json"
- };
-
- public static final int[] FIELD_TYPES = {
- IValueMeta.TYPE_STRING, // provider
- IValueMeta.TYPE_STRING, // entity_type
- IValueMeta.TYPE_STRING, // repo_owner
- IValueMeta.TYPE_STRING, // repo_name
- IValueMeta.TYPE_STRING, // id
- IValueMeta.TYPE_INTEGER, // number
- IValueMeta.TYPE_STRING, // title
- IValueMeta.TYPE_STRING, // state
- IValueMeta.TYPE_STRING, // author
- IValueMeta.TYPE_DATE, // created_at
- IValueMeta.TYPE_DATE, // updated_at
- IValueMeta.TYPE_DATE, // closed_at
- IValueMeta.TYPE_STRING, // url
- IValueMeta.TYPE_STRING, // body
- IValueMeta.TYPE_STRING, // sha
- IValueMeta.TYPE_STRING, // source_branch
- IValueMeta.TYPE_STRING, // target_branch
- IValueMeta.TYPE_STRING, // merged
- IValueMeta.TYPE_STRING // raw_json
- };
-
- /** Display length per field; {@code -1} leaves the length unset. */
- public static final int[] FIELD_LENGTHS = {
- 32, 32, 128, 128, 64, -1, 512, 32, 128, -1, -1, -1, 512, -1, 64, 128, 128,
8, -1
- };
-
- /** Index of {@code raw_json} in {@link #FIELD_NAMES}. */
- public static final int RAW_JSON_FIELD_INDEX = FIELD_NAMES.length - 1;
-
- /** Number of output fields, which depends on whether {@code raw_json} is
included. */
- public static int fieldCount(boolean includeRawJson) {
- return includeRawJson ? FIELD_NAMES.length : FIELD_NAMES.length - 1;
+ /** A single output field: its name in the row, its Hop type, and its
display length. */
+ @Getter
+ public enum Field {
+ PROVIDER("provider", IValueMeta.TYPE_STRING, 32),
+ ENTITY_TYPE("entity_type", IValueMeta.TYPE_STRING, 32),
+ REPO_OWNER("repo_owner", IValueMeta.TYPE_STRING, 128),
+ REPO_NAME("repo_name", IValueMeta.TYPE_STRING, 128),
+ ID("id", IValueMeta.TYPE_STRING, 64),
+ NUMBER("number", IValueMeta.TYPE_INTEGER, -1),
+ TITLE("title", IValueMeta.TYPE_STRING, 512),
+ STATE("state", IValueMeta.TYPE_STRING, 32),
+ AUTHOR("author", IValueMeta.TYPE_STRING, 128),
+ AUTHOR_EMAIL("author_email", IValueMeta.TYPE_STRING, 256),
+ AUTHOR_LOGIN("author_login", IValueMeta.TYPE_STRING, 128),
+ COMMITTER("committer", IValueMeta.TYPE_STRING, 128),
+ COMMITTER_EMAIL("committer_email", IValueMeta.TYPE_STRING, 256),
+ LABELS("labels", IValueMeta.TYPE_STRING, 512),
+ ASSIGNEES("assignees", IValueMeta.TYPE_STRING, 512),
+ CREATED_AT("created_at", IValueMeta.TYPE_DATE, -1),
+ UPDATED_AT("updated_at", IValueMeta.TYPE_DATE, -1),
+ CLOSED_AT("closed_at", IValueMeta.TYPE_DATE, -1),
+ MERGED_AT("merged_at", IValueMeta.TYPE_DATE, -1),
+ URL("url", IValueMeta.TYPE_STRING, 512),
+ BODY("body", IValueMeta.TYPE_STRING, -1),
+ SHA("sha", IValueMeta.TYPE_STRING, 64),
+ SOURCE_BRANCH("source_branch", IValueMeta.TYPE_STRING, 128),
+ TARGET_BRANCH("target_branch", IValueMeta.TYPE_STRING, 128),
+ MERGED("merged", IValueMeta.TYPE_BOOLEAN, -1),
+ IS_MERGE("is_merge", IValueMeta.TYPE_BOOLEAN, -1),
+ RAW_JSON("raw_json", IValueMeta.TYPE_STRING, -1);
+
+ private final String fieldName;
+ private final int type;
+ private final int length;
+
+ Field(String fieldName, int type, int length) {
+ this.fieldName = fieldName;
+ this.type = type;
+ this.length = length;
+ }
+ }
+
+ /**
+ * The fields every type reports, in the order they lead each row. {@code
id} is the provider
+ * identifier and {@code url} the link back to the item.
+ */
+ private static final List<Field> IDENTITY =
+ List.of(Field.PROVIDER, Field.ENTITY_TYPE, Field.REPO_OWNER,
Field.REPO_NAME, Field.ID);
+
+ private static final Map<GitResourceType, List<Field>> LAYOUTS =
+ new EnumMap<>(GitResourceType.class);
+
+ static {
+ // A commit has two idents, author and committer, which differ on a
rebase, a squash, a
+ // cherry-pick or a merge made through a provider's web UI.
+ LAYOUTS.put(
+ GitResourceType.COMMITS,
+ concat(
+ Field.SHA,
+ Field.TITLE,
+ Field.BODY,
+ Field.AUTHOR,
+ Field.AUTHOR_EMAIL,
+ Field.AUTHOR_LOGIN,
+ Field.COMMITTER,
+ Field.COMMITTER_EMAIL,
+ Field.CREATED_AT,
+ Field.IS_MERGE,
+ Field.URL));
+
+ // One row per changed file: title is the path, state the change type,
body the previous path
+ // of a rename or a copy.
+ LAYOUTS.put(
+ GitResourceType.COMMIT_FILES,
+ concat(
+ Field.SHA,
+ Field.TITLE,
+ Field.STATE,
+ Field.BODY,
+ Field.AUTHOR,
+ Field.AUTHOR_EMAIL,
+ Field.AUTHOR_LOGIN,
+ Field.CREATED_AT,
+ Field.URL));
+
+ // Issues and pull requests identify people by account, so there is no
e-mail to report.
+ LAYOUTS.put(
+ GitResourceType.ISSUES,
+ concat(
+ Field.NUMBER,
+ Field.TITLE,
+ Field.STATE,
+ Field.BODY,
+ Field.AUTHOR,
+ Field.AUTHOR_LOGIN,
+ Field.LABELS,
+ Field.ASSIGNEES,
+ Field.CREATED_AT,
+ Field.UPDATED_AT,
+ Field.CLOSED_AT,
+ Field.URL));
+
+ LAYOUTS.put(
+ GitResourceType.PULL_REQUESTS,
+ concat(
+ Field.NUMBER,
+ Field.TITLE,
+ Field.STATE,
+ Field.BODY,
+ Field.AUTHOR,
+ Field.AUTHOR_LOGIN,
+ Field.LABELS,
+ Field.ASSIGNEES,
+ Field.SOURCE_BRANCH,
+ Field.TARGET_BRANCH,
+ Field.MERGED,
+ Field.MERGED_AT,
+ Field.CREATED_AT,
+ Field.UPDATED_AT,
+ Field.CLOSED_AT,
+ Field.URL));
+
+ // For a comment, number is the parent issue or pull request.
+ List<Field> comment =
+ concat(
+ Field.NUMBER,
+ Field.TITLE,
+ Field.STATE,
+ Field.BODY,
+ Field.AUTHOR,
+ Field.AUTHOR_LOGIN,
+ Field.CREATED_AT,
+ Field.UPDATED_AT,
+ Field.URL);
+ LAYOUTS.put(GitResourceType.ISSUE_COMMENTS, comment);
+ LAYOUTS.put(GitResourceType.PR_COMMENTS, comment);
+
+ // For an event, title is the event type and state its detail.
+ LAYOUTS.put(
+ GitResourceType.ISSUE_EVENTS,
+ concat(
+ Field.NUMBER,
+ Field.TITLE,
+ Field.STATE,
+ Field.BODY,
+ Field.AUTHOR,
+ Field.AUTHOR_LOGIN,
+ Field.CREATED_AT,
+ Field.URL));
+ }
+
+ private static List<Field> concat(Field... typeFields) {
+ return java.util.stream.Stream.concat(IDENTITY.stream(),
java.util.stream.Stream.of(typeFields))
+ .toList();
+ }
+
+ /**
+ * The output fields for a resource type, in row order. {@code raw_json} is
appended last when it
+ * is requested, so dropping it never shifts the fields in front of it.
+ */
+ public static List<Field> layout(GitResourceType resourceType, boolean
includeRawJson) {
+ List<Field> fields = LAYOUTS.get(resourceType);
+ if (fields == null) {
+ throw new IllegalArgumentException("No output layout for resource type "
+ resourceType);
+ }
+ if (!includeRawJson) {
+ return fields;
+ }
+ return java.util.stream.Stream.concat(
+ fields.stream(), java.util.stream.Stream.of(Field.RAW_JSON))
+ .toList();
+ }
+
+ /** Number of output fields for a resource type. */
+ public static int fieldCount(GitResourceType resourceType, boolean
includeRawJson) {
+ return layout(resourceType, includeRawJson).size();
+ }
+
+ /** Field names for a resource type, for the dialog and for tests. */
+ public static String[] fieldNames(GitResourceType resourceType, boolean
includeRawJson) {
+ return layout(resourceType, includeRawJson).stream()
+ .map(Field::getFieldName)
+ .toArray(String[]::new);
}
}
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitIssueActivityMapper.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitIssueActivityMapper.java
index dd1acc0651..2c287eae6d 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitIssueActivityMapper.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitIssueActivityMapper.java
@@ -38,26 +38,23 @@ final class GitIssueActivityMapper {
String url,
String body,
JSONObject rawJson) {
- return new GitResourceRecord(
- provider,
- commentType.getEntityType(),
- owner,
- repository,
- id,
- issueNumber,
- issueTitle,
- issueState,
- author,
- createdAt,
- updatedAt,
- "",
- url,
- body,
- "",
- "",
- "",
- "",
- rawJson.toJSONString());
+ return GitResourceRecord.builder()
+ .provider(provider)
+ .entityType(commentType.getEntityType())
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(id)
+ .number(issueNumber)
+ .title(issueTitle)
+ .state(issueState)
+ .body(body)
+ .author(author)
+ .authorLogin(author)
+ .createdAt(createdAt)
+ .updatedAt(updatedAt)
+ .url(url)
+ .rawJson(rawJson.toJSONString())
+ .build();
}
static GitResourceRecord event(
@@ -74,25 +71,21 @@ final class GitIssueActivityMapper {
String createdAt,
String url,
JSONObject rawJson) {
- return new GitResourceRecord(
- provider,
- GitResourceType.ISSUE_EVENTS.getEntityType(),
- owner,
- repository,
- id,
- issueNumber,
- eventType,
- eventDetail,
- actor,
- createdAt,
- "",
- "",
- url,
- eventDetail,
- "",
- "",
- "",
- "",
- rawJson.toJSONString());
+ return GitResourceRecord.builder()
+ .provider(provider)
+ .entityType(GitResourceType.ISSUE_EVENTS.getEntityType())
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(id)
+ .number(issueNumber)
+ .title(eventType)
+ .state(eventDetail)
+ .body(eventDetail)
+ .author(actor)
+ .authorLogin(actor)
+ .createdAt(createdAt)
+ .url(url)
+ .rawJson(rawJson.toJSONString())
+ .build();
}
}
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitJsonLists.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitJsonLists.java
new file mode 100644
index 0000000000..41a6b5a6e9
--- /dev/null
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitJsonLists.java
@@ -0,0 +1,81 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.git.provider;
+
+import java.util.ArrayList;
+import java.util.List;
+import org.json.simple.JSONArray;
+import org.json.simple.JSONObject;
+
+/**
+ * Flattens the small JSON arrays that providers use for labels, assignees and
commit parents into
+ * the single output field the row carries.
+ */
+final class GitJsonLists {
+
+ /** Separator between the entries of a flattened list field. */
+ static final String SEPARATOR = ", ";
+
+ private GitJsonLists() {}
+
+ /**
+ * Joins a list of labels or assignees into one comma-separated value.
+ *
+ * <p>Providers disagree on the shape: GitHub and Gitea return objects to
read {@code nameKey}
+ * from, GitLab returns plain strings, and a single-valued field such as a
Bitbucket assignee is
+ * an object rather than an array. All three are accepted, and anything else
yields an empty value
+ * rather than a failed row.
+ */
+ static String names(JSONObject json, String arrayKey, String nameKey) {
+ if (json == null) {
+ return "";
+ }
+ Object value = json.get(arrayKey);
+ if (value instanceof JSONObject single) {
+ return GitApiHttp.getString(single, nameKey);
+ }
+ if (!(value instanceof JSONArray array)) {
+ return "";
+ }
+ List<String> names = new ArrayList<>(array.size());
+ for (Object entry : array) {
+ String name =
+ switch (entry) {
+ case JSONObject object -> GitApiHttp.getString(object, nameKey);
+ case String plain -> plain;
+ case null, default -> "";
+ };
+ if (!name.isBlank()) {
+ names.add(name);
+ }
+ }
+ return String.join(SEPARATOR, names);
+ }
+
+ /**
+ * True when a commit has more than one parent. Providers report the parents
as an array of
+ * objects ({@code parents}) or of ids ({@code parent_ids}); an absent array
means the payload
+ * does not say, which is reported as false rather than as a merge.
+ */
+ static boolean mergeFlag(JSONObject json, String arrayKey) {
+ if (json == null || !(json.get(arrayKey) instanceof JSONArray array)) {
+ return false;
+ }
+ return array.size() > 1;
+ }
+}
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitLabResourceClient.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitLabResourceClient.java
index 9681f06b28..45edeb72d4 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitLabResourceClient.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitLabResourceClient.java
@@ -330,48 +330,51 @@ class GitLabResourceClient implements GitResourceClient {
return switch (resourceType) {
case COMMITS ->
- new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "id"),
- 0L,
- firstLine(GitApiHttp.getString(json, "message")),
- "",
- GitApiHttp.getString(json, "author_name"),
- GitApiHttp.getString(json, "created_at"),
- "",
- "",
- GitApiHttp.getString(json, "web_url"),
- GitApiHttp.getString(json, "message"),
- GitApiHttp.getString(json, "id"),
- "",
- "",
- "",
- rawJson);
+ GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "id"))
+ .sha(GitApiHttp.getString(json, "id"))
+ .title(firstLine(GitApiHttp.getString(json, "message")))
+ .body(GitApiHttp.getString(json, "message"))
+ .author(GitApiHttp.getString(json, "author_name"))
+ .authorEmail(GitApiHttp.getString(json, "author_email"))
+ .committer(GitApiHttp.getString(json, "committer_name"))
+ .committerEmail(GitApiHttp.getString(json, "committer_email"))
+ .createdAt(GitApiHttp.getString(json, "created_at"))
+ .isMerge(GitJsonLists.mergeFlag(json, "parent_ids"))
+ .url(GitApiHttp.getString(json, "web_url"))
+ .rawJson(rawJson)
+ .build();
case ISSUES, PULL_REQUESTS -> {
JSONObject author = (JSONObject) json.get("author");
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "id"),
- GitApiHttp.getLong(json, "iid"),
- GitApiHttp.getString(json, "title"),
- GitApiHttp.getString(json, "state"),
- author != null ? GitApiHttp.getString(author, "username") : "",
- GitApiHttp.getString(json, "created_at"),
- GitApiHttp.getString(json, "updated_at"),
- GitApiHttp.getString(json, "closed_at"),
- GitApiHttp.getString(json, "web_url"),
- GitApiHttp.getString(json, "description"),
- "",
- GitApiHttp.getString(json, "source_branch"),
- GitApiHttp.getString(json, "target_branch"),
- "merged".equalsIgnoreCase(GitApiHttp.getString(json, "state")) ?
"Y" : "N",
- rawJson);
+ String login = author != null ? GitApiHttp.getString(author,
"username") : "";
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "id"))
+ .number(GitApiHttp.getLong(json, "iid"))
+ .title(GitApiHttp.getString(json, "title"))
+ .state(GitApiHttp.getString(json, "state"))
+ .body(GitApiHttp.getString(json, "description"))
+ .author(login)
+ .authorLogin(login)
+ .labels(GitJsonLists.names(json, "labels", "name"))
+ .assignees(GitJsonLists.names(json, "assignees", "username"))
+ .sourceBranch(GitApiHttp.getString(json, "source_branch"))
+ .targetBranch(GitApiHttp.getString(json, "target_branch"))
+ .merged("merged".equalsIgnoreCase(GitApiHttp.getString(json,
"state")))
+ .mergedAt(GitApiHttp.getString(json, "merged_at"))
+ .createdAt(GitApiHttp.getString(json, "created_at"))
+ .updatedAt(GitApiHttp.getString(json, "updated_at"))
+ .closedAt(GitApiHttp.getString(json, "closed_at"))
+ .url(GitApiHttp.getString(json, "web_url"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUE_COMMENTS, PR_COMMENTS, ISSUE_EVENTS, COMMIT_FILES ->
throw new IllegalStateException();
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitResourceRecord.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitResourceRecord.java
index b44265fa12..a0dae61a2f 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitResourceRecord.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GitResourceRecord.java
@@ -17,10 +17,20 @@
package org.apache.hop.git.provider;
+import java.util.List;
+import lombok.Builder;
import lombok.Getter;
+import org.apache.hop.git.provider.GitInputFields.Field;
-/** Normalized row from a Git hosting provider API (commit, issue, or pull
request). */
+/**
+ * Normalized row from a Git hosting provider or a local clone.
+ *
+ * <p>A record carries every value any resource type can report; {@link
#toRow} then keeps the ones
+ * the type's layout actually asks for. A provider that does not report a
field simply never sets
+ * it, and the field arrives in the row as an empty value rather than as a
missing column.
+ */
@Getter
+@Builder
public class GitResourceRecord {
private final String provider;
@@ -32,85 +42,78 @@ public class GitResourceRecord {
private final String title;
private final String state;
private final String author;
+ private final String authorEmail;
+ private final String authorLogin;
+ private final String committer;
+ private final String committerEmail;
+ private final String labels;
+ private final String assignees;
private final String createdAt;
private final String updatedAt;
private final String closedAt;
+ private final String mergedAt;
private final String url;
private final String body;
private final String sha;
private final String sourceBranch;
private final String targetBranch;
- private final String merged;
+
+ /** Whether a pull request was merged. Null when the row is not a pull
request. */
+ private final Boolean merged;
+
+ /** Whether a commit has more than one parent. Null when the row is not a
commit. */
+ private final Boolean isMerge;
+
private final String rawJson;
- public GitResourceRecord(
- String provider,
- String entityType,
- String repoOwner,
- String repoName,
- String id,
- long number,
- String title,
- String state,
- String author,
- String createdAt,
- String updatedAt,
- String closedAt,
- String url,
- String body,
- String sha,
- String sourceBranch,
- String targetBranch,
- String merged,
- String rawJson) {
- this.provider = provider;
- this.entityType = entityType;
- this.repoOwner = repoOwner;
- this.repoName = repoName;
- this.id = id;
- this.number = number;
- this.title = title;
- this.state = state;
- this.author = author;
- this.createdAt = createdAt;
- this.updatedAt = updatedAt;
- this.closedAt = closedAt;
- this.url = url;
- this.body = body;
- this.sha = sha;
- this.sourceBranch = sourceBranch;
- this.targetBranch = targetBranch;
- this.merged = merged;
- this.rawJson = rawJson;
+ /** Builds the output row for a resource type, in the order {@link
GitInputFields} declares. */
+ public Object[] toRow(GitResourceType resourceType, boolean includeRawJson) {
+ List<Field> layout = GitInputFields.layout(resourceType, includeRawJson);
+ Object[] row = new Object[layout.size()];
+ for (int i = 0; i < layout.size(); i++) {
+ row[i] = valueOf(layout.get(i));
+ }
+ return row;
}
- public Object[] toRow() {
- return toRow(true);
+ /**
+ * The value for one field. Strings are never null, so a provider that does
not report a field
+ * yields a blank cell rather than a null; timestamps are parsed to real
Dates, and an unparseable
+ * one becomes null with the original text left in {@code raw_json}.
+ */
+ private Object valueOf(Field field) {
+ return switch (field) {
+ case PROVIDER -> text(provider);
+ case ENTITY_TYPE -> text(entityType);
+ case REPO_OWNER -> text(repoOwner);
+ case REPO_NAME -> text(repoName);
+ case ID -> text(id);
+ case NUMBER -> number;
+ case TITLE -> text(title);
+ case STATE -> text(state);
+ case AUTHOR -> text(author);
+ case AUTHOR_EMAIL -> text(authorEmail);
+ case AUTHOR_LOGIN -> text(authorLogin);
+ case COMMITTER -> text(committer);
+ case COMMITTER_EMAIL -> text(committerEmail);
+ case LABELS -> text(labels);
+ case ASSIGNEES -> text(assignees);
+ case CREATED_AT -> GitTimestamps.toDate(createdAt);
+ case UPDATED_AT -> GitTimestamps.toDate(updatedAt);
+ case CLOSED_AT -> GitTimestamps.toDate(closedAt);
+ case MERGED_AT -> GitTimestamps.toDate(mergedAt);
+ case URL -> text(url);
+ case BODY -> text(body);
+ case SHA -> text(sha);
+ case SOURCE_BRANCH -> text(sourceBranch);
+ case TARGET_BRANCH -> text(targetBranch);
+ case MERGED -> merged;
+ case IS_MERGE -> isMerge;
+ case RAW_JSON -> text(rawJson);
+ };
}
- public Object[] toRow(boolean includeRawJson) {
- Object[] row = new Object[GitInputFields.fieldCount(includeRawJson)];
- row[0] = provider;
- row[1] = entityType;
- row[2] = repoOwner;
- row[3] = repoName;
- row[4] = id;
- row[5] = number;
- row[6] = title;
- row[7] = state;
- row[8] = author;
- row[9] = GitTimestamps.toDate(createdAt);
- row[10] = GitTimestamps.toDate(updatedAt);
- row[11] = GitTimestamps.toDate(closedAt);
- row[12] = url;
- row[13] = body;
- row[14] = sha;
- row[15] = sourceBranch;
- row[16] = targetBranch;
- row[17] = merged;
- if (includeRawJson) {
- row[GitInputFields.RAW_JSON_FIELD_INDEX] = rawJson;
- }
- return row;
+ private static String text(String value) {
+ return value == null ? "" : value;
}
}
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GiteaResourceClient.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GiteaResourceClient.java
index c8b6f78ca7..2db96855d0 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GiteaResourceClient.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/GiteaResourceClient.java
@@ -191,49 +191,55 @@ class GiteaResourceClient implements GitResourceClient {
commit = json;
}
JSONObject author = (JSONObject) commit.get("author");
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "sha"),
- 0L,
- firstLine(GitApiHttp.getString(commit, "message")),
- "",
- author != null ? GitApiHttp.getString(author, "name") : "",
- author != null ? GitApiHttp.getString(author, "date") : "",
- "",
- "",
- GitApiHttp.getString(json, "html_url"),
- GitApiHttp.getString(commit, "message"),
- GitApiHttp.getString(json, "sha"),
- "",
- "",
- "",
- rawJson);
+ JSONObject committer = (JSONObject) commit.get("committer");
+ JSONObject authorAccount = (JSONObject) json.get("author");
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "sha"))
+ .sha(GitApiHttp.getString(json, "sha"))
+ .title(firstLine(GitApiHttp.getString(commit, "message")))
+ .body(GitApiHttp.getString(commit, "message"))
+ .author(author != null ? GitApiHttp.getString(author, "name") : "")
+ .authorEmail(author != null ? GitApiHttp.getString(author,
"email") : "")
+ .authorLogin(authorAccount != null ?
GitApiHttp.getString(authorAccount, "login") : "")
+ .committer(committer != null ? GitApiHttp.getString(committer,
"name") : "")
+ .committerEmail(committer != null ?
GitApiHttp.getString(committer, "email") : "")
+ .createdAt(author != null ? GitApiHttp.getString(author, "date") :
"")
+ .isMerge(GitJsonLists.mergeFlag(json, "parents"))
+ .url(GitApiHttp.getString(json, "html_url"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUES, PULL_REQUESTS -> {
JSONObject user = (JSONObject) json.get("user");
- yield new GitResourceRecord(
- PROVIDER,
- entityType,
- owner,
- repository,
- GitApiHttp.getString(json, "id"),
- GitApiHttp.getLong(json, "number"),
- GitApiHttp.getString(json, "title"),
- GitApiHttp.getString(json, "state"),
- user != null ? GitApiHttp.getString(user, "login") : "",
- GitApiHttp.getString(json, "created_at"),
- GitApiHttp.getString(json, "updated_at"),
- GitApiHttp.getString(json, "closed_at"),
- GitApiHttp.getString(json, "html_url"),
- GitApiHttp.getString(json, "body"),
- "",
- nestedRef(json, "head"),
- nestedRef(json, "base"),
- Boolean.TRUE.equals(json.get("merged")) ? "Y" : "N",
- rawJson);
+ String login = user != null ? GitApiHttp.getString(user, "login") : "";
+ yield GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(entityType)
+ .repoOwner(owner)
+ .repoName(repository)
+ .id(GitApiHttp.getString(json, "id"))
+ .number(GitApiHttp.getLong(json, "number"))
+ .title(GitApiHttp.getString(json, "title"))
+ .state(GitApiHttp.getString(json, "state"))
+ .body(GitApiHttp.getString(json, "body"))
+ .author(login)
+ .authorLogin(login)
+ .labels(GitJsonLists.names(json, "labels", "name"))
+ .assignees(GitJsonLists.names(json, "assignees", "login"))
+ .sourceBranch(nestedRef(json, "head"))
+ .targetBranch(nestedRef(json, "base"))
+ .merged(Boolean.TRUE.equals(json.get("merged")))
+ .mergedAt(GitApiHttp.getString(json, "merged_at"))
+ .createdAt(GitApiHttp.getString(json, "created_at"))
+ .updatedAt(GitApiHttp.getString(json, "updated_at"))
+ .closedAt(GitApiHttp.getString(json, "closed_at"))
+ .url(GitApiHttp.getString(json, "html_url"))
+ .rawJson(rawJson)
+ .build();
}
case ISSUE_COMMENTS, PR_COMMENTS, ISSUE_EVENTS, COMMIT_FILES ->
throw new IllegalStateException();
diff --git
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/LocalGitResourceClient.java
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/LocalGitResourceClient.java
index 99be900002..05dad6f5f6 100644
---
a/plugins/misc/git/src/main/java/org/apache/hop/git/provider/LocalGitResourceClient.java
+++
b/plugins/misc/git/src/main/java/org/apache/hop/git/provider/LocalGitResourceClient.java
@@ -33,6 +33,7 @@ import org.eclipse.jgit.diff.DiffEntry;
import org.eclipse.jgit.diff.DiffFormatter;
import org.eclipse.jgit.lib.ObjectId;
import org.eclipse.jgit.lib.ObjectReader;
+import org.eclipse.jgit.lib.PersonIdent;
import org.eclipse.jgit.lib.Ref;
import org.eclipse.jgit.lib.Repository;
import org.eclipse.jgit.revwalk.RevCommit;
@@ -389,7 +390,9 @@ public final class LocalGitResourceClient {
String oldPath = realPath(entry.getOldPath());
String newPath = realPath(entry.getNewPath());
String filePath = pickFilePath(entry.getChangeType(), oldPath, newPath);
- String author = commit.getAuthorIdent().getName();
+ PersonIdent authorIdent = commit.getAuthorIdent();
+ String author = identName(authorIdent);
+ String authorEmail = identEmail(authorIdent);
String createdAt =
Instant.ofEpochSecond(commit.getCommitTime()).toString();
String sha = commit.getName();
String url =
@@ -400,27 +403,25 @@ public final class LocalGitResourceClient {
raw.put("change_type", changeType);
raw.put("old_path", oldPath);
raw.put("new_path", newPath);
-
- return new GitResourceRecord(
- PROVIDER,
- GitResourceType.COMMIT_FILES.getEntityType(),
- info.getOwner(),
- info.getRepositoryName(),
- sha + ":" + filePath,
- 0L,
- filePath,
- changeType,
- author,
- createdAt,
- "",
- "",
- url,
- oldPath,
- sha,
- "",
- "",
- "",
- raw.toJSONString());
+ raw.put("author", author);
+ raw.put("author_email", authorEmail);
+
+ return GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(GitResourceType.COMMIT_FILES.getEntityType())
+ .repoOwner(info.getOwner())
+ .repoName(info.getRepositoryName())
+ .id(sha + ":" + filePath)
+ .sha(sha)
+ .title(filePath)
+ .state(changeType)
+ .body(oldPath)
+ .author(author)
+ .authorEmail(authorEmail)
+ .createdAt(createdAt)
+ .url(url)
+ .rawJson(raw.toJSONString())
+ .build();
}
static String mapChangeType(DiffEntry.ChangeType changeType) {
@@ -443,9 +444,13 @@ public final class LocalGitResourceClient {
private static GitResourceRecord toCommitRecord(LocalRepositoryInfo info,
RevCommit commit) {
String message = commit.getFullMessage() == null ? "" :
commit.getFullMessage();
- String author = commit.getAuthorIdent().getName();
+ PersonIdent authorIdent = commit.getAuthorIdent();
+ PersonIdent committerIdent = commit.getCommitterIdent();
+ String author = identName(authorIdent);
+ String authorEmail = identEmail(authorIdent);
String createdAt =
Instant.ofEpochSecond(commit.getCommitTime()).toString();
String sha = commit.getName();
+ boolean isMerge = commit.getParentCount() > 1;
String url =
info.getRemoteUrl().isBlank() ? info.directory().getAbsolutePath() :
info.getRemoteUrl();
@@ -453,28 +458,30 @@ public final class LocalGitResourceClient {
raw.put("sha", sha);
raw.put("message", message);
raw.put("author", author);
+ raw.put("author_email", authorEmail);
+ raw.put("committer", identName(committerIdent));
+ raw.put("committer_email", identEmail(committerIdent));
raw.put("date", createdAt);
-
- return new GitResourceRecord(
- PROVIDER,
- GitResourceType.COMMITS.getEntityType(),
- info.getOwner(),
- info.getRepositoryName(),
- sha,
- 0L,
- firstLine(message),
- "",
- author,
- createdAt,
- "",
- "",
- url,
- message,
- sha,
- "",
- "",
- "",
- raw.toJSONString());
+ raw.put("parent_count", (long) commit.getParentCount());
+
+ return GitResourceRecord.builder()
+ .provider(PROVIDER)
+ .entityType(GitResourceType.COMMITS.getEntityType())
+ .repoOwner(info.getOwner())
+ .repoName(info.getRepositoryName())
+ .id(sha)
+ .sha(sha)
+ .title(firstLine(message))
+ .body(message)
+ .author(author)
+ .authorEmail(authorEmail)
+ .committer(identName(committerIdent))
+ .committerEmail(identEmail(committerIdent))
+ .createdAt(createdAt)
+ .isMerge(isMerge)
+ .url(url)
+ .rawJson(raw.toJSONString())
+ .build();
}
private static ObjectId resolveStart(Repository repo, String branch) throws
Exception {
@@ -516,6 +523,15 @@ public final class LocalGitResourceClient {
}
}
+ /** JGit leaves an unset ident, name or e-mail null; the output row wants a
blank cell instead. */
+ private static String identName(PersonIdent ident) {
+ return ident == null || ident.getName() == null ? "" : ident.getName();
+ }
+
+ private static String identEmail(PersonIdent ident) {
+ return ident == null || ident.getEmailAddress() == null ? "" :
ident.getEmailAddress();
+ }
+
private static String firstLine(String message) {
if (message == null || message.isBlank()) {
return "";
diff --git
a/plugins/misc/git/src/main/samples/transforms/git-input-github-pull-requests.hpl
b/plugins/misc/git/src/main/samples/transforms/git-input-github-pull-requests.hpl
index 14374ed5de..73e8cd4bc2 100644
---
a/plugins/misc/git/src/main/samples/transforms/git-input-github-pull-requests.hpl
+++
b/plugins/misc/git/src/main/samples/transforms/git-input-github-pull-requests.hpl
@@ -45,12 +45,20 @@ The 'github' connection has no token, so this uses the
anonymous GitHub API:
enough for a public repository, but limited to 60 requests per hour. Add a
personal access token for higher limits, or to read a private repository.
-Every provider returns the same row layout, so pointing this at a GitLab,
-Bitbucket, Forgejo or Gitea connection needs no other change.</note>
+Every provider returns the same row layout for a given resource type, so
+pointing this at a GitLab, Bitbucket, Forgejo or Gitea connection needs no
+other change. The layout is per resource type: a pull request row carries
+labels, assignees and merge fields, where a commit row carries the author
+and committer idents instead.
+
+labels and assignees are comma-separated lists flattened from the provider's
+arrays. author_login is the account the pull request was opened from, which
+is what joins it to commits and comments. merged_at is the merge time, which
+is not the same as closed_at.</note>
<xloc>64</xloc>
<yloc>48</yloc>
<width>560</width>
- <heigth>200</heigth>
+ <heigth>280</heigth>
<fontname>Noto Sans</fontname>
<fontsize>11</fontsize>
<fontbold>N</fontbold>
@@ -129,9 +137,18 @@ Bitbucket, Forgejo or Gitea connection needs no other
change.</note>
<field>
<name>author</name>
</field>
+ <field>
+ <name>author_login</name>
+ </field>
<field>
<name>state</name>
</field>
+ <field>
+ <name>labels</name>
+ </field>
+ <field>
+ <name>assignees</name>
+ </field>
<field>
<name>merged</name>
</field>
@@ -141,6 +158,9 @@ Bitbucket, Forgejo or Gitea connection needs no other
change.</note>
<field>
<name>closed_at</name>
</field>
+ <field>
+ <name>merged_at</name>
+ </field>
<field>
<name>source_branch</name>
</field>
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMetaTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMetaTest.java
index d3f5f89042..728228c969 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMetaTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputMetaTest.java
@@ -32,6 +32,8 @@ import org.apache.hop.core.row.IValueMeta;
import org.apache.hop.core.row.RowMeta;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.core.xml.XmlHandler;
+import org.apache.hop.git.provider.GitInputFields;
+import org.apache.hop.git.provider.GitResourceType;
import org.apache.hop.metadata.serializer.xml.XmlMetadataUtil;
import org.apache.hop.pipeline.transform.TransformMeta;
import org.junit.jupiter.api.BeforeAll;
@@ -101,13 +103,14 @@ class GitInputMetaTest {
@Test
void timestampFieldsAreDatesNotStrings() throws Exception {
+ meta.setResourceType(GitResourceType.PULL_REQUESTS.name());
IRowMeta rowMeta = new RowMeta();
meta.getFields(rowMeta, "git", null, null, new Variables(), null);
- assertEquals(19, rowMeta.size());
assertEquals(IValueMeta.TYPE_DATE,
rowMeta.searchValueMeta("created_at").getType());
assertEquals(IValueMeta.TYPE_DATE,
rowMeta.searchValueMeta("updated_at").getType());
assertEquals(IValueMeta.TYPE_DATE,
rowMeta.searchValueMeta("closed_at").getType());
+ assertEquals(IValueMeta.TYPE_DATE,
rowMeta.searchValueMeta("merged_at").getType());
assertEquals(IValueMeta.TYPE_INTEGER,
rowMeta.searchValueMeta("number").getType());
assertEquals(IValueMeta.TYPE_STRING,
rowMeta.searchValueMeta("title").getType());
assertEquals("git", rowMeta.searchValueMeta("title").getOrigin());
@@ -119,10 +122,69 @@ class GitInputMetaTest {
IRowMeta rowMeta = new RowMeta();
meta.getFields(rowMeta, "git", null, null, new Variables(), null);
- assertEquals(18, rowMeta.size());
+ assertEquals(GitInputFields.fieldCount(GitResourceType.COMMITS, false),
rowMeta.size());
assertEquals(null, rowMeta.searchValueMeta("raw_json"));
}
+ @Test
+ void eachResourceTypeGetsItsOwnFieldLayout() throws Exception {
+ // A commit has idents and a merge flag but no labels, assignees or
branches; an issue is the
+ // other way round. A shared row would have to carry all of it and leave
most of it empty.
+ meta.setResourceType(GitResourceType.COMMITS.name());
+ IRowMeta commits = new RowMeta();
+ meta.getFields(commits, "git", null, null, new Variables(), null);
+
+ assertNotNull(commits.searchValueMeta("author_email"));
+ assertNotNull(commits.searchValueMeta("committer_email"));
+ assertNotNull(commits.searchValueMeta("is_merge"));
+ assertEquals(
+ IValueMeta.TYPE_BOOLEAN,
+ commits.searchValueMeta("is_merge").getType(),
+ "is_merge is a real Boolean, not a Y/N string");
+ assertNull(commits.searchValueMeta("labels"));
+ assertNull(commits.searchValueMeta("assignees"));
+ assertNull(commits.searchValueMeta("source_branch"));
+ assertNull(commits.searchValueMeta("closed_at"));
+
+ meta.setResourceType(GitResourceType.PULL_REQUESTS.name());
+ IRowMeta pullRequests = new RowMeta();
+ meta.getFields(pullRequests, "git", null, null, new Variables(), null);
+
+ assertNotNull(pullRequests.searchValueMeta("labels"));
+ assertNotNull(pullRequests.searchValueMeta("assignees"));
+ assertNotNull(pullRequests.searchValueMeta("merged_at"));
+ assertEquals(
+ IValueMeta.TYPE_BOOLEAN,
+ pullRequests.searchValueMeta("merged").getType(),
+ "merged is a real Boolean, not a Y/N string");
+ assertNotNull(pullRequests.searchValueMeta("source_branch"));
+ assertNull(pullRequests.searchValueMeta("committer"));
+ assertNull(pullRequests.searchValueMeta("is_merge"));
+
+ // Issues share the people and label fields with pull requests but never
the merge fields.
+ meta.setResourceType(GitResourceType.ISSUES.name());
+ IRowMeta issues = new RowMeta();
+ meta.getFields(issues, "git", null, null, new Variables(), null);
+
+ assertNotNull(issues.searchValueMeta("labels"));
+ assertNull(issues.searchValueMeta("merged"));
+ assertNull(issues.searchValueMeta("merged_at"));
+ }
+
+ @Test
+ void rawJsonIsAlwaysTheLastFieldWhateverTheType() throws Exception {
+ for (GitResourceType type : GitResourceType.values()) {
+ meta.setResourceType(type.name());
+ IRowMeta rowMeta = new RowMeta();
+ meta.getFields(rowMeta, "git", null, null, new Variables(), null);
+
+ assertEquals(
+ "raw_json",
+ rowMeta.getValueMeta(rowMeta.size() - 1).getName(),
+ "raw_json should be last for " + type);
+ }
+ }
+
@Test
void checkReportsMissingRemoteSettings() {
meta.setSource(GitInputSource.REMOTE.name());
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputTest.java
index 174dd7951b..a7d0aedd51 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/pipeline/transforms/gitinput/GitInputTest.java
@@ -35,6 +35,7 @@ import org.apache.hop.core.exception.HopException;
import org.apache.hop.core.row.IRowMeta;
import org.apache.hop.core.variables.Variables;
import org.apache.hop.git.provider.GitInputFields;
+import org.apache.hop.git.provider.GitResourceType;
import org.apache.hop.pipeline.transforms.mock.TransformMockHelper;
import org.eclipse.jgit.api.Git;
import org.junit.jupiter.api.AfterEach;
@@ -236,7 +237,8 @@ class GitInputTest {
// drain
}
- assertEquals(GitInputFields.FIELD_NAMES.length - 1,
data.outputRowMeta.size());
+ assertEquals(
+ GitInputFields.fieldCount(GitResourceType.COMMITS, false),
data.outputRowMeta.size());
assertEquals(-1, data.outputRowMeta.indexOfValue("raw_json"));
}
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/BitbucketResourceClientWireMockTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/BitbucketResourceClientWireMockTest.java
index a554c5a724..f885d8d5e6 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/BitbucketResourceClientWireMockTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/BitbucketResourceClientWireMockTest.java
@@ -108,7 +108,8 @@ class BitbucketResourceClientWireMockTest {
envelopeWithNext(
"{\"hash\":\"deadbeef\",\"message\":\"a
commit\\n\\nwith a body\","
+
"\"date\":\"2026-05-01T12:00:00.000000+00:00\","
- +
"\"author\":{\"user\":{\"display_name\":\"Ada Lovelace\"}},"
+ + "\"author\":{\"raw\":\"Ada Lovelace
<[email protected]>\","
+ + "\"user\":{\"display_name\":\"Ada
Lovelace\"}},"
+
"\"links\":{\"html\":{\"href\":\"https://bitbucket.org/team/repo/commits/deadbeef\"}}}";))));
wireMock.stubFor(
get(urlPathEqualTo("/repositories/team/repo/commits"))
@@ -124,10 +125,12 @@ class BitbucketResourceClientWireMockTest {
assertEquals("deadbeef", record.getSha());
assertEquals("a commit", record.getTitle());
assertEquals("Ada Lovelace", record.getAuthor());
+ assertEquals("[email protected]", record.getAuthorEmail());
// Bitbucket's microsecond precision must still parse to a Date.
- Object[] row = record.toRow(true);
- int createdAt = List.of(GitInputFields.FIELD_NAMES).indexOf("created_at");
+ Object[] row = record.toRow(GitResourceType.COMMITS, true);
+ int createdAt =
+ List.of(GitInputFields.fieldNames(GitResourceType.COMMITS,
true)).indexOf("created_at");
assertEquals(Date.from(Instant.parse("2026-05-01T12:00:00Z")),
row[createdAt]);
}
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientTest.java
index ab5aba1d3e..0eee3409ea 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientTest.java
@@ -17,7 +17,8 @@
package org.apache.hop.git.provider;
-import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
import org.json.simple.JSONObject;
import org.junit.jupiter.api.Test;
@@ -30,7 +31,7 @@ class GitHubResourceClientTest {
listResponse.put("merged_at", "2026-05-01T12:00:00Z");
listResponse.put("state", "closed");
- assertEquals("Y", GitHubResourceClient.mapGithubMerged(listResponse));
+ assertTrue(GitHubResourceClient.mapGithubMerged(listResponse));
}
@Test
@@ -41,8 +42,8 @@ class GitHubResourceClientTest {
notMerged.put("merged", Boolean.FALSE);
notMerged.put("merged_at", "2026-05-01T12:00:00Z");
- assertEquals("Y", GitHubResourceClient.mapGithubMerged(merged));
- assertEquals("N", GitHubResourceClient.mapGithubMerged(notMerged));
+ assertTrue(GitHubResourceClient.mapGithubMerged(merged));
+ assertFalse(GitHubResourceClient.mapGithubMerged(notMerged));
}
@Test
@@ -51,6 +52,6 @@ class GitHubResourceClientTest {
closedUnmerged.put("state", "closed");
closedUnmerged.put("closed_at", "2026-05-01T12:00:00Z");
- assertEquals("N", GitHubResourceClient.mapGithubMerged(closedUnmerged));
+ assertFalse(GitHubResourceClient.mapGithubMerged(closedUnmerged));
}
}
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientWireMockTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientWireMockTest.java
index 61b3445882..22defb4443 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientWireMockTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitHubResourceClientWireMockTest.java
@@ -24,7 +24,6 @@ import static
com.github.tomakehurst.wiremock.client.WireMock.getRequestedFor;
import static com.github.tomakehurst.wiremock.client.WireMock.urlPathEqualTo;
import static org.junit.jupiter.api.Assertions.assertEquals;
import static org.junit.jupiter.api.Assertions.assertNotNull;
-import static org.junit.jupiter.api.Assertions.assertNull;
import static org.junit.jupiter.api.Assertions.assertThrows;
import static org.junit.jupiter.api.Assertions.assertTrue;
@@ -109,7 +108,7 @@ class GitHubResourceClientWireMockTest {
+ message
+ "\",\"author\":{\"name\":\""
+ author
- + "\",\"date\":\""
+ + "\",\"email\":\"[email protected]\",\"date\":\""
+ date
+ "\"}}}";
}
@@ -129,6 +128,7 @@ class GitHubResourceClientWireMockTest {
assertEquals("a1", record.getSha());
assertEquals("first", record.getTitle());
assertEquals("Ada", record.getAuthor());
+ assertEquals("[email protected]", record.getAuthorEmail());
}
@Test
@@ -139,12 +139,11 @@ class GitHubResourceClientWireMockTest {
List<GitResourceRecord> records =
drain(client.openReader(GitResourceType.COMMITS, "apache", "hop",
options(1, 10)));
- Object[] row = records.get(0).toRow(true);
+ Object[] row = records.get(0).toRow(GitResourceType.COMMITS, true);
+ List<String> fieldNames =
List.of(GitInputFields.fieldNames(GitResourceType.COMMITS, true));
assertEquals(
- Date.from(Instant.parse("2026-05-01T12:00:00Z")),
- row[GitInputFields.FIELD_NAMES.length - 10]);
- assertEquals(Date.from(Instant.parse("2026-05-01T12:00:00Z")), row[9]);
- assertNull(row[10], "updated_at is absent for a commit");
+ Date.from(Instant.parse("2026-05-01T12:00:00Z")),
row[fieldNames.indexOf("created_at")]);
+ assertEquals(-1, fieldNames.indexOf("updated_at"), "a commit layout has no
updated_at column");
}
@Test
@@ -229,6 +228,79 @@ class GitHubResourceClientWireMockTest {
assertTrue(reader.getTruncationNote().contains("1,000 items"));
}
+ @Test
+ void pullRequestRowsCarryLabelsAssigneesAndMergeTime() throws Exception {
+ wireMock.stubFor(
+ get(urlPathEqualTo("/repos/apache/hop/pulls"))
+ .withQueryParam("page", equalTo("1"))
+ .willReturn(
+ aResponse()
+ .withStatus(200)
+ .withBody(
+ "[{\"node_id\":\"n7\",\"number\":7,\"title\":\"a pull
request\","
+ +
"\"state\":\"closed\",\"user\":{\"login\":\"ada\"},"
+ +
"\"labels\":[{\"name\":\"bug\"},{\"name\":\"needs review\"}],"
+ +
"\"assignees\":[{\"login\":\"grace\"},{\"login\":\"alan\"}],"
+ + "\"merged_at\":\"2026-05-02T09:30:00Z\","
+ +
"\"head\":{\"ref\":\"feature\"},\"base\":{\"ref\":\"main\"}}]")));
+ wireMock.stubFor(
+ get(urlPathEqualTo("/repos/apache/hop/pulls"))
+ .withQueryParam("page", equalTo("2"))
+ .willReturn(aResponse().withStatus(200).withBody("[]")));
+
+ List<GitResourceRecord> records =
+ drain(client.openReader(GitResourceType.PULL_REQUESTS, "apache",
"hop", options(1, 10)));
+
+ assertEquals(1, records.size());
+ GitResourceRecord record = records.get(0);
+ assertEquals("bug, needs review", record.getLabels());
+ assertEquals("grace, alan", record.getAssignees());
+ assertEquals("ada", record.getAuthorLogin());
+ assertTrue(record.getMerged());
+
+ List<String> fieldNames =
+ List.of(GitInputFields.fieldNames(GitResourceType.PULL_REQUESTS,
true));
+ Object[] row = record.toRow(GitResourceType.PULL_REQUESTS, true);
+ assertEquals(
+ Date.from(Instant.parse("2026-05-02T09:30:00Z")),
row[fieldNames.indexOf("merged_at")]);
+ assertEquals(-1, fieldNames.indexOf("committer"), "a pull request layout
has no committer");
+ }
+
+ @Test
+ void commitRowsSeparateTheGitIdentFromTheGithubAccount() throws Exception {
+ // commit.author is what the commit was signed with; the top-level author
is the matched
+ // account, and the two are routinely different people or absent
altogether.
+ wireMock.stubFor(
+ get(urlPathEqualTo("/repos/apache/hop/commits"))
+ .withQueryParam("page", equalTo("1"))
+ .willReturn(
+ aResponse()
+ .withStatus(200)
+ .withBody(
+ "[{\"sha\":\"a1\",\"author\":{\"login\":\"ada-gh\"},"
+ +
"\"parents\":[{\"sha\":\"p1\"},{\"sha\":\"p2\"}],"
+ + "\"commit\":{\"message\":\"a merge\","
+ +
"\"author\":{\"name\":\"Ada\",\"email\":\"[email protected]\","
+ + "\"date\":\"2026-05-01T12:00:00Z\"},"
+ + "\"committer\":{\"name\":\"GitHub\","
+ + "\"email\":\"[email protected]\"}}}]")));
+ wireMock.stubFor(
+ get(urlPathEqualTo("/repos/apache/hop/commits"))
+ .withQueryParam("page", equalTo("2"))
+ .willReturn(aResponse().withStatus(200).withBody("[]")));
+
+ List<GitResourceRecord> records =
+ drain(client.openReader(GitResourceType.COMMITS, "apache", "hop",
options(1, 10)));
+
+ GitResourceRecord record = records.get(0);
+ assertEquals("Ada", record.getAuthor());
+ assertEquals("[email protected]", record.getAuthorEmail());
+ assertEquals("ada-gh", record.getAuthorLogin());
+ assertEquals("GitHub", record.getCommitter());
+ assertEquals("[email protected]", record.getCommitterEmail());
+ assertTrue(record.getIsMerge(), "two parents is a merge");
+ }
+
@Test
void pullRequestsAreExcludedFromTheIssuesFeed() throws Exception {
wireMock.stubFor(
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitJsonListsTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitJsonListsTest.java
new file mode 100644
index 0000000000..e049d6a630
--- /dev/null
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitJsonListsTest.java
@@ -0,0 +1,82 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hop.git.provider;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+import org.json.simple.JSONObject;
+import org.json.simple.parser.JSONParser;
+import org.junit.jupiter.api.Test;
+
+class GitJsonListsTest {
+
+ private static JSONObject parse(String json) throws Exception {
+ return (JSONObject) new JSONParser().parse(json);
+ }
+
+ @Test
+ void joinsGithubStyleObjectArrays() throws Exception {
+ JSONObject json = parse("{\"labels\":[{\"name\":\"bug\"},{\"name\":\"needs
review\"}]}");
+ assertEquals("bug, needs review", GitJsonLists.names(json, "labels",
"name"));
+ }
+
+ @Test
+ void joinsGitlabStylePlainStringArrays() throws Exception {
+ JSONObject json = parse("{\"labels\":[\"bug\",\"needs review\"]}");
+ assertEquals("bug, needs review", GitJsonLists.names(json, "labels",
"name"));
+ }
+
+ @Test
+ void readsASingleValuedFieldThatIsNotAnArray() throws Exception {
+ // Bitbucket carries one assignee as an object rather than a list of them.
+ JSONObject json = parse("{\"assignee\":{\"display_name\":\"Ada
Lovelace\"}}");
+ assertEquals("Ada Lovelace", GitJsonLists.names(json, "assignee",
"display_name"));
+ }
+
+ @Test
+ void skipsBlankAndUnusableEntriesRatherThanFailingTheRow() throws Exception {
+ JSONObject json =
parse("{\"labels\":[{\"name\":\"bug\"},{\"other\":\"x\"},null,42,\"\"]}");
+ assertEquals("bug", GitJsonLists.names(json, "labels", "name"));
+ }
+
+ @Test
+ void anAbsentOrEmptyListIsAnEmptyValue() throws Exception {
+ assertEquals("", GitJsonLists.names(parse("{}"), "labels", "name"));
+ assertEquals("", GitJsonLists.names(parse("{\"labels\":[]}"), "labels",
"name"));
+ assertEquals("", GitJsonLists.names(null, "labels", "name"));
+ }
+
+ @Test
+ void moreThanOneParentIsAMerge() throws Exception {
+ assertTrue(
+ GitJsonLists.mergeFlag(
+ parse("{\"parents\":[{\"sha\":\"a\"},{\"sha\":\"b\"}]}"),
"parents"));
+
assertFalse(GitJsonLists.mergeFlag(parse("{\"parents\":[{\"sha\":\"a\"}]}"),
"parents"));
+ assertFalse(GitJsonLists.mergeFlag(parse("{\"parents\":[]}"), "parents"));
+ }
+
+ @Test
+ void anAbsentParentListIsNotReportedAsAMerge() throws Exception {
+ // The payload does not say; guessing "merge" here would silently corrupt
commit counts.
+ assertFalse(GitJsonLists.mergeFlag(parse("{}"), "parents"));
+
assertFalse(GitJsonLists.mergeFlag(parse("{\"parent_ids\":[\"a\",\"b\"]}"),
"parents"));
+ assertTrue(GitJsonLists.mergeFlag(parse("{\"parent_ids\":[\"a\",\"b\"]}"),
"parent_ids"));
+ }
+}
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitLabResourceClientWireMockTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitLabResourceClientWireMockTest.java
index 0034d09760..8dff2c0de4 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitLabResourceClientWireMockTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GitLabResourceClientWireMockTest.java
@@ -92,7 +92,7 @@ class GitLabResourceClientWireMockTest {
.withBody(
"[{\"id\":\"abc123\",\"title\":\"fix things\","
+ "\"message\":\"fix things\\n\\nwith detail\","
- + "\"author_name\":\"Ada\","
+ +
"\"author_name\":\"Ada\",\"author_email\":\"[email protected]\","
+ "\"created_at\":\"2026-05-01T12:00:00.000Z\","
+
"\"web_url\":\"https://gitlab.com/apache/hop/-/commit/abc123\"}]";)));
wireMock.stubFor(
@@ -108,10 +108,12 @@ class GitLabResourceClientWireMockTest {
assertEquals("gitlab", record.getProvider());
assertEquals("abc123", record.getSha());
assertEquals("Ada", record.getAuthor());
+ assertEquals("[email protected]", record.getAuthorEmail());
// GitLab's millisecond form still has to reach the row as a real Date.
- Object[] row = record.toRow(true);
- int createdAt = List.of(GitInputFields.FIELD_NAMES).indexOf("created_at");
+ Object[] row = record.toRow(GitResourceType.COMMITS, true);
+ int createdAt =
+ List.of(GitInputFields.fieldNames(GitResourceType.COMMITS,
true)).indexOf("created_at");
assertEquals(Date.from(Instant.parse("2026-05-01T12:00:00Z")),
row[createdAt]);
}
@@ -142,7 +144,7 @@ class GitLabResourceClientWireMockTest {
assertEquals(42L, record.getNumber(), "GitLab numbers rows by iid, not
id");
assertEquals("feature", record.getSourceBranch());
assertEquals("main", record.getTargetBranch());
- assertEquals("Y", record.getMerged());
+ assertTrue(record.getMerged());
}
@Test
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GiteaResourceClientWireMockTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GiteaResourceClientWireMockTest.java
index 4e00295612..702136b08f 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GiteaResourceClientWireMockTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/GiteaResourceClientWireMockTest.java
@@ -91,7 +91,8 @@ class GiteaResourceClientWireMockTest {
.withBody(
"[{\"sha\":\"cafe01\",\"html_url\":\"https://codeberg.org/bart/putki/commit/cafe01\",";
+ "\"commit\":{\"message\":\"a commit\","
- +
"\"author\":{\"name\":\"Ada\",\"date\":\"2026-05-01T12:00:00Z\"}}}]")));
+ +
"\"author\":{\"name\":\"Ada\",\"email\":\"[email protected]\","
+ + "\"date\":\"2026-05-01T12:00:00Z\"}}}]")));
wireMock.stubFor(
get(urlPathEqualTo("/repos/bart/putki/commits"))
.withQueryParam("page", equalTo("2"))
@@ -104,6 +105,7 @@ class GiteaResourceClientWireMockTest {
assertEquals("gitea", records.get(0).getProvider());
assertEquals("cafe01", records.get(0).getSha());
assertEquals("Ada", records.get(0).getAuthor());
+ assertEquals("[email protected]", records.get(0).getAuthorEmail());
}
@Test
diff --git
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/LocalGitResourceClientTest.java
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/LocalGitResourceClientTest.java
index 82141b2ec2..cbe845b345 100644
---
a/plugins/misc/git/src/test/java/org/apache/hop/git/provider/LocalGitResourceClientTest.java
+++
b/plugins/misc/git/src/test/java/org/apache/hop/git/provider/LocalGitResourceClientTest.java
@@ -177,11 +177,97 @@ class LocalGitResourceClientTest {
List<GitResourceRecord> records = readAll(GitResourceType.COMMITS);
- int createdAt = List.of(GitInputFields.FIELD_NAMES).indexOf("created_at");
- Object value = records.get(0).toRow(true)[createdAt];
+ int createdAt =
+ List.of(GitInputFields.fieldNames(GitResourceType.COMMITS,
true)).indexOf("created_at");
+ Object value = records.get(0).toRow(GitResourceType.COMMITS,
true)[createdAt];
assertInstanceOf(Date.class, value);
}
+ @Test
+ void commitRowsCarryTheAuthorEmail() throws Exception {
+ initRepoWithTwoCommits();
+
+ List<GitResourceRecord> records = readAll(GitResourceType.COMMITS);
+
+ int authorEmail =
+ List.of(GitInputFields.fieldNames(GitResourceType.COMMITS,
true)).indexOf("author_email");
+ for (GitResourceRecord record : records) {
+ assertEquals("tester", record.getAuthor());
+ assertEquals("[email protected]", record.getAuthorEmail());
+ assertEquals("[email protected]", record.toRow(GitResourceType.COMMITS,
true)[authorEmail]);
+ assertTrue(
+
record.getRawJson().contains("\"author_email\":\"[email protected]\""),
+ "raw_json should carry the e-mail too: " + record.getRawJson());
+ }
+ }
+
+ @Test
+ void commitFileRowsCarryTheAuthorEmail() throws Exception {
+ initRepoWithTwoCommits();
+
+ List<GitResourceRecord> records = readAll(GitResourceType.COMMIT_FILES);
+
+ assertFalse(records.isEmpty());
+ for (GitResourceRecord record : records) {
+ assertEquals("[email protected]", record.getAuthorEmail());
+ }
+ }
+
+ @Test
+ void commitRowsSeparateAuthorFromCommitterAndFlagMerges() throws Exception {
+ // A rebase, a squash or a web merge rewrites the committer but keeps the
original author.
+ writeFile("hello.txt", "v1");
+ try (Git git = Git.init().setDirectory(repoDir).call()) {
+ git.add().addFilepattern(".").call();
+ git.commit()
+ .setMessage("initial")
+ .setAuthor("Ada", "[email protected]")
+ .setCommitter("Release Bot", "[email protected]")
+ .call();
+ }
+
+ List<GitResourceRecord> records = readAll(GitResourceType.COMMITS);
+
+ GitResourceRecord record = records.get(0);
+ assertEquals("Ada", record.getAuthor());
+ assertEquals("[email protected]", record.getAuthorEmail());
+ assertEquals("Release Bot", record.getCommitter());
+ assertEquals("[email protected]", record.getCommitterEmail());
+ assertFalse(record.getIsMerge(), "a root commit has one parent at most");
+ }
+
+ @Test
+ void aMergeCommitIsFlaggedAsOne() throws Exception {
+ try (Git git = Git.init().setDirectory(repoDir).call()) {
+ writeFile("hello.txt", "v1");
+ git.add().addFilepattern(".").call();
+ git.commit().setMessage("initial").setAuthor("tester",
"[email protected]").call();
+ String main = git.getRepository().getBranch();
+
+ git.checkout().setCreateBranch(true).setName("side").call();
+ writeFile("side.txt", "side");
+ git.add().addFilepattern(".").call();
+ git.commit().setMessage("side work").setAuthor("tester",
"[email protected]").call();
+
+ git.checkout().setName(main).call();
+ writeFile("hello.txt", "v2");
+ git.add().addFilepattern(".").call();
+ git.commit().setMessage("main work").setAuthor("tester",
"[email protected]").call();
+
+ git.merge()
+ .include(git.getRepository().resolve("side"))
+ .setCommit(true)
+ .setMessage("merge side")
+ .call();
+ }
+
+ List<GitResourceRecord> records = readAll(GitResourceType.COMMITS);
+
+ long merges = records.stream().filter(r ->
Boolean.TRUE.equals(r.getIsMerge())).count();
+ assertEquals(1, merges, "exactly the merge commit should be flagged");
+ assertTrue(records.get(0).getIsMerge(), "the merge is the most recent
commit");
+ }
+
@Test
void closingAReaderEarlyReleasesTheRepository() throws Exception {
initRepoWithTwoCommits();
