(celeborn-website) 02/08: .github/bin/build_docs.sh
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/celeborn-website.git
commit d24ad6475e5d65528bb11f4b49aa688f3bcc6e96
Author: Celeborn
AuthorDate: Sat Jun 8 01:08:52 2024 +
.github/bin/build_docs.sh
---
.github/bin/build_docs.sh | 59 +++
1 file changed, 59 insertions(+)
diff --git a/.github/bin/build_docs.sh b/.github/bin/build_docs.sh
new file mode 100755
index 0..2e8d8f02e
--- /dev/null
+++ b/.github/bin/build_docs.sh
@@ -0,0 +1,59 @@
+#!/usr/bin/env bash
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+set -o pipefail
+set -e
+set -x
+
+function exit_with_usage {
+ echo "./github/bin/build_docs.sh "
+ exit 1
+}
+
+if [ $# -ne 2 ]; then
+ exit_with_usage
+fi
+
+REF_VERSION_NAME="$1"
+DOC_LINK_PATH="$2"
+
+TAR_NAME=${REF_VERSION_NAME##*/}
+
+if [[ "$REF_VERSION_NAME" == *"-tm"* ]] ;then
+DIR_VERSION_NAME=$(echo "$TAR_NAME" | sed -r "s/v*(.*)\.tar\.gz/\1/g")
+if [[ "$REF_VERSION_NAME" == *"-incubating"* ]] ;then
+ TAR_DIR_NAME=incubator-celeborn-$DOC_LINK_PATH
+else
+ TAR_DIR_NAME=celeborn-$DOC_LINK_PATH
+fi
+wget
"https://github.com/apache/celeborn/releases/download/${REF_VERSION_NAME};
+ else
+DIR_VERSION_NAME=$(echo "$TAR_NAME" | sed -r "s/v*(.*)\.tar\.gz/\1/g")
+TAR_DIR_NAME=celeborn-$DIR_VERSION_NAME
+wget "https://github.com/apache/celeborn/archive/refs/${REF_VERSION_NAME};
+fi
+
+tar -xzf $TAR_NAME
+cd $TAR_DIR_NAME
+mkdocs build
+cd ..
+mkdir -p docs
+if [ -d docs/$DOC_LINK_PATH ]; then rm -r docs/$DOC_LINK_PATH; fi
+mv $TAR_DIR_NAME/site docs/$DOC_LINK_PATH
+git add .
+git commit -m "docs/$DOC_LINK_PATH"
(celeborn-website) 08/08: .asf.yaml
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/celeborn-website.git commit 3d31d55fcfe1715391f503f91471a3742b9e0a7f Author: Celeborn AuthorDate: Sat Jun 8 01:09:04 2024 + .asf.yaml --- .asf.yaml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/.asf.yaml b/.asf.yaml new file mode 100644 index 0..07355508b --- /dev/null +++ b/.asf.yaml @@ -0,0 +1,2 @@ +publish: + whoami: asf-site
(celeborn-website) 01/08: gitignore
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/celeborn-website.git commit 145b46e9d6bd399e25fc9e4eb6eca256cd40e739 Author: Celeborn AuthorDate: Sat Jun 8 01:08:52 2024 + gitignore --- .gitignore | 22 ++ 1 file changed, 22 insertions(+) diff --git a/.gitignore b/.gitignore new file mode 100644 index 0..933a50729 --- /dev/null +++ b/.gitignore @@ -0,0 +1,22 @@ +*#*# +*.#* +*.iml +*.ipr +*.iws +*.pyc +*.pyo +*.swp +*~ +.DS_Store +.cache +.classpath +.idea/ +.idea_modules/ +.project +.pydevproject +.python-version +.settings +/site/ +celeborn-*/ +*.zip +*.tar.gz
(nuttx-website) branch asf-site updated: Publishing web: c0bb376dae649378f137872f2791869ca6c6876c docs: c885ba7080f843be0d2881886ff0f48412bc70cc
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/nuttx-website.git
The following commit(s) were added to refs/heads/asf-site by this push:
new d9940d2b1 Publishing web: c0bb376dae649378f137872f2791869ca6c6876c
docs: c885ba7080f843be0d2881886ff0f48412bc70cc
d9940d2b1 is described below
commit d9940d2b1a76de9266c4b3b7ecfc93e9376ea6a4
Author: chao an
AuthorDate: Sat Jun 8 00:18:57 2024 +
Publishing web: c0bb376dae649378f137872f2791869ca6c6876c docs:
c885ba7080f843be0d2881886ff0f48412bc70cc
---
content/docs/10.0.0/index.html | 2 +-
content/docs/10.0.0/searchindex.js | 2 +-
content/docs/10.0.1/index.html | 2 +-
content/docs/10.0.1/searchindex.js | 2 +-
content/docs/10.1.0/index.html | 2 +-
content/docs/10.1.0/searchindex.js | 2 +-
content/docs/10.2.0/index.html | 2 +-
content/docs/10.2.0/searchindex.js | 2 +-
content/docs/10.3.0/index.html | 2 +-
content/docs/10.3.0/searchindex.js | 2 +-
content/docs/11.0.0/index.html | 2 +-
content/docs/11.0.0/searchindex.js | 2 +-
content/docs/12.0.0/index.html | 2 +-
content/docs/12.0.0/searchindex.js | 2 +-
content/docs/12.1.0/index.html | 2 +-
content/docs/12.1.0/searchindex.js | 2 +-
content/docs/12.2.0/index.html | 2 +-
content/docs/12.2.0/searchindex.js | 2 +-
content/docs/12.2.1/index.html | 2 +-
content/docs/12.2.1/searchindex.js | 2 +-
content/docs/12.3.0/index.html | 2 +-
content/docs/12.3.0/searchindex.js | 2 +-
content/docs/12.4.0/index.html | 2 +-
content/docs/12.4.0/searchindex.js | 2 +-
content/docs/12.5.0/index.html | 2 +-
content/docs/12.5.0/searchindex.js | 2 +-
content/docs/12.5.1/index.html | 2 +-
content/docs/12.5.1/searchindex.js | 2 +-
content/docs/latest/index.html | 2 +-
content/docs/latest/searchindex.js | 2 +-
content/feed.xml | 4 ++--
31 files changed, 32 insertions(+), 32 deletions(-)
diff --git a/content/docs/10.0.0/index.html b/content/docs/10.0.0/index.html
index 84ae973bd..54371532b 100644
--- a/content/docs/10.0.0/index.html
+++ b/content/docs/10.0.0/index.html
@@ -131,7 +131,7 @@ by following these
NuttX Documentation
NuttX is a real-time operating system (RTOS) with an emphasis on standards
compliance and small footprint. Scalable from 8-bit to 32-bit microcontroller
environments, the primary governing standards in NuttX are Posix and ANSI
standards. Additional standard APIs from Unix and other common RTOS’s (such as
VxWorks) are adopted for functionality not available under these standards, or
for functionality that is not appropriate for deeply-embedded environments
(such as fork()).
-Last Updated: 07 June 24 at 00:10
+Last Updated: 08 June 24 at 00:11
Table of
Contents
diff --git a/content/docs/10.0.0/searchindex.js
b/content/docs/10.0.0/searchindex.js
index 03d3020c1..9fa76ca77 100644
--- a/content/docs/10.0.0/searchindex.js
+++ b/content/docs/10.0.0/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"docnames": ["applications/index", "boards/index",
"components/binfmt", "components/drivers/block/index",
"components/drivers/character/analog", "components/drivers/character/can",
"components/drivers/character/index", "components/drivers/character/keypad",
"components/drivers/character/pwm", "components/drivers/character/quadrature",
"components/drivers/character/rtc", "components/drivers/character/serial",
"components/drivers/character/timer", "components/drivers/chara [...]
\ No newline at end of file
+Search.setIndex({"docnames": ["applications/index", "boards/index",
"components/binfmt", "components/drivers/block/index",
"components/drivers/character/analog", "components/drivers/character/can",
"components/drivers/character/index", "components/drivers/character/keypad",
"components/drivers/character/pwm", "components/drivers/character/quadrature",
"components/drivers/character/rtc", "components/drivers/character/serial",
"components/drivers/character/timer", "components/drivers/chara [...]
\ No newline at end of file
diff --git a/content/docs/10.0.1/index.html b/content/docs/10.0.1/index.html
index 2b4023c5a..4a4db95d1 100644
--- a/content/docs/10.0.1/index.html
+++ b/content/docs/10.0.1/index.html
@@ -159,7 +159,7 @@ by following these
NuttX Documentation
NuttX is a real-time operating system (RTOS) with an emphasis on standards
compliance and small footprint. Scalable from 8-bit to 32-bit microcontroller
environments, the primary governing standards in NuttX are Posix and ANSI
standards. Additional standard APIs from Unix and other common RTOS’s (such as
VxWorks) a
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a9517b5d5e8 Updating config from bot
a9517b5d5e8 is described below
commit a9517b5d5e8d3871af697f3f046885175135feb7
Author: github-actions
AuthorDate: Sat Jun 8 00:56:34 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31550.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31550.json
b/scripts/ci/pr-bot/state/pr-state/pr-31550.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31550.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(camel-k) branch release-2.3.x updated: chore: nightly automatic updates
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch release-2.3.x in repository https://gitbox.apache.org/repos/asf/camel-k.git The following commit(s) were added to refs/heads/release-2.3.x by this push: new f746ce3ef chore: nightly automatic updates f746ce3ef is described below commit f746ce3efe268d3b9cc61673e40c2837b20ed721 Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Sat Jun 8 00:31:33 2024 + chore: nightly automatic updates --- CHANGELOG.md | 402 ++- 1 file changed, 207 insertions(+), 195 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index df4ac126e..7ddf7bc58 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,7 +2,19 @@ ## [Unreleased](https://github.com/apache/camel-k/tree/HEAD) -[Full Changelog](https://github.com/apache/camel-k/compare/2.3.3-nightly...HEAD) +[Full Changelog](https://github.com/apache/camel-k/compare/v2.3.3...HEAD) + +**Closed issues:** + +- ImagePullBackOff Error for pod build strategy and s2i publish strategy [\#5595](https://github.com/apache/camel-k/issues/5595) + +## [v2.3.3](https://github.com/apache/camel-k/tree/v2.3.3) (2024-06-07) + +[Full Changelog](https://github.com/apache/camel-k/compare/camel-k-crds-2.3.3...v2.3.3) + +## [camel-k-crds-2.3.3](https://github.com/apache/camel-k/tree/camel-k-crds-2.3.3) (2024-06-07) + +[Full Changelog](https://github.com/apache/camel-k/compare/2.3.3-nightly...camel-k-crds-2.3.3) **Closed issues:** @@ -205,7 +217,7 @@ ## [camel-k-crds-2.3.1](https://github.com/apache/camel-k/tree/camel-k-crds-2.3.1) (2024-04-29) -[Full Changelog](https://github.com/apache/camel-k/compare/2.4.0-nightly...camel-k-crds-2.3.1) +[Full Changelog](https://github.com/apache/camel-k/compare/2.3.1-nightly...camel-k-crds-2.3.1) **Closed issues:** @@ -251,13 +263,13 @@ - Garbage-collect unused kits [\#254](https://github.com/apache/camel-k/issues/254) - Enable services to be associated with a version [\#182](https://github.com/apache/camel-k/issues/182) -## [2.4.0-nightly](https://github.com/apache/camel-k/tree/2.4.0-nightly) (2024-04-08) +## [2.3.1-nightly](https://github.com/apache/camel-k/tree/2.3.1-nightly) (2024-04-08) -[Full Changelog](https://github.com/apache/camel-k/compare/2.3.1-nightly...2.4.0-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/2.4.0-nightly...2.3.1-nightly) -## [2.3.1-nightly](https://github.com/apache/camel-k/tree/2.3.1-nightly) (2024-04-08) +## [2.4.0-nightly](https://github.com/apache/camel-k/tree/2.4.0-nightly) (2024-04-08) -[Full Changelog](https://github.com/apache/camel-k/compare/v2.3.0...2.3.1-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/v2.3.0...2.4.0-nightly) **Closed issues:** @@ -1027,7 +1039,7 @@ ## [2.0.1-nightly](https://github.com/apache/camel-k/tree/2.0.1-nightly) (2023-07-24) -[Full Changelog](https://github.com/apache/camel-k/compare/pkg/client/camel/v2.0.0...2.0.1-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/pkg/kamelet/repository/v2.0.0...2.0.1-nightly) **Closed issues:** @@ -1101,6 +1113,10 @@ - fix\(build\): complete repo name for default baseImage [\#4480](https://github.com/apache/camel-k/pull/4480) ([gansheer](https://github.com/gansheer)) - fix\(e2e\): remove workaround pre 1.12.1 [\#4479](https://github.com/apache/camel-k/pull/4479) ([squakez](https://github.com/squakez)) +## [pkg/kamelet/repository/v2.0.0](https://github.com/apache/camel-k/tree/pkg/kamelet/repository/v2.0.0) (2023-07-17) + +[Full Changelog](https://github.com/apache/camel-k/compare/pkg/client/camel/v2.0.0...pkg/kamelet/repository/v2.0.0) + ## [pkg/client/camel/v2.0.0](https://github.com/apache/camel-k/tree/pkg/client/camel/v2.0.0) (2023-07-17) [Full Changelog](https://github.com/apache/camel-k/compare/pkg/apis/camel/v2.0.0...pkg/client/camel/v2.0.0) @@ -,11 +1127,7 @@ ## [v2.0.0](https://github.com/apache/camel-k/tree/v2.0.0) (2023-07-17) -[Full Changelog](https://github.com/apache/camel-k/compare/pkg/kamelet/repository/v2.0.0...v2.0.0) - -## [pkg/kamelet/repository/v2.0.0](https://github.com/apache/camel-k/tree/pkg/kamelet/repository/v2.0.0) (2023-07-17) - -[Full Changelog](https://github.com/apache/camel-k/compare/camel-k-crds-2.0.0...pkg/kamelet/repository/v2.0.0) +[Full Changelog](https://github.com/apache/camel-k/compare/camel-k-crds-2.0.0...v2.0.0) ## [camel-k-crds-2.0.0](https://github.com/apache/camel-k/tree/camel-k-crds-2.0.0) (2023-07-17) @@ -1169,7 +1181,7 @@ ## [1.12.2-nightly](https://github.com/apache/camel-k/tree/1.12.2-nightly) (2023-06-13) -[Full Changelog](https://github.com/apache/camel-k/compare/pkg/client/camel/v1.12.1...1.12.2-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/v1.12.1...1.12.2-nightly) **Closed issues:** @@ -1
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922695=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922695
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 08/Jun/24 00:01
Start Date: 08/Jun/24 00:01
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631785813
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -90,9 +70,28 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
-JobStatus jobStatus = dagNodeWithJobStatus.getRight().get();
-ExecutionStatus executionStatus = dagNode.getValue().getExecutionStatus();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // Usually reevaluate dag action is created by JobStatusMonitor when a
finished job status is available,
+ // but when reevaluate/resume/launch dag proc found multiple parallel
jobs to run next, it creates reevaluate
+ // dag actions for each of those parallel job and in this scenario there
is no job status available.
Review Comment:
I commented on this earlier...
is it accurate to say that the re-eval w/ no job status available occurs
when reevaluate itself handles the multi-job scenario? I don't believe so...
rather, isn't it only in the case of launch and resume handling multiple jobs?
Issue Time Tracking
---
Worklog Id: (was: 922695)
Time Spent: 5h 40m (was: 5.5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 5h 40m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922694=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922694
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 08/Jun/24 00:00
Start Date: 08/Jun/24 00:00
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631783796
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -82,41 +86,66 @@ public void tearDown() throws Exception {
}
@Test
- public void launchDag()
- throws IOException, InterruptedException, URISyntaxException {
-Dag dag = DagManagerTest.buildDag("1",
System.currentTimeMillis(), DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(),
-5, "user5",
ConfigFactory.empty().withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef("fg")));
+ public void launchDag() throws IOException, InterruptedException,
URISyntaxException, ExecutionException {
+String flowGroup = "fg";
+String flowName = "fn";
+String flowExecutionId = "12345";
+Dag dag = DagManagerTest.buildDag("1",
Long.parseLong(flowExecutionId),
+DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(), 5, "user5",
ConfigFactory.empty()
+.withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+.withValue(ConfigurationKeys.FLOW_NAME_KEY,
ConfigValueFactory.fromAnyRef(flowName)));
FlowCompilationValidationHelper flowCompilationValidationHelper =
mock(FlowCompilationValidationHelper.class);
doReturn(com.google.common.base.Optional.of(dag)).when(flowCompilationValidationHelper).createExecutionPlanIfValid(any());
+SpecProducer specProducer =
DagManagerUtils.getSpecProducer(dag.getNodes().get(0));
Review Comment:
looks like you're more or less doing this
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -90,9 +70,28 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
-JobStatus jobStatus = dagNodeWithJobStatus.getRight().get();
-ExecutionStatus executionStatus = dagNode.getValue().getExecutionStatus();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // Usually reevaluate dag action is created by JobStatusMonitor when a
finished job status is available,
+ // but when reevaluate/resume/launch dag proc found multiple parallel
jobs to run next, it creates reevaluate
+ // dag actions for each of those parallel job and in this scenario there
is no job status available.
Review Comment:
I commented on this earlier...
is it accurate to say that the re-eval w/ no job status available occurs
when reevaluate itself handles the multi-job scenario? isn't it only in the
case of launch and resume handling multiple jobs?
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProcTest.java:
##
@@ -206,4 +195,102 @@ public void testNoNextJobToRun() throws Exception {
Assert.assertEquals(Mockito.mockingDetails(this.dagManagementStateStore).getInvocations().stream()
.filter(a ->
a.getMethod().getName().equals("deleteDagAction")).count(), 1);
}
+
+ @Test
+ public void testCurrentJobToRun() throws Exception {
+String flowName = "fn3";
+Dag dag = DagManagerTest.buildDag("1", flowExecutionId,
DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(),
+2, "user5", ConfigFactory.empty()
+.withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+.withValue(ConfigurationKeys.FLOW_NAME_KEY,
ConfigValueFactory.fromAnyRef(flowName))
+.withValue(ConfigurationKeys.JOB_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+);
+List> startDagNodes = dag.getStartNodes();
+List> specProducers = getDagSpecProducers(dag);
+
+doReturn(Optional.of(dag)).when(dagManagementStateStore).getDag(any());
+doReturn(new ImmutablePair<>(Optional.of(startDagNodes.get(0)),
Optional.empty()))
+.when(dagManagementStateStore).getDagNodeWithJobStatus(any());
+
+ReevaluateDagProc
+reEvaluateDagProc = new ReevaluateDagProc(new ReevaluateDagTask(new
DagActionStore.DagAction(flowGroup, flowName,
+String.valueOf(flowExecutionId), "job0",
DagActionStore.DagActionType.REEVALUATE), null,
+dagManagementStateStore));
+reEvaluateDagProc.process(dagManagementStateStore);
+
+long addSpecCount = spe
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922693=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922693
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 23:48
Start Date: 07/Jun/24 23:48
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631782997
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -57,25 +55,7 @@ public ReevaluateDagProc(ReevaluateDagTask
reEvaluateDagTask) {
@Override
protected Pair>, Optional>
initialize(DagManagementStateStore dagManagementStateStore)
throws IOException {
-Pair>, Optional>
dagNodeWithJobStatus =
-dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
- // this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
- // already "reevaluated" and cleaned up.
- return ImmutablePair.of(Optional.empty(), Optional.empty());
-}
Review Comment:
ok, makes sense
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -57,25 +55,7 @@ public ReevaluateDagProc(ReevaluateDagTask
reEvaluateDagTask) {
@Override
protected Pair>, Optional>
initialize(DagManagementStateStore dagManagementStateStore)
throws IOException {
-Pair>, Optional>
dagNodeWithJobStatus =
-dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
- // this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
- // already "reevaluated" and cleaned up.
- return ImmutablePair.of(Optional.empty(), Optional.empty());
-}
Review Comment:
ok, makes sense... I agree
Issue Time Tracking
---
Worklog Id: (was: 922693)
Time Spent: 5h 20m (was: 5h 10m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 5h 20m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (KNOX-3040) Support multiple ways to verify JWT tokens
[ https://issues.apache.org/jira/browse/KNOX-3040?focusedWorklogId=922692=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922692 ] ASF GitHub Bot logged work on KNOX-3040: Author: ASF GitHub Bot Created on: 07/Jun/24 23:47 Start Date: 07/Jun/24 23:47 Worklog Time Spent: 10m Work Description: moresandeep merged PR #915: URL: https://github.com/apache/knox/pull/915 Issue Time Tracking --- Worklog Id: (was: 922692) Time Spent: 20m (was: 10m) > Support multiple ways to verify JWT tokens > -- > > Key: KNOX-3040 > URL: https://issues.apache.org/jira/browse/KNOX-3040 > Project: Apache Knox > Issue Type: Bug >Reporter: Sandeep More >Assignee: Sandeep More >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > > Currently we can only have one way to validate JWT token either > # Using JWKS endpoint > # Using PEM > # Using the signing-key > We should be able to support multiple verifications together. -- This message was sent by Atlassian Jira (v8.20.10#820010)
(camel-k) branch main updated: chore: nightly automatic updates
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch main in repository https://gitbox.apache.org/repos/asf/camel-k.git The following commit(s) were added to refs/heads/main by this push: new 6d89fd624 chore: nightly automatic updates 6d89fd624 is described below commit 6d89fd6246a3df50cdcce0f27ab54e2099337bce Author: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> AuthorDate: Fri Jun 7 23:46:48 2024 + chore: nightly automatic updates --- CHANGELOG.md | 406 ++- README.adoc | 2 +- badge.out| 2 +- coverage | 2 +- coverage.out | 6 +- 5 files changed, 216 insertions(+), 202 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index dca97650d..2ac8ba38c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -2,17 +2,31 @@ ## [Unreleased](https://github.com/apache/camel-k/tree/HEAD) -[Full Changelog](https://github.com/apache/camel-k/compare/2.3.3-nightly...HEAD) +[Full Changelog](https://github.com/apache/camel-k/compare/v2.3.3...HEAD) **Closed issues:** -- Join native checks into one [\#5526](https://github.com/apache/camel-k/issues/5526) +- ImagePullBackOff Error for pod build strategy and s2i publish strategy [\#5595](https://github.com/apache/camel-k/issues/5595) **Merged pull requests:** +- Use 'kubectl create' with prometheus bundle [\#5604](https://github.com/apache/camel-k/pull/5604) ([tdiesler](https://github.com/tdiesler)) - fix\(core\): Remove status.Image update post publish [\#5600](https://github.com/apache/camel-k/pull/5600) ([gansheer](https://github.com/gansheer)) +- test: Improve tests for maven proxy settings [\#5594](https://github.com/apache/camel-k/pull/5594) ([gansheer](https://github.com/gansheer)) - fix\(ci\): join low and high memory Quarkus native test [\#5593](https://github.com/apache/camel-k/pull/5593) ([squakez](https://github.com/squakez)) +## [v2.3.3](https://github.com/apache/camel-k/tree/v2.3.3) (2024-06-07) + +[Full Changelog](https://github.com/apache/camel-k/compare/camel-k-crds-2.3.3...v2.3.3) + +## [camel-k-crds-2.3.3](https://github.com/apache/camel-k/tree/camel-k-crds-2.3.3) (2024-06-07) + +[Full Changelog](https://github.com/apache/camel-k/compare/2.3.3-nightly...camel-k-crds-2.3.3) + +**Closed issues:** + +- Join native checks into one [\#5526](https://github.com/apache/camel-k/issues/5526) + ## [2.3.3-nightly](https://github.com/apache/camel-k/tree/2.3.3-nightly) (2024-06-05) [Full Changelog](https://github.com/apache/camel-k/compare/v2.3.2...2.3.3-nightly) @@ -210,7 +224,7 @@ ## [camel-k-crds-2.3.1](https://github.com/apache/camel-k/tree/camel-k-crds-2.3.1) (2024-04-29) -[Full Changelog](https://github.com/apache/camel-k/compare/2.4.0-nightly...camel-k-crds-2.3.1) +[Full Changelog](https://github.com/apache/camel-k/compare/2.3.1-nightly...camel-k-crds-2.3.1) **Closed issues:** @@ -256,13 +270,13 @@ - Garbage-collect unused kits [\#254](https://github.com/apache/camel-k/issues/254) - Enable services to be associated with a version [\#182](https://github.com/apache/camel-k/issues/182) -## [2.4.0-nightly](https://github.com/apache/camel-k/tree/2.4.0-nightly) (2024-04-08) +## [2.3.1-nightly](https://github.com/apache/camel-k/tree/2.3.1-nightly) (2024-04-08) -[Full Changelog](https://github.com/apache/camel-k/compare/2.3.1-nightly...2.4.0-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/2.4.0-nightly...2.3.1-nightly) -## [2.3.1-nightly](https://github.com/apache/camel-k/tree/2.3.1-nightly) (2024-04-08) +## [2.4.0-nightly](https://github.com/apache/camel-k/tree/2.4.0-nightly) (2024-04-08) -[Full Changelog](https://github.com/apache/camel-k/compare/v2.3.0...2.3.1-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/v2.3.0...2.4.0-nightly) **Closed issues:** @@ -1032,7 +1046,7 @@ ## [2.0.1-nightly](https://github.com/apache/camel-k/tree/2.0.1-nightly) (2023-07-24) -[Full Changelog](https://github.com/apache/camel-k/compare/pkg/client/camel/v2.0.0...2.0.1-nightly) +[Full Changelog](https://github.com/apache/camel-k/compare/pkg/kamelet/repository/v2.0.0...2.0.1-nightly) **Closed issues:** @@ -1106,6 +1120,10 @@ - fix\(build\): complete repo name for default baseImage [\#4480](https://github.com/apache/camel-k/pull/4480) ([gansheer](https://github.com/gansheer)) - fix\(e2e\): remove workaround pre 1.12.1 [\#4479](https://github.com/apache/camel-k/pull/4479) ([squakez](https://github.com/squakez)) +## [pkg/kamelet/repository/v2.0.0](https://github.com/apache/camel-k/tree/pkg/kamelet/repository/v2.0.0) (2023-07-17) + +[Full Changelog](https://github.com/apache/camel-k/compare/pkg/client/camel/v2.0.0...pkg/kamelet/repository/v2.0.0) + ## [pkg/client/camel/v2.0.0](https://github.com/apache/camel-k/tree/pkg/client/camel/v2.0.0) (2023-07-17) [Full Changelog]
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922691=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922691
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 23:43
Start Date: 07/Jun/24 23:43
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631781565
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -82,41 +86,66 @@ public void tearDown() throws Exception {
}
@Test
- public void launchDag()
- throws IOException, InterruptedException, URISyntaxException {
-Dag dag = DagManagerTest.buildDag("1",
System.currentTimeMillis(), DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(),
-5, "user5",
ConfigFactory.empty().withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef("fg")));
+ public void launchDag() throws IOException, InterruptedException,
URISyntaxException, ExecutionException {
+String flowGroup = "fg";
+String flowName = "fn";
+String flowExecutionId = "12345";
+Dag dag = DagManagerTest.buildDag("1",
Long.parseLong(flowExecutionId),
+DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(), 5, "user5",
ConfigFactory.empty()
+.withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+.withValue(ConfigurationKeys.FLOW_NAME_KEY,
ConfigValueFactory.fromAnyRef(flowName)));
FlowCompilationValidationHelper flowCompilationValidationHelper =
mock(FlowCompilationValidationHelper.class);
doReturn(com.google.common.base.Optional.of(dag)).when(flowCompilationValidationHelper).createExecutionPlanIfValid(any());
+SpecProducer specProducer =
DagManagerUtils.getSpecProducer(dag.getNodes().get(0));
Review Comment:
then probably best to assert that the other specProducers aren't invoked
Issue Time Tracking
---
Worklog Id: (was: 922691)
Time Spent: 5h 10m (was: 5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 5h 10m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(commons-beanutils) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 08ce0661)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-beanutils.git at 08ce0661 Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
(commons-validator) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now f5b2352c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-validator.git at f5b2352c Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[ https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922690=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922690 ] ASF GitHub Bot logged work on GOBBLIN-2017: --- Author: ASF GitHub Bot Created on: 07/Jun/24 23:19 Start Date: 07/Jun/24 23:19 Worklog Time Spent: 10m Work Description: codecov-commenter commented on PR #3965: URL: https://github.com/apache/gobblin/pull/3965#issuecomment-2155687853 ## [Codecov](https://app.codecov.io/gh/apache/gobblin/pull/3965?dropdown=coverage=pr=h1_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) Report All modified and coverable lines are covered by tests :white_check_mark: > Project coverage is 50.17%. Comparing base [(`a6f648b`)](https://app.codecov.io/gh/apache/gobblin/commit/a6f648b9e9521dceca1c0b6c2477e0a0f9072cd6?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache) to head [(`bd80253`)](https://app.codecov.io/gh/apache/gobblin/commit/bd802531d1fba9f918f60d0328ac0b89e1d00675?dropdown=coverage=desc_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). > Report is 6 commits behind head on master. Additional details and impacted files ```diff @@ Coverage Diff @@ ## master#3965 +/- ## - Coverage 50.18% 50.17% -0.02% - Complexity 5908 5909 +1 Files 1075 1075 Lines 4122641224 -2 Branches 4625 4625 - Hits 2069120685 -6 - Misses1873218737 +5 + Partials 1803 1802 -1 ``` [:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/gobblin/pull/3965?dropdown=coverage=pr=continue_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). :loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral_source=github_content=comment_campaign=pr+comments_term=apache). Issue Time Tracking --- Worklog Id: (was: 922690) Time Spent: 5h (was: 4h 50m) > divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions > -- > > Key: GOBBLIN-2017 > URL: https://issues.apache.org/jira/browse/GOBBLIN-2017 > Project: Apache Gobblin > Issue Type: Task >Reporter: Arjun Singh Bora >Priority: Major > Time Spent: 5h > Remaining Estimate: 0h > > divide multiple job launches in a LaunchDagProc into multiple > LaunchDagActions for two reasons > 1) it will then spend less time in each dag proc processing and have more > chance to complete the operation in the lease time > 2) handling partial job submissions in one LaunchDagProc sending N jobs is > difficult -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-28314) Support literals as filter expression with non-boolean return type
[
https://issues.apache.org/jira/browse/HIVE-28314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-28314:
--
Labels: pull-request-available (was: )
> Support literals as filter expression with non-boolean return type
> --
>
> Key: HIVE-28314
> URL: https://issues.apache.org/jira/browse/HIVE-28314
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
>
> h3. Filter expression with non-boolean return type
> fname=annotate_stats_filter.q
> {code:sql}
> explain select * from loc_orc where 'foo'
> {code}
> {noformat}
> org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Filter
> expression with non-boolean return type.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922689=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922689
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 23:05
Start Date: 07/Jun/24 23:05
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631769843
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
in multi job cases, at the dag start or at in the middle of the flow
execution (say on the 4th job), whenever "next jobs to run" are more than one,
we create reevaluate dag proc for **those next jobs**. next jobs status is not
present.
in resume case, status maybe actually present, if the dag ran in the earlier
attempt.
did that answer your question?
Issue Time Tracking
---
Worklog Id: (was: 922689)
Time Spent: 4h 50m (was: 4h 40m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 4h 50m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922688
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 22:56
Start Date: 07/Jun/24 22:56
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631766942
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##
@@ -55,6 +56,37 @@
*/
@Slf4j
public class DagProcUtils {
+
+ /**
+ * If there is a single job to run next, it runs it. If there are multiple
jobs to run, it creates a
+ * {@link
org.apache.gobblin.service.modules.orchestration.DagActionStore.DagActionType#REEVALUATE}
dag action for
+ * each of them and those jobs will be launched in respective Reevaluate dag
proc.
+ */
+ public static void submitNextNodes(DagManagementStateStore
dagManagementStateStore, Dag dag,
+ DagManager.DagId dagId) throws IOException {
+Set> nextNodes =
DagManagerUtils.getNext(dag);
+
+if (nextNodes.size() > 1) {
+ handleMultipleJobs(dagManagementStateStore, nextNodes);
+ return;
+}
+
+//Submit jobs from the dag ready for execution.
+for (Dag.DagNode dagNode : nextNodes) {
+ DagProcUtils.submitJobToExecutor(dagManagementStateStore, dagNode,
dagId);
+ log.info("Submitted job {} for dagId {}",
DagManagerUtils.getJobName(dagNode), dagId);
+}
Review Comment:
missed out it during refactor
Issue Time Tracking
---
Worklog Id: (was: 922688)
Time Spent: 4h 40m (was: 4.5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 4h 40m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922686=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922686
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 22:48
Start Date: 07/Jun/24 22:48
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631763309
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -82,41 +86,66 @@ public void tearDown() throws Exception {
}
@Test
- public void launchDag()
- throws IOException, InterruptedException, URISyntaxException {
-Dag dag = DagManagerTest.buildDag("1",
System.currentTimeMillis(), DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(),
-5, "user5",
ConfigFactory.empty().withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef("fg")));
+ public void launchDag() throws IOException, InterruptedException,
URISyntaxException, ExecutionException {
+String flowGroup = "fg";
+String flowName = "fn";
+String flowExecutionId = "12345";
+Dag dag = DagManagerTest.buildDag("1",
Long.parseLong(flowExecutionId),
+DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(), 5, "user5",
ConfigFactory.empty()
+.withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+.withValue(ConfigurationKeys.FLOW_NAME_KEY,
ConfigValueFactory.fromAnyRef(flowName)));
FlowCompilationValidationHelper flowCompilationValidationHelper =
mock(FlowCompilationValidationHelper.class);
doReturn(com.google.common.base.Optional.of(dag)).when(flowCompilationValidationHelper).createExecutionPlanIfValid(any());
+SpecProducer specProducer =
DagManagerUtils.getSpecProducer(dag.getNodes().get(0));
Review Comment:
there are 5 spec producers, as many as there are dag nodes.
Issue Time Tracking
---
Worklog Id: (was: 922686)
Time Spent: 4.5h (was: 4h 20m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922685=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922685
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 22:30
Start Date: 07/Jun/24 22:30
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631754377
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -57,25 +55,7 @@ public ReevaluateDagProc(ReevaluateDagTask
reEvaluateDagTask) {
@Override
protected Pair>, Optional>
initialize(DagManagementStateStore dagManagementStateStore)
throws IOException {
-Pair>, Optional>
dagNodeWithJobStatus =
-dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
- // this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
- // already "reevaluated" and cleaned up.
- return ImmutablePair.of(Optional.empty(), Optional.empty());
-}
Review Comment:
if dag node is not present, the return object is already empty.
Issue Time Tracking
---
Worklog Id: (was: 922685)
Time Spent: 4h 20m (was: 4h 10m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 4h 20m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(syncope) branch dependabot/maven/wicket-bootstrap.version-7.0.5 created (now 94432b11d6)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/wicket-bootstrap.version-7.0.5 in repository https://gitbox.apache.org/repos/asf/syncope.git at 94432b11d6 Bump wicket-bootstrap.version from 7.0.4 to 7.0.5 No new revisions were added by this update.
(iceberg-python) branch dependabot/pip/typing-extensions-4.12.2 created (now 563ca631)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/typing-extensions-4.12.2 in repository https://gitbox.apache.org/repos/asf/iceberg-python.git at 563ca631 Bump typing-extensions from 4.12.1 to 4.12.2 No new revisions were added by this update.
(iceberg-python) branch dependabot/pip/getdaft-0.2.27 created (now e0d8e451)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/getdaft-0.2.27 in repository https://gitbox.apache.org/repos/asf/iceberg-python.git at e0d8e451 Bump getdaft from 0.2.25 to 0.2.27 No new revisions were added by this update.
(iceberg-python) branch dependabot/pip/getdaft-0.2.26 deleted (was 3f6b9ea7)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/pip/getdaft-0.2.26 in repository https://gitbox.apache.org/repos/asf/iceberg-python.git was 3f6b9ea7 Bump getdaft from 0.2.25 to 0.2.26 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[jira] [Commented] (MDEP-939) Lock down classifier in dependency:sources goal
[
https://issues.apache.org/jira/browse/MDEP-939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853292#comment-17853292
]
ASF GitHub Bot commented on MDEP-939:
-
michael-o commented on code in PR #409:
URL:
https://github.com/apache/maven-dependency-plugin/pull/409#discussion_r1631743763
##
src/it/projects/sources/test.properties:
##
@@ -16,3 +16,4 @@
# under the License.
outputFile = target/resolved.txt
+classifier = foo
Review Comment:
This may require a comment that it has no influence anymore...
> Lock down classifier in dependency:sources goal
> ---
>
> Key: MDEP-939
> URL: https://issues.apache.org/jira/browse/MDEP-939
> Project: Maven Dependency Plugin
> Issue Type: Improvement
> Components: sources
>Affects Versions: 3.6.1
>Reporter: Michael Osipov
>Assignee: Michael Osipov
>Priority: Major
> Fix For: 3.7.0
>
>
> At the moment the classifier in {{dependency:sources}} is only set to
> {{sources}} if nothing is passed from the user, but allowing passing
> contradicts the purpose of this goal. We should make this parameter read-only
> and ignore any input to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4809) Make intermediateMessageReferences initial capacity configurable
[
https://issues.apache.org/jira/browse/ARTEMIS-4809?focusedWorklogId=922678=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922678
]
ASF GitHub Bot logged work on ARTEMIS-4809:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:57
Start Date: 07/Jun/24 21:57
Worklog Time Spent: 10m
Work Description: jbertram commented on PR #4966:
URL:
https://github.com/apache/activemq-artemis/pull/4966#issuecomment-2155625196
Thanks for the PR!
I think this would be better as an address-setting so you could easily
configure large swaths of queues. Right now there's no way to configure this
via `broker.xml`. It can only be configured if you're directly creating the
queue(s) programmatically.
Lastly, there's no tests or documentation for this new setting. Tests are
mandatory to ensure the functionality works properly and to mitigate future
regressions.
Issue Time Tracking
---
Worklog Id: (was: 922678)
Time Spent: 20m (was: 10m)
> Make intermediateMessageReferences initial capacity configurable
>
>
> Key: ARTEMIS-4809
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4809
> Project: ActiveMQ Artemis
> Issue Type: New Feature
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In some setups, there could be a few hundred thousand queues that are created
> due to many consumers that are connecting. However, most of these are empty
> and stay empty for the entire day since there aren't necessarily messages to
> be sent.
> The 8K {{intermediateMessageReferences}} instantiates an 64KB buffer
> ({{Object[]}}). This means we have large allocation and live heap that
> ultimately remains empty for almost the entire day.
> It would be quite nice if we could configure this initial size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (GOBBLIN-2082) Manifest distcp creates extra folders when publishing files
[
https://issues.apache.org/jira/browse/GOBBLIN-2082?focusedWorklogId=922670=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922670
]
ASF GitHub Bot logged work on GOBBLIN-2082:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:28
Start Date: 07/Jun/24 21:28
Worklog Time Spent: 10m
Work Description: Will-Lo opened a new pull request, #3966:
URL: https://github.com/apache/gobblin/pull/3966
Dear Gobblin maintainers,
Please accept this PR. I understand that it will not be reviewed until I
have checked off all the steps below!
### JIRA
- [ ] My PR addresses the following [Gobblin
JIRA](https://issues.apache.org/jira/browse/GOBBLIN/) issues and references
them in the PR title. For example, "[GOBBLIN-XXX] My Gobblin PR"
- https://issues.apache.org/jira/browse/GOBBLIN-2071
### Description
- [ ] Here are some details about my PR, including screenshots (if
applicable):
Manifest distcp often creates large file directory structures when copying
files from one location to another. There is a risk when concurrently
publishing files from a directory structure where the folders may not exist
before renaming files, described by the javadoc:
```
* Renames a src {@link Path} on fs {@link FileSystem} to a dst {@link Path}.
If fs is a {@link LocalFileSystem} and
* src is a directory then {@link File#renameTo} is called directly to avoid
a directory rename race condition where
* {@link org.apache.hadoop.fs.RawLocalFileSystem#rename} copies the
conflicting src directory into dst resulting in
* an extra nested level, such as /root/a/b/c/e/e where e is repeated.
```
Given that on HDFS it does not use a RawLocalFileSystem in many
implementations, we want to pre-create the folders copied sequentially to
safely create the folders before publishing the files.
This PR changes `SetPermissionCommitStep` to
`CreateAndSetDirectoryPermissionCommitStep` which also creates folders BEFORE
the commit is completed, instead of after.
### Tests
- [x] My PR adds the following unit tests __OR__ does not need testing for
this extremely good reason:
unit tests
### Commits
- [ ] My commits all reference JIRA issues in their subject lines, and I
have squashed multiple commits if they address the same issue. In addition, my
commits follow the guidelines from "[How to write a good git commit
message](http://chris.beams.io/posts/git-commit/)":
1. Subject is separated from body by a blank line
2. Subject is limited to 50 characters
3. Subject does not end with a period
4. Subject uses the imperative mood ("add", not "adding")
5. Body wraps at 72 characters
6. Body explains "what" and "why", not "how"
Issue Time Tracking
---
Worklog Id: (was: 922670)
Remaining Estimate: 0h
Time Spent: 10m
> Manifest distcp creates extra folders when publishing files
> ---
>
> Key: GOBBLIN-2082
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2082
> Project: Apache Gobblin
> Issue Type: Bug
> Components: gobblin-core
>Reporter: William Lo
>Assignee: Abhishek Tiwari
>Priority: Major
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Manifest distcp often creates large file directory structures when copying
> files from one location to another. There is a risk when concurrently
> publishing files from a directory structure where the folders may not exist
> before renaming files, described by the javadoc:
> {code:java}
> * Renames a src {@link Path} on fs {@link FileSystem} to a dst {@link Path}.
> If fs is a {@link LocalFileSystem} and
> * src is a directory then {@link File#renameTo} is called directly to avoid a
> directory rename race condition where
> * {@link org.apache.hadoop.fs.RawLocalFileSystem#rename} copies the
> conflicting src directory into dst resulting in
> * an extra nested level, such as /root/a/b/c/e/e where e is repeated. {code}
> Given that on HDFS it does not use a RawLocalFileSystem in many
> implementations, we want to pre-create the folders copied sequentially to
> safely create the folders before publishing the files.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MJAVADOC-796) Do not follow links for Java 12+
[
https://issues.apache.org/jira/browse/MJAVADOC-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853289#comment-17853289
]
ASF GitHub Bot commented on MJAVADOC-796:
-
michael-o commented on PR #287:
URL:
https://github.com/apache/maven-javadoc-plugin/pull/287#issuecomment-2155582625
See reasoning in JIRA issue and the linked one.
> Do not follow links for Java 12+
>
>
> Key: MJAVADOC-796
> URL: https://issues.apache.org/jira/browse/MJAVADOC-796
> Project: Maven Javadoc Plugin
> Issue Type: Improvement
> Components: javadoc
>Affects Versions: 3.7.0
>Reporter: Michael Osipov
>Assignee: Michael Osipov
>Priority: Major
> Fix For: 3.7.1
>
>
> For years, we have been following redirects manually because {{javadoc}} was
> not able to. with Java 12 (JDK-8190312) is was implemented and our logic can
> be skipped when {{javadoc}} 12+ is used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4800) Simplify QueueConfiguration API
[ https://issues.apache.org/jira/browse/ARTEMIS-4800?focusedWorklogId=922667=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922667 ] ASF GitHub Bot logged work on ARTEMIS-4800: --- Author: ASF GitHub Bot Created on: 07/Jun/24 21:18 Start Date: 07/Jun/24 21:18 Worklog Time Spent: 10m Work Description: tabish121 commented on PR #4964: URL: https://github.com/apache/activemq-artemis/pull/4964#issuecomment-2155566464 This looks good,I looked for awhile and didn't see any missed references. Issue Time Tracking --- Worklog Id: (was: 922667) Time Spent: 20m (was: 10m) > Simplify QueueConfiguration API > --- > > Key: ARTEMIS-4800 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4800 > Project: ActiveMQ Artemis > Issue Type: Improvement >Reporter: Justin Bertram >Assignee: Justin Bertram >Priority: Major > Time Spent: 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org For additional commands, e-mail: issues-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
(datafusion-comet) branch asf-site updated: Publish built docs triggered by f75aeefab58dc7e14cb70742b9d7bb656b727dbd
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/datafusion-comet.git The following commit(s) were added to refs/heads/asf-site by this push: new a6a1a97c Publish built docs triggered by f75aeefab58dc7e14cb70742b9d7bb656b727dbd a6a1a97c is described below commit a6a1a97c8c644bb682191c825f41312b6117a030 Author: github-actions[bot] AuthorDate: Fri Jun 7 21:15:56 2024 + Publish built docs triggered by f75aeefab58dc7e14cb70742b9d7bb656b727dbd --- _sources/user-guide/expressions.md.txt | 267 - _sources/user-guide/operators.md.txt | 27 +- searchindex.js | 2 +- user-guide/expressions.html| 668 +++-- user-guide/operators.html | 60 ++- 5 files changed, 778 insertions(+), 246 deletions(-) diff --git a/_sources/user-guide/expressions.md.txt b/_sources/user-guide/expressions.md.txt index 775ebedb..14b6f18d 100644 --- a/_sources/user-guide/expressions.md.txt +++ b/_sources/user-guide/expressions.md.txt @@ -19,99 +19,174 @@ # Supported Spark Expressions -The following Spark expressions are currently available: - -- Literals -- Arithmetic Operators - - UnaryMinus - - Add/Minus/Multiply/Divide/Remainder -- Conditional functions - - Case When - - If -- Cast -- Coalesce -- BloomFilterMightContain -- Boolean functions - - And - - Or - - Not - - EqualTo - - EqualNullSafe - - GreaterThan - - GreaterThanOrEqual - - LessThan - - LessThanOrEqual - - IsNull - - IsNotNull - - In -- String functions - - Substring - - Coalesce - - StringSpace - - Like - - Contains - - Startswith - - Endswith - - Ascii - - Bit_length - - Octet_length - - Upper - - Lower - - Chr - - Initcap - - Trim/Btrim/Ltrim/Rtrim - - Concat_ws - - Repeat - - Length - - Reverse - - Instr - - Replace - - Translate -- Bitwise functions - - Shiftright/Shiftleft -- Date/Time functions - - Year/Hour/Minute/Second -- Hash functions - - Md5 - - Sha2 - - Hash - - Xxhash64 -- Math functions - - Abs - - Acos - - Asin - - Atan - - Atan2 - - Cos - - Exp - - Ln - - Log10 - - Log2 - - Pow - - Round - - Signum - - Sin - - Sqrt - - Tan - - Ceil - - Floor -- Aggregate functions - - Count - - Sum - - Max - - Min - - Avg - - First - - Last - - BitAnd - - BitOr - - BitXor - - BoolAnd - - BoolOr - - CovPopulation - - CovSample - - VariancePop - - VarianceSamp - - StddevPop - - StddevSamp - - Corr +The following Spark expressions are currently available. Any known compatibility issues are noted in the following tables. + +## Literal Values + +| Expression | Notes | +| -- | - | +| Literal values of supported data types | | + +## Unary Arithmetic + +| Expression | Notes | +| | - | +| UnaryMinus (`-`) | | + +## Binary Arithmeticx + +| Expression | Notes | +| --- | --- | +| Add (`+`) | | +| Subtract (`-`) | | +| Multiply (`*`) | | +| Divide (`/`)| | +| Remainder (`%`) | Comet produces `NaN` instead of `NULL` for `% -0.0` | + +## Conditional Expressions + +| Expression | Notes | +| -- | - | +| CaseWhen | | +| If | | + +## Comparison + +| Expression| Notes | +| - | - | +| EqualTo (`=`) | | +| EqualNullSafe (`<=>`) | | +| GreaterThan (`>`) | | +| GreaterThanOrEqual (`>=`) | | +| LessThan (`<`)| | +| LessThanOrEqual (`<=`)| | +| IsNull (`IS NULL`)| | +| IsNotNull (`IS NOT NULL`) | | +| In (`IN`) | | + +## String Functions + +| Expression | Notes | +| --- | --- | +| Ascii | | +| BitLength | | +| Chr | | +| ConcatWs|
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922666=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922666
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:13
Start Date: 07/Jun/24 21:13
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631696425
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -123,6 +122,11 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode) {
+DagProcUtils.submitJobToExecutor(dagManagementStateStore, dagNode,
getDagId());
+log.info("Submitted job {} for dagId {}",
DagManagerUtils.getJobName(dagNode), getDagId());
+ }
Review Comment:
logging like this could arguably go in `DagProcUtils.submitJobToExecutor`.
if so this class could call that directly, rather than wrapping the call in
this method that's merely forwarding
Issue Time Tracking
---
Worklog Id: (was: 922666)
Time Spent: 4h 10m (was: 4h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922665=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922665
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:12
Start Date: 07/Jun/24 21:12
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631686698
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -57,25 +55,7 @@ public ReevaluateDagProc(ReevaluateDagTask
reEvaluateDagTask) {
@Override
protected Pair>, Optional>
initialize(DagManagementStateStore dagManagementStateStore)
throws IOException {
-Pair>, Optional>
dagNodeWithJobStatus =
-dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
- // this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
- // already "reevaluated" and cleaned up.
- return ImmutablePair.of(Optional.empty(), Optional.empty());
-}
Review Comment:
don't we still need this check for `if (!dnwjs.getLeft().isPresent())` (so
`initialize` returns empty)?
Issue Time Tracking
---
Worklog Id: (was: 922665)
Time Spent: 4h (was: 3h 50m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 4h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922663=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922663
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:11
Start Date: 07/Jun/24 21:11
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631653256
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
in the multi-job case of reevaluate wouldn't there be job status present? I
thought it's only in the multi-job launch and resume cases that no status
present...
Issue Time Tracking
---
Worklog Id: (was: 922663)
Time Spent: 3h 40m (was: 3.5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922664=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922664
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:11
Start Date: 07/Jun/24 21:11
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631657793
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
OK, that sounds reasonable. let's document the crux of this very clearly
stated rationale somewhere in source code... perhaps inside the new
`DagProcUtils` method?
Issue Time Tracking
---
Worklog Id: (was: 922664)
Time Spent: 3h 50m (was: 3h 40m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4796) Simplify SimpleString API
[
https://issues.apache.org/jira/browse/ARTEMIS-4796?focusedWorklogId=922662=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922662
]
ASF GitHub Bot logged work on ARTEMIS-4796:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:10
Start Date: 07/Jun/24 21:10
Worklog Time Spent: 10m
Work Description: jbertram commented on PR #4967:
URL:
https://github.com/apache/activemq-artemis/pull/4967#issuecomment-218579
This is a follow-up from #4959 to address some comments from @gemmellr.
Issue Time Tracking
---
Worklog Id: (was: 922662)
Time Spent: 3h 20m (was: 3h 10m)
> Simplify SimpleString API
> -
>
> Key: ARTEMIS-4796
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4796
> Project: ActiveMQ Artemis
> Issue Type: Improvement
>Reporter: Justin Bertram
>Assignee: Justin Bertram
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Instead of using the verbosely named {{toSimpleString}} static factory method
> (and its overloaded counterparts), {{SimpleString}} should provide {{of}}
> implementations. This change would make the code more concise and readable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922661=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922661
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:10
Start Date: 07/Jun/24 21:10
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631652388
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -94,29 +93,50 @@ public void launchDag()
flowCompilationValidationHelper);
launchDagProc.process(this.dagManagementStateStore);
-int expectedNumOfSavingDagNodeStates = 1; // = number of start nodes
-Assert.assertEquals(expectedNumOfSavingDagNodeStates,
+int numOfLaunchedJobs = 1; // = number of start nodes
+Assert.assertEquals(numOfLaunchedJobs,
Mockito.mockingDetails(this.dagManagementStateStore).getInvocations().stream()
.filter(a ->
a.getMethod().getName().equals("addDagNodeState")).count());
-Mockito.verify(this.dagManagementStateStore, Mockito.times(1))
+Mockito.verify(this.dagManagementStateStore,
Mockito.times(numOfLaunchedJobs))
.addFlowDagAction(any(), any(), any(),
eq(DagActionStore.DagActionType.ENFORCE_FLOW_FINISH_DEADLINE));
}
+ @Test
+ public void launchDagWithMultipleParallelJobs() throws IOException,
InterruptedException, URISyntaxException {
Review Comment:
I know and agree w/ that.
the reason I'm urging you to add it is that specProducer validation (that
never called for all of them) would have caught the bug
[here](https://github.com/apache/gobblin/pull/3965#discussion_r1631547273)
Issue Time Tracking
---
Worklog Id: (was: 922661)
Time Spent: 3.5h (was: 3h 20m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (ARTEMIS-4796) Simplify SimpleString API
[
https://issues.apache.org/jira/browse/ARTEMIS-4796?focusedWorklogId=922660=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922660
]
ASF GitHub Bot logged work on ARTEMIS-4796:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:10
Start Date: 07/Jun/24 21:10
Worklog Time Spent: 10m
Work Description: jbertram opened a new pull request, #4967:
URL: https://github.com/apache/activemq-artemis/pull/4967
(no comment)
Issue Time Tracking
---
Worklog Id: (was: 922660)
Time Spent: 3h 10m (was: 3h)
> Simplify SimpleString API
> -
>
> Key: ARTEMIS-4796
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4796
> Project: ActiveMQ Artemis
> Issue Type: Improvement
>Reporter: Justin Bertram
>Assignee: Justin Bertram
>Priority: Major
> Time Spent: 3h 10m
> Remaining Estimate: 0h
>
> Instead of using the verbosely named {{toSimpleString}} static factory method
> (and its overloaded counterparts), {{SimpleString}} should provide {{of}}
> implementations. This change would make the code more concise and readable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922659=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922659
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:09
Start Date: 07/Jun/24 21:09
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631635899
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##
@@ -55,6 +56,37 @@
*/
@Slf4j
public class DagProcUtils {
+
+ /**
+ * If there is a single job to run next, it runs it. If there are multiple
jobs to run, it creates a
+ * {@link
org.apache.gobblin.service.modules.orchestration.DagActionStore.DagActionType#REEVALUATE}
dag action for
+ * each of them and those jobs will be launched in respective Reevaluate dag
proc.
+ */
+ public static void submitNextNodes(DagManagementStateStore
dagManagementStateStore, Dag dag,
+ DagManager.DagId dagId) throws IOException {
+Set> nextNodes =
DagManagerUtils.getNext(dag);
+
+if (nextNodes.size() > 1) {
+ handleMultipleJobs(dagManagementStateStore, nextNodes);
+ return;
+}
+
+//Submit jobs from the dag ready for execution.
+for (Dag.DagNode dagNode : nextNodes) {
+ DagProcUtils.submitJobToExecutor(dagManagementStateStore, dagNode,
dagId);
+ log.info("Submitted job {} for dagId {}",
DagManagerUtils.getJobName(dagNode), dagId);
+}
Review Comment:
what's your take? IMO, this feels clear and direct:
```
if (nextNodes.size() == 1) {
submitJob(dmss, nextNodes.get(0), dagId)
log.info(...);
} else {
for (Dag.DagNode dagNode : nextNodes) {
dmss.addJobDagAction(...);
}
}
```
so only one loop, when it's actually needed, and no separate `private
static` method merely to perform looping to call a single method
Issue Time Tracking
---
Worklog Id: (was: 922659)
Time Spent: 3h 20m (was: 3h 10m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922658=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922658
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 21:09
Start Date: 07/Jun/24 21:09
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631651319
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -82,41 +86,66 @@ public void tearDown() throws Exception {
}
@Test
- public void launchDag()
- throws IOException, InterruptedException, URISyntaxException {
-Dag dag = DagManagerTest.buildDag("1",
System.currentTimeMillis(), DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(),
-5, "user5",
ConfigFactory.empty().withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef("fg")));
+ public void launchDag() throws IOException, InterruptedException,
URISyntaxException, ExecutionException {
+String flowGroup = "fg";
+String flowName = "fn";
+String flowExecutionId = "12345";
+Dag dag = DagManagerTest.buildDag("1",
Long.parseLong(flowExecutionId),
+DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(), 5, "user5",
ConfigFactory.empty()
+.withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+.withValue(ConfigurationKeys.FLOW_NAME_KEY,
ConfigValueFactory.fromAnyRef(flowName)));
FlowCompilationValidationHelper flowCompilationValidationHelper =
mock(FlowCompilationValidationHelper.class);
doReturn(com.google.common.base.Optional.of(dag)).when(flowCompilationValidationHelper).createExecutionPlanIfValid(any());
+SpecProducer specProducer =
DagManagerUtils.getSpecProducer(dag.getNodes().get(0));
Review Comment:
to dot-I's and cross-T's I'd suggest to first assert there's only one
specProducer
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##
@@ -55,6 +56,37 @@
*/
@Slf4j
public class DagProcUtils {
+
+ /**
+ * If there is a single job to run next, it runs it. If there are multiple
jobs to run, it creates a
+ * {@link
org.apache.gobblin.service.modules.orchestration.DagActionStore.DagActionType#REEVALUATE}
dag action for
+ * each of them and those jobs will be launched in respective Reevaluate dag
proc.
Review Comment:
`{@link ReevaluateDagProc}`
##
gobblin-metastore/src/test/java/org/apache/gobblin/metastore/testing/TestMetastoreDatabaseServer.java:
##
@@ -97,7 +97,7 @@ class TestMetastoreDatabaseServer implements Closeable {
.withUser(this.dbUserName, this.dbUserPassword)
.withServerVariable("explicit_defaults_for_timestamp", "off")
// default `max_connections` is apparently 151 - see:
https://dev.mysql.com/doc/refman/8.0/en/server-system-variables.html#sysvar_max_connections
-.withServerVariable("max_connections", "501")
+.withServerVariable("max_connections", "2000")
Review Comment:
hopefully this helps! ... I didn't have a lot of luck setting this myself
(to avoid the "too many conns" unit test failures elsewhere)
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -141,4 +170,14 @@ public static Dag
buildDagWithMultipleNodesAtDifferentLevels(S
}
return new JobExecutionPlanDagFactory().createDag(jobExecutionPlans);
}
+
+ public static List>
getDagSpecProducers(Dag dag) {
Review Comment:
interesting this is only called from `ReevaluateDagProcTest`. does it
belong there?
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##
@@ -55,6 +56,37 @@
*/
@Slf4j
public class DagProcUtils {
+
+ /**
+ * If there is a single job to run next, it runs it. If there are multiple
jobs to run, it creates a
+ * {@link
org.apache.gobblin.service.modules.orchestration.DagActionStore.DagActionType#REEVALUATE}
dag action for
+ * each of them and those jobs will be launched in respective Reevaluate dag
proc.
+ */
+ public static void submitNextNodes(DagManagementStateStore
dagManagementStateStore, Dag dag,
+ DagManager.DagId dagId) throws IOException {
+Set> nextNodes =
DagManagerUtils.getNext(dag);
+
+if (nextNodes.size() > 1) {
+ handleMultipleJobs(dagManagementStateStore, nextNodes);
+ return;
+}
+
+//Submit jobs from the dag ready for execution.
+for (Da
[jira] [Updated] (HIVE-28313) Support ambiguous column references when CBO fallback is disabled
[
https://issues.apache.org/jira/browse/HIVE-28313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-28313:
--
Labels: pull-request-available (was: )
> Support ambiguous column references when CBO fallback is disabled
> -
>
> Key: HIVE-28313
> URL: https://issues.apache.org/jira/browse/HIVE-28313
> Project: Hive
> Issue Type: Sub-task
> Components: CBO
>Reporter: Soumyakanti Das
>Assignee: Soumyakanti Das
>Priority: Major
> Labels: pull-request-available
>
> h3. Ambiguous column references test failures
> fname=ambiguous_col.q
> {code:sql}
> explain select * from (select a.key, a.* from (select * from src) a join
> (select * from src1) b on (a.key = b.key)) t
> {code}
> {noformat}
> org.apache.hadoop.hive.ql.optimizer.calcite.CalciteSemanticException: Cannot
> add column to RR: a.key => _col1: string due to duplication, see previous
> warnings
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MDEP-895) dependency:sources fails for multi-module project
[
https://issues.apache.org/jira/browse/MDEP-895?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853284#comment-17853284
]
ASF GitHub Bot commented on MDEP-895:
-
michael-o commented on PR #349:
URL:
https://github.com/apache/maven-dependency-plugin/pull/349#issuecomment-212938
This PR will be reverted with #408.
> dependency:sources fails for multi-module project
> -
>
> Key: MDEP-895
> URL: https://issues.apache.org/jira/browse/MDEP-895
> Project: Maven Dependency Plugin
> Issue Type: Bug
> Components: sources
>Affects Versions: 3.6.1
>Reporter: Gerd Aschemann
>Assignee: Slawomir Jaranowski
>Priority: Critical
>
> When executing `mvn dependency:sources` on a multi-module project it fails to
> retrieve sources for the modules themselves (which is not necessary as the
> source code is provided locally already):
> {code:java}
> mvn dependency:sources
> ...
> [INFO] -< org.apache.maven.its.dependency:tree-multimodule-module-b
> >--
> [INFO] Building MultimoduleTest Module B 1.0-SNAPSHOT
> [3/3]
> [INFO] from module-b/pom.xml
> [INFO] [ jar
> ]-
> [INFO]
>
> [INFO] Reactor Summary for MultimoduleTest 1.0-SNAPSHOT:
> [INFO]
> [INFO] MultimoduleTest SUCCESS [ 0.390
> s]
> [INFO] MultimoduleTest Module A ... SUCCESS [ 0.063
> s]
> [INFO] MultimoduleTest Module B ... FAILURE [ 0.005
> s]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 0.587 s
> [INFO] Finished at: 2023-11-09T10:38:16+01:00
> [INFO]
>
> [ERROR] Failed to execute goal on project tree-multimodule-module-b: Could
> not resolve dependencies for project
> org.apache.maven.its.dependency:tree-multimodule-module-b:jar:1.0-SNAPSHOT:
> The following artifacts could not be resolved:
> org.apache.maven.its.dependency:tree-multimodule-module-a:jar:1.0-SNAPSHOT
> (absent): Could not find artifact
> org.apache.maven.its.dependency:tree-multimodule-module-a:jar:1.0-SNAPSHOT ->
> [Help 1]{code}
> This is similar to the behaviour of MDEP-409.
> A test and PR is already prepared:
> https://github.com/jmle/maven-dependency-plugin/tree/fix-sources-multimodule
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MDEP-900) dependency:resolve fails for multi-module project
[
https://issues.apache.org/jira/browse/MDEP-900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853281#comment-17853281
]
ASF GitHub Bot commented on MDEP-900:
-
michael-o commented on PR #406:
URL:
https://github.com/apache/maven-dependency-plugin/pull/406#issuecomment-2155545377
Revert is here: #408
> dependency:resolve fails for multi-module project
> -
>
> Key: MDEP-900
> URL: https://issues.apache.org/jira/browse/MDEP-900
> Project: Maven Dependency Plugin
> Issue Type: Bug
> Components: resolve
>Affects Versions: 3.6.1
>Reporter: Gerd Aschemann
>Assignee: Slawomir Jaranowski
>Priority: Critical
> Fix For: 3.7.0
>
>
> When executing {{mvn dependency:resolve}} on a multi-module project it fails
> to retrieve artifacts for the modules themselves (which is not necessary as
> these can be installed by {{mvn install}} any time).
> This can be seen on an [OSS multi-module
> project|https://github.com/WASdev/sample.daytrader7]:
> {code:java}
> mvn dependency:resolve
> [INFO] Scanning for projects...
> [INFO]
>
> [INFO] Reactor Build Order:
> [INFO]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample - EJB Module
> [jar]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample - Web Module
> [war]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample
> [pom]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample - EAR
> [ear]
> [INFO]
> [INFO] --< net.wasdev.wlp.sample:daytrader-ee7-ejb
> >---
> [INFO] Building WAS Liberty Sample - Java EE7 Benchmark Sample - EJB Module
> 1.0-SNAPSHOT [1/4]
> [INFO] from daytrader-ee7-ejb/pom.xml
> [INFO] [ jar
> ]-
> [INFO]
> [INFO] --- dependency:3.6.0:resolve (default-cli) @ daytrader-ee7-ejb ---
> [INFO]
> [INFO] The following files have been resolved:
> [INFO] javax:javaee-api:jar:7.0:provided -- module javaee.api (auto)
> [INFO] com.sun.mail:javax.mail:jar:1.5.0:provided -- module javax.mail
> (auto)
> [INFO] javax.activation:activation:jar:1.1:provided -- module activation
> (auto)
> [INFO] javax.annotation:javax.annotation-api:jar:1.3.2:provided -- module
> java.annotation [auto]
> [INFO]
> [INFO]
> [INFO] --< net.wasdev.wlp.sample:daytrader-ee7-web
> >---
> [INFO] Building WAS Liberty Sample - Java EE7 Benchmark Sample - Web Module
> 1.0-SNAPSHOT [2/4]
> [INFO] from daytrader-ee7-web/pom.xml
> [INFO] [ war
> ]-
> [INFO]
>
> [INFO] Reactor Summary for WAS Liberty Sample - Java EE7 Benchmark Sample
> 1.0-SNAPSHOT:
> [INFO]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample - EJB Module SUCCESS [
> 0.402 s]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample - Web Module FAILURE [
> 0.012 s]
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample . SKIPPED
> [INFO] WAS Liberty Sample - Java EE7 Benchmark Sample - EAR SKIPPED
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 0.552 s
> [INFO] Finished at: 2023-11-14T12:37:02+01:00
> [INFO]
>
> [ERROR] Failed to execute goal on project daytrader-ee7-web: Could not
> resolve dependencies for project
> net.wasdev.wlp.sample:daytrader-ee7-web:war:1.0-SNAPSHOT: The following
> artifacts could not be resolved:
> net.wasdev.wlp.sample:daytrader-ee7-ejb:jar:1.0-SNAPSHOT (absent): Could not
> find artifact net.wasdev.wlp.sample:daytrader-ee7-ejb:jar:1.0-SNAPSHOT ->
> [Help 1]{code}
> This is similar to the behaviour of MDEP-409, and MDEP-895
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MSKINS-248) If $bodyContent is empty HTML code is distorted
[
https://issues.apache.org/jira/browse/MSKINS-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853280#comment-17853280
]
ASF GitHub Bot commented on MSKINS-248:
---
michael-o opened a new pull request, #61:
URL: https://github.com/apache/maven-fluido-skin/pull/61
This closes #61
> If $bodyContent is empty HTML code is distorted
> ---
>
> Key: MSKINS-248
> URL: https://issues.apache.org/jira/browse/MSKINS-248
> Project: Maven Skins
> Issue Type: Bug
> Components: Fluido Skin
>Affects Versions: fluido-2.0.0-M9
>Reporter: Michael Osipov
>Assignee: Michael Osipov
>Priority: Major
> Fix For: fluido-2.0.0, fluido-2.0.0-M10
>
>
> If {{$bodyContent}} is empty HTML code is distorted because Velocity does not
> expand the variable, but leaves the literal {{$bodyContent}}. We need to
> check it with an {{#if}} clause.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MDEP-923) Code cleanups
[
https://issues.apache.org/jira/browse/MDEP-923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853279#comment-17853279
]
ASF GitHub Bot commented on MDEP-923:
-
slawekjaranowski merged PR #407:
URL: https://github.com/apache/maven-dependency-plugin/pull/407
> Code cleanups
> -
>
> Key: MDEP-923
> URL: https://issues.apache.org/jira/browse/MDEP-923
> Project: Maven Dependency Plugin
> Issue Type: Task
>Reporter: Slawomir Jaranowski
>Assignee: Slawomir Jaranowski
>Priority: Major
> Fix For: 3.7.0
>
>
> * remove usage of deprecated API where possible
> * cleanup pom after update to 42
> * exclude transitive dependencies on org.apache.maven
> * add {{@project.version@}} in ITs
> * Remove plexus logger from DependencySilentLog
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MJAVADOC-796) Do not follow links for Java 12+
[
https://issues.apache.org/jira/browse/MJAVADOC-796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853277#comment-17853277
]
ASF GitHub Bot commented on MJAVADOC-796:
-
michael-o opened a new pull request, #287:
URL: https://github.com/apache/maven-javadoc-plugin/pull/287
This closes #287
Following this checklist to help us incorporate your
contribution quickly and easily:
- [ ] Make sure there is a [JIRA
issue](https://issues.apache.org/jira/browse/MJAVADOC) filed
for the change (usually before you start working on it). Trivial
changes like typos do not
require a JIRA issue. Your pull request should address just this
issue, without
pulling in other changes.
- [ ] Each commit in the pull request should have a meaningful subject line
and body.
- [ ] Format the pull request title like `[MJAVADOC-XXX] - Fixes bug in
ApproximateQuantiles`,
where you replace `MJAVADOC-XXX` with the appropriate JIRA issue.
Best practice
is to use the JIRA issue title in the pull request title and in the
first line of the
commit message.
- [ ] Write a pull request description that is detailed enough to
understand what the pull request does, how, and why.
- [ ] Run `mvn clean verify -Prun-its` to make sure basic checks pass. A
more thorough check will
be performed on your pull request automatically.
If your pull request is about ~20 lines of code you don't need to sign an
[Individual Contributor License
Agreement](https://www.apache.org/licenses/icla.pdf) if you are unsure
please ask on the developers list.
To make clear that you license your contribution under
the [Apache License Version 2.0, January
2004](http://www.apache.org/licenses/LICENSE-2.0)
you have to acknowledge this by using the following check-box.
- [ ] I hereby declare this contribution to be licensed under the [Apache
License Version 2.0, January 2004](http://www.apache.org/licenses/LICENSE-2.0)
- [ ] In any other case, please file an [Apache Individual Contributor
License Agreement](https://www.apache.org/licenses/icla.pdf).
> Do not follow links for Java 12+
>
>
> Key: MJAVADOC-796
> URL: https://issues.apache.org/jira/browse/MJAVADOC-796
> Project: Maven Javadoc Plugin
> Issue Type: Improvement
> Components: javadoc
>Affects Versions: 3.7.0
>Reporter: Michael Osipov
>Assignee: Michael Osipov
>Priority: Major
> Fix For: 3.7.1
>
>
> For years, we have been following redirects manually because {{javadoc}} was
> not able to. with Java 12 (JDK-8190312) is was implemented and our logic can
> be skipped when {{javadoc}} 12+ is used.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-28312) Save query error message on Driver level
[
https://issues.apache.org/jira/browse/HIVE-28312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HIVE-28312:
--
Labels: pull-request-available (was: )
> Save query error message on Driver level
>
>
> Key: HIVE-28312
> URL: https://issues.apache.org/jira/browse/HIVE-28312
> Project: Hive
> Issue Type: Improvement
>Reporter: László Bodor
>Assignee: László Bodor
>Priority: Major
> Labels: pull-request-available
>
> current call path is *SQLOperation -> Driver.run*
> Instead of just throwing CommandProcessorException, Driver might save the the
> error message for query-tracking purposes, this is a single message for later
> reference, like:
> {code}
> FAILED: SemanticException [Error 10001]: Line 1:14 Table not found 'abcd'
> {code}
> 2 ways to achieve this
> 1. wrap every occurrence of throwing CommandProcessorException inside driver
> e.g.
> https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/ql/src/java/org/apache/hadoop/hive/ql/Driver.java#L146
> PRO: storing of error message stays inside the Driver (SQLCommand doesn't
> need to know about it)
> CONS: looks bad, not future-proof (later, anyone can miss this call while
> throwing a cpe), needs Driver code changes at several places
> 2. catch in SQLOperation and propagate back to Driver
> https://github.com/apache/hive/blob/98d9d22398370f817fe64449368671b978fff096/service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java#L237
> PRO: it looks better, every exception can be caught in a single place
> CON: it looks weird, because SQLOperation has to call back to the driver to
> store the error message (which was already passed through the driver)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(commons-email) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 979e07c)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-email.git at 979e07c Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
(commons-text) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 1d45a95b)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-text.git at 1d45a95b Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
(pinot) branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 deleted (was db3a4c5918)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git was db3a4c5918 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository. - To unsubscribe, e-mail: commits-unsubscr...@pinot.apache.org For additional commands, e-mail: commits-h...@pinot.apache.org
(arrow-nanoarrow) branch asf-site updated: update documentation for tag main
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/arrow-nanoarrow.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 3eb2c9ef update documentation for tag main
3eb2c9ef is described below
commit 3eb2c9eff047f1dc5f7759bd8031f92a82f0b73d
Author: GitHub Actions
AuthorDate: Fri Jun 7 19:59:26 2024 +
update documentation for tag main
---
main/genindex.html| 8 ++--
main/objects.inv | Bin 20974 -> 21095 bytes
main/r/pkgdown.yml| 2 +-
main/reference/c.html | 14 ++
main/searchindex.js | 2 +-
5 files changed, 22 insertions(+), 4 deletions(-)
diff --git a/main/genindex.html b/main/genindex.html
index 7203b8e6..34de8324 100644
--- a/main/genindex.html
+++ b/main/genindex.html
@@ -803,10 +803,10 @@ document.write(`
ArrowDeviceArrayStream::get_schema
(C++ member)
-
-
ArrowDeviceArrayStream::private_data
(C++ member)
+
+
ArrowDeviceArrayStream::release
(C++ member)
ArrowDeviceArrayView
(C++ struct)
@@ -1082,6 +1082,8 @@ document.write(`
ArrowSchemaSetTypeDecimal
(C++ function)
ArrowSchemaSetTypeFixedSize
(C++ function)
+
+ ArrowSchemaSetTypeRunEndEncoded
(C++ function)
ArrowSchemaSetTypeStruct
(C++ function)
@@ -1194,6 +1196,8 @@ document.write(`
ArrowType::NANOARROW_TYPE_MAP
(C++ enumerator)
ArrowType::NANOARROW_TYPE_NA
(C++ enumerator)
+
+ ArrowType::NANOARROW_TYPE_RUN_END_ENCODED
(C++ enumerator)
ArrowType::NANOARROW_TYPE_SPARSE_UNION
(C++ enumerator)
diff --git a/main/objects.inv b/main/objects.inv
index 7d89f131..85c77f25 100644
Binary files a/main/objects.inv and b/main/objects.inv differ
diff --git a/main/r/pkgdown.yml b/main/r/pkgdown.yml
index afcd7b26..3af1b692 100644
--- a/main/r/pkgdown.yml
+++ b/main/r/pkgdown.yml
@@ -2,5 +2,5 @@ pandoc: 3.1.3
pkgdown: 2.0.9
pkgdown_sha: ~
articles: {}
-last_built: 2024-06-04T13:14Z
+last_built: 2024-06-07T19:59Z
diff --git a/main/reference/c.html b/main/reference/c.html
index 9e12d40f..5f7dcf41 100644
--- a/main/reference/c.html
+++ b/main/reference/c.html
@@ -465,6 +465,13 @@ document.write(`
Returns EINVAL for scale = 0 or for type that is not
NANOARROW_TYPE_DECIMAL128 or NANOARROW_TYPE_DECIMAL256. Schema must have been
initialized using ArrowSchemaInit() or ArrowSchemaDeepCopy().
+
+
+ArrowErrorCodeSet the format field of a run-end encoded schema.
+Returns EINVAL for run_end_type that is not NANOARROW_TYPE_INT16,
NANOARROW_TYPE_INT32 or NANOARROW_TYPE_INT64. Schema must have been initialized
using ArrowSchemaInit() or ArrowSchemaDeepCopy(). The caller must call
+
Arro [...]
@@ -1936,6 +1943,11 @@ document.write(`
enumerator
NANOARROW_TYPE_INTERVAL_MONTH_DAY_NANO
+
+
+enumerator
NANOARROW_TYPE_RUN_END_ENCODED
@@ -2543,6 +2555,7 @@ document.write(`
ArrowSchemaSetTypeStruct()
ArrowSchemaSetTypeFixedSize()
ArrowSchemaSetTypeDecimal()
+ArrowSchemaSetTypeRunEndEncoded()
ArrowSchemaSetTypeDateTime()
ArrowSchemaSetTypeUnion()
ArrowSchemaDeepCopy()
@@ -2795,6 +2808,7 @@ document.write(`
ArrowType::NANOARROW_TYPE_LARGE_BINARY
ArrowType::NANOARROW_TYPE_LARGE_LIST
ArrowType::NANOARROW_TYPE_INTERVAL_MONTH_DAY_NANO
+ArrowType::NANOARROW_TYPE_RUN_END_ENCODED
ArrowTimeUnit
diff --git a/main/searchindex.js b/main/searchindex.js
index 886677f3..14dfafe9 100644
--- a/main/searchindex.js
+++ b/main/searchindex.js
@@ -1 +1 @@
-Search.setIndex({"alltitles": {"API Reference": [[8, "api-reference"]], "Array
Stream utilities": [[6, "array-stream-utilities"]], "Array Streams": [[3,
"array-streams"]], "Array streams": [[2, "array-streams"]], "Arrays": [[2,
"arrays"], [3, "arrays"]], "Arrow C Data Interface": [[5,
"arrow-c-data-interface"]], "Arrow C Device Interface": [[7,
"arrow-c-device-interface"]], "Arrow C data/nanoarrow interface basics": [[0,
"arrow-c-data-nanoarrow-interface-basics"]], "Bitmap utilities": [[ [...]
\ No newline at end of file
+Search.setIndex({"alltitles": {"API Reference": [[8, "api-reference"]], "Array
Stream utilities": [[6, "array-stream-utilities"]], "Array Streams": [[3,
"array-streams"]], "Array streams": [[2, "array-streams"]], "Arrays": [[2,
"arrays"], [3, "arrays"]], "Arrow C Data Interface": [[5,
"arrow-c-data-interface"]], "Arrow C Device Interface": [[7,
"arrow-c-device-interface"]], "Arrow C data/nanoarrow interface basics": [[0,
"arrow-c-data-nanoarrow-interface-basics"]], "Bitmap utilities": [[ [...]
\ No newline at end of file
(commons-bcel) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 7131e280)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-bcel.git at 7131e280 Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
[jira] [Updated] (HDDS-10987) Add explicit dependency for hadoop-auth
[
https://issues.apache.org/jira/browse/HDDS-10987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-10987:
--
Labels: pull-request-available (was: )
> Add explicit dependency for hadoop-auth
> ---
>
> Key: HDDS-10987
> URL: https://issues.apache.org/jira/browse/HDDS-10987
> Project: Apache Ozone
> Issue Type: Improvement
> Components: build
>Reporter: Attila Doroszlai
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
>
> Ozone server components use code from {{hadoop-auth}}, e.g.
> {{org.apache.hadoop.security.authentication.client.AuthenticationException}}.
> However, dependency on {{hadoop-auth}} is only transitive via
> {{hadoop-common}}. We should add explicit dependency.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
(airflow) branch constraints-main updated: Updating constraints. Github run id:9421049843
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch constraints-main in repository https://gitbox.apache.org/repos/asf/airflow.git The following commit(s) were added to refs/heads/constraints-main by this push: new 60fb517032 Updating constraints. Github run id:9421049843 60fb517032 is described below commit 60fb517032317a45c046401fa992840940140da7 Author: Automated GitHub Actions commit AuthorDate: Fri Jun 7 19:40:51 2024 + Updating constraints. Github run id:9421049843 This update in constraints is automatically committed by the CI 'constraints-push' step based on 'refs/heads/main' in the 'apache/airflow' repository with commit sha 65dbf86f72ed7be779e7dadd8e8e57c1216c7c07. The action that build those constraints can be found at https://github.com/apache/airflow/actions/runs/9421049843/ The image tag used for that build was: 65dbf86f72ed7be779e7dadd8e8e57c1216c7c07. You can enter Breeze environment with this image by running 'breeze shell --image-tag 65dbf86f72ed7be779e7dadd8e8e57c1216c7c07' All tests passed in this build so we determined we can push the updated constraints. See https://github.com/apache/airflow/blob/main/README.md#installing-from-pypi for details. --- constraints-3.10.txt | 51 ++- constraints-3.11.txt | 51 ++- constraints-3.12.txt | 51 ++- constraints-3.8.txt | 51 ++- constraints-3.9.txt | 51 ++- constraints-no-providers-3.10.txt | 10 +++ constraints-no-providers-3.11.txt | 10 +++ constraints-no-providers-3.12.txt | 10 +++ constraints-no-providers-3.8.txt | 10 +++ constraints-no-providers-3.9.txt | 10 +++ constraints-source-providers-3.10.txt | 49 - constraints-source-providers-3.11.txt | 49 - constraints-source-providers-3.12.txt | 49 - constraints-source-providers-3.8.txt | 49 - constraints-source-providers-3.9.txt | 49 - 15 files changed, 280 insertions(+), 270 deletions(-) diff --git a/constraints-3.10.txt b/constraints-3.10.txt index 11eeb9cf39..427fa4c913 100644 --- a/constraints-3.10.txt +++ b/constraints-3.10.txt @@ -1,6 +1,6 @@ # -# This constraints file was automatically generated on 2024-06-04T21:08:24.730115 +# This constraints file was automatically generated on 2024-06-07T18:09:25.049586 # via "eager-upgrade" mechanism of PIP. For the "main" branch of Airflow. # This variant of constraints install uses the HEAD of the branch version for 'apache-airflow' but installs # the providers from PIP-released packages at the moment of the constraint generation. @@ -117,7 +117,7 @@ apache-airflow-providers-apprise==1.3.1 apache-airflow-providers-arangodb==2.5.1 apache-airflow-providers-asana==2.5.1 apache-airflow-providers-atlassian-jira==2.6.1 -apache-airflow-providers-celery==3.7.1 +apache-airflow-providers-celery==3.7.0 apache-airflow-providers-cloudant==3.5.1 apache-airflow-providers-cncf-kubernetes==8.3.0 apache-airflow-providers-cohere==1.2.1 @@ -200,10 +200,10 @@ atlasclient==1.0.0 atlassian-python-api==3.41.13 attrs==23.2.0 aws-sam-translator==1.89.0 -aws-xray-sdk==2.13.1 +aws-xray-sdk==2.14.0 azure-batch==14.2.0 azure-common==1.1.28 -azure-core==1.30.1 +azure-core==1.30.2 azure-cosmos==4.7.0 azure-datalake-store==0.0.53 azure-identity==1.16.0 @@ -213,7 +213,7 @@ azure-mgmt-containerinstance==10.1.0 azure-mgmt-containerregistry==10.3.0 azure-mgmt-core==1.4.0 azure-mgmt-cosmosdb==9.5.0 -azure-mgmt-datafactory==7.1.0 +azure-mgmt-datafactory==8.0.0 azure-mgmt-datalake-nspkg==3.0.1 azure-mgmt-datalake-store==0.5.0 azure-mgmt-nspkg==3.0.2 @@ -224,7 +224,7 @@ azure-servicebus==7.12.2 azure-storage-blob==12.20.0 azure-storage-file-datalake==12.15.0 azure-storage-file-share==12.16.0 -azure-synapse-artifacts==0.18.0 +azure-synapse-artifacts==0.19.0 azure-synapse-spark==0.7.0 backoff==2.2.1 backports.tarfile==1.2.0 @@ -275,7 +275,7 @@ db-dtypes==1.2.0 debugpy==1.8.1 decorator==5.1.1 defusedxml==0.7.1 -deltalake==0.17.4 +deltalake==0.18.0 diagrams==0.23.4 dill==0.3.1.1 distlib==0.3.8 @@ -287,7 +287,7 @@ docstring_parser==0.16 docutils==0.16 duckdb==1.0.0 elastic-transport==8.13.1 -elasticsearch==8.13.2 +elasticsearch==8.14.0 email_validator==2.1.1 entrypoints==0.4 eralchemy2==1.4.1 @@ -312,18 +312,18 @@ gcsfs==2023.12.2.post1 geomet==0.2.1.post1 gevent==24.2.1 gitdb==4.0.11 -google-ads==24.0.0 +google-ads==24.1.0 google-analytics-admin==0.22.7 google-api-core==2.19.0 google-api-python-client==2.132.0 google-auth-httplib2
[jira] [Work logged] (ARTEMIS-4809) Make intermediateMessageReferences initial capacity configurable
[
https://issues.apache.org/jira/browse/ARTEMIS-4809?focusedWorklogId=922644=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922644
]
ASF GitHub Bot logged work on ARTEMIS-4809:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 19:40
Start Date: 07/Jun/24 19:40
Worklog Time Spent: 10m
Work Description: joshb1050 opened a new pull request, #4966:
URL: https://github.com/apache/activemq-artemis/pull/4966
In some setups, there could be a few hundred thousand queues that are
created due to many consumers that are connecting. However, most of these are
empty and stay empty for the entire day since there aren't necessarily messages
to be sent. The 8K `intermediateMessageReferences` instantiates an `64KB`
buffer (`Object[]`). This means we have large allocation and live heap that
ultimately remains empty for almost the entire day.
In this commit, we introduce `intermediate-message-buffer-initial-size`,
which defaults to the current value of `8192`. It can be set programmatically
via
`QueueConfiguration#setIntermediateMessageBufferInitialSize(int)`.
Note that this must be a power of 2.
Issue Time Tracking
---
Worklog Id: (was: 922644)
Remaining Estimate: 0h
Time Spent: 10m
> Make intermediateMessageReferences initial capacity configurable
>
>
> Key: ARTEMIS-4809
> URL: https://issues.apache.org/jira/browse/ARTEMIS-4809
> Project: ActiveMQ Artemis
> Issue Type: New Feature
> Components: Broker
>Reporter: Josh Byster
>Priority: Minor
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In some setups, there could be a few hundred thousand queues that are created
> due to many consumers that are connecting. However, most of these are empty
> and stay empty for the entire day since there aren't necessarily messages to
> be sent.
> The 8K {{intermediateMessageReferences}} instantiates an 64KB buffer
> ({{Object[]}}). This means we have large allocation and live heap that
> ultimately remains empty for almost the entire day.
> It would be quite nice if we could configure this initial size.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org
For additional commands, e-mail: issues-h...@activemq.apache.org
For further information, visit: https://activemq.apache.org/contact
(commons-rdf) branch dependabot/maven/org.eclipse.rdf4j-rdf4j-bom-4.3.11 deleted (was 6f2976fd)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.eclipse.rdf4j-rdf4j-bom-4.3.11 in repository https://gitbox.apache.org/repos/asf/commons-rdf.git was 6f2976fd Bump org.eclipse.rdf4j:rdf4j-bom from 3.7.7 to 4.3.11 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(commons-rdf) branch dependabot/maven/org.eclipse.rdf4j-rdf4j-bom-4.3.12 created (now 34944777)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.eclipse.rdf4j-rdf4j-bom-4.3.12 in repository https://gitbox.apache.org/repos/asf/commons-rdf.git at 34944777 Bump org.eclipse.rdf4j:rdf4j-bom from 3.7.7 to 4.3.12 No new revisions were added by this update.
[jira] [Work logged] (ARTEMIS-4545) Allow node ID to be configured
[ https://issues.apache.org/jira/browse/ARTEMIS-4545?focusedWorklogId=922641=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922641 ] ASF GitHub Bot logged work on ARTEMIS-4545: --- Author: ASF GitHub Bot Created on: 07/Jun/24 19:35 Start Date: 07/Jun/24 19:35 Worklog Time Spent: 10m Work Description: AntonRoskvist commented on PR #4951: URL: https://github.com/apache/activemq-artemis/pull/4951#issuecomment-2155417744 Thanks for your feedback @gtully If I am not mistaken the first case should be addressed already, where the coordination-id takes precedence over the configured nodeID. It should be verified by: `org.apache.activemq.artemis.tests.integration.replication.LockManagerReplicationTest#testPrimaryPeers()` let me know if I misunderstood though. I will take a look at the JDBC case, thanks! Issue Time Tracking --- Worklog Id: (was: 922641) Time Spent: 40m (was: 0.5h) > Allow node ID to be configured > -- > > Key: ARTEMIS-4545 > URL: https://issues.apache.org/jira/browse/ARTEMIS-4545 > Project: ActiveMQ Artemis > Issue Type: New Feature >Reporter: Justin Bertram >Assignee: Justin Bertram >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > In certain situations it would be beneficial to configure the node ID rather > than having it automatically generated. > For example, when using replication + failback if the primary server fails > the backup will take over. Then when the primary is restarted it will > initiate failback. However, if the primary broker's journal is damaged or > lost during the initial failure then it won't be able to initiate failback > because it won't have the same node ID as the backup. This kind of situation > is not uncommon in cloud environments where there is no persistent, attached > storage. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: issues-unsubscr...@activemq.apache.org For additional commands, e-mail: issues-h...@activemq.apache.org For further information, visit: https://activemq.apache.org/contact
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 0770df9338f Updating config from bot
0770df9338f is described below
commit 0770df9338f991b04cd556ce4738277ecd6f8636
Author: github-actions
AuthorDate: Fri Jun 7 19:35:20 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31547.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31547.json
b/scripts/ci/pr-bot/state/pr-state/pr-31547.json
index 242a48d7d3b..37f7ab41238 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-31547.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31547.json
@@ -2,7 +2,7 @@
"commentedAboutFailingChecks": true,
"reviewersAssignedForLabels": {},
"nextAction": "Author",
- "stopReviewerNotifications": false,
+ "stopReviewerNotifications": true,
"remindAfterTestsPass": [],
"committerAssigned": false
}
\ No newline at end of file
(commons-rdf) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 88243bbc)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-rdf.git at 88243bbc Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
[jira] [Updated] (HDDS-10988) Let zero OzoneQuota use byte as unit
[
https://issues.apache.org/jira/browse/HDDS-10988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-10988:
--
Labels: pull-request-available (was: )
> Let zero OzoneQuota use byte as unit
>
>
> Key: HDDS-10988
> URL: https://issues.apache.org/jira/browse/HDDS-10988
> Project: Apache Ozone
> Issue Type: Improvement
> Components: Ozone Manager
>Reporter: Tsz-wo Sze
>Assignee: Attila Doroszlai
>Priority: Minor
> Labels: pull-request-available
>
> Just found that {{valueOf(0)}} will use {{EB}}. (Fortunately, we added
> {{EB}}; otherwise, it becomes an {{ArrayIndexOutOfBoundsException}}). Not a
> big deal, although we probably should fix it.
> {code}
> public static void main(String[] args) {
> final RawQuotaInBytes q = RawQuotaInBytes.valueOf(0);
> System.out.println("q = " + q);
> }
> // q = 0 EB
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@ozone.apache.org
For additional commands, e-mail: issues-h...@ozone.apache.org
[jira] [Updated] (SPARK-48566) [Bug] Partition indices are incorrect when UDTF analyze() uses both select and partitionColumns
[
https://issues.apache.org/jira/browse/SPARK-48566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-48566:
---
Labels: pull-request-available (was: )
> [Bug] Partition indices are incorrect when UDTF analyze() uses both select
> and partitionColumns
> ---
>
> Key: SPARK-48566
> URL: https://issues.apache.org/jira/browse/SPARK-48566
> Project: Spark
> Issue Type: Sub-task
> Components: PySpark
>Affects Versions: 4.0.0
>Reporter: Daniel
>Priority: Major
> Labels: pull-request-available
>
> There is a bug that results in an internal error with some combination of the
> Python UDTF "select" and "partitionBy" options of the "analyze" method.
> To reproduce:
> {code:java}
> from pyspark.sql.functions import (
> AnalyzeArgument,
> AnalyzeResult,
> PartitioningColumn,
> SelectedColumn,
> udtf
> )
> from pyspark.sql.types import (
> DoubleType,
> StringType,
> StructType,
> )
> @udtf
> class TestTvf:
> @staticmethod
> def analyze(observed: AnalyzeArgument) -> AnalyzeResult:
> out_schema = StructType()
> out_schema.add("partition_col", StringType())
> out_schema.add("double_col", DoubleType())
> return AnalyzeResult(
> schema=out_schema,
> partitionBy=[PartitioningColumn("partition_col")],
> select=[
> SelectedColumn("partition_col"),
> SelectedColumn("double_col"),
> ],
> )
> def eval(self, *args, **kwargs):
> pass
> def terminate(self):
> for _ in range(10):
> yield {
> "partition_col": None,
> "double_col": 1.0,
> }
> spark.udtf.register("serialize_test", TestTvf)
> # Fails
> (
> spark
> .sql(
> """
> SELECT * FROM serialize_test(
> TABLE(
> SELECT
> 5 AS unused_col,
> 'hi' AS partition_col,
> 1.0 AS double_col
>
> UNION ALL
> SELECT
> 4 AS unused_col,
> 'hi' AS partition_col,
> 1.0 AS double_col
> )
> )
> """
> )
> .toPandas()
> ){code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922637=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922637
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 19:06
Start Date: 07/Jun/24 19:06
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631594293
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -94,29 +93,50 @@ public void launchDag()
flowCompilationValidationHelper);
launchDagProc.process(this.dagManagementStateStore);
-int expectedNumOfSavingDagNodeStates = 1; // = number of start nodes
-Assert.assertEquals(expectedNumOfSavingDagNodeStates,
+int numOfLaunchedJobs = 1; // = number of start nodes
+Assert.assertEquals(numOfLaunchedJobs,
Mockito.mockingDetails(this.dagManagementStateStore).getInvocations().stream()
.filter(a ->
a.getMethod().getName().equals("addDagNodeState")).count());
-Mockito.verify(this.dagManagementStateStore, Mockito.times(1))
+Mockito.verify(this.dagManagementStateStore,
Mockito.times(numOfLaunchedJobs))
.addFlowDagAction(any(), any(), any(),
eq(DagActionStore.DagActionType.ENFORCE_FLOW_FINISH_DEADLINE));
}
+ @Test
+ public void launchDagWithMultipleParallelJobs() throws IOException,
InterruptedException, URISyntaxException {
Review Comment:
when there are multiple jobs to launch, they are launched by adding
ReevaluateDagAction.
when there is only one, it is launched directly.
so in 1st case, we test how many times ReevaluateDagAction was added, while
in 2nd case, we check spec producer for job addition.
Issue Time Tracking
---
Worklog Id: (was: 922637)
Time Spent: 3h (was: 2h 50m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 038d23d4349 Updating config from bot
038d23d4349 is described below
commit 038d23d4349a8b4bf694c721dfc724ff1048621a
Author: github-actions
AuthorDate: Fri Jun 7 19:05:27 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31548.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31548.json
b/scripts/ci/pr-bot/state/pr-state/pr-31548.json
new file mode 100644
index 000..321b31c0f05
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31548.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "tvalentyn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 06f37f2674a Updating config from bot 06f37f2674a is described below commit 06f37f2674afa0c576c2140f3c0c2a950c9254b6 Author: github-actions AuthorDate: Fri Jun 7 19:05:29 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 5625eee28a5..7e655193697 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -5,7 +5,7 @@ "yeandy": 1665802753763, "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548, -"tvalentyn": 1717637497698, +"tvalentyn": 1717787126513, "pabloem": 1681281324703, "y1chi": 1667002607045, "damccorm": 1717700753576,
(maven-clean-plugin) branch dependabot/maven/org.junit.jupiter-junit-jupiter-api-5.10.2 deleted (was 0c746b5)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.junit.jupiter-junit-jupiter-api-5.10.2 in repository https://gitbox.apache.org/repos/asf/maven-clean-plugin.git was 0c746b5 Bump org.junit.jupiter:junit-jupiter-api from 5.9.3 to 5.10.2 The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922635=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922635
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 19:02
Start Date: 07/Jun/24 19:02
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631590659
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
```// Usually reevaluate dag action is created by JobStatusMonitor when a
finished job status is available,
// but when reevaluate/resume/launch dag proc found multiple parallel
jobs to run next, it creates reevaluate
// dag actions for each of those parallel job and in this scenario
there is no job status available.
// If the job status is not present, this job was never launched,
submit it now.```
Issue Time Tracking
---
Worklog Id: (was: 922635)
Time Spent: 2h 40m (was: 2.5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922636=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922636
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 19:02
Start Date: 07/Jun/24 19:02
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631590659
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
```
// Usually reevaluate dag action is created by JobStatusMonitor when a
finished job status is available,
// but when reevaluate/resume/launch dag proc found multiple parallel
jobs to run next, it creates reevaluate
// dag actions for each of those parallel job and in this scenario
there is no job status available.
// If the job status is not present, this job was never launched,
submit it now.```
Issue Time Tracking
---
Worklog Id: (was: 922636)
Time Spent: 2h 50m (was: 2h 40m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922634=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922634
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 19:00
Start Date: 07/Jun/24 19:00
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631589630
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
It comes down to how we define the responsibilities of launch and reevaluate
dagprocs
I see LaunchDagProc's main responsibility to get do the preparation of
starting a dag, which includes **creating** a dag out of a flowConfig, and
bring the flow into a launchable state.
I imagine it having **only one** LaunchDagProc in the whole process. If we
create multiple LaunchDagProcs we may have to again add clumsy code to handle
cases and possible do duplicate "preparation" work.
ReevaluateDagProc comes handy here because it is not a
"LaunchNextJobsDagProc", it may assume additional responsibility of doing a
broader reevaluation and may decide to launch the current job also when needed.
Issue Time Tracking
---
Worklog Id: (was: 922634)
Time Spent: 2.5h (was: 2h 20m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MCLEAN-119) Upgrade Parent to 42
[ https://issues.apache.org/jira/browse/MCLEAN-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853255#comment-17853255 ] ASF GitHub Bot commented on MCLEAN-119: --- slawekjaranowski merged PR #44: URL: https://github.com/apache/maven-clean-plugin/pull/44 > Upgrade Parent to 42 > > > Key: MCLEAN-119 > URL: https://issues.apache.org/jira/browse/MCLEAN-119 > Project: Maven Clean Plugin > Issue Type: Dependency upgrade >Reporter: Slawomir Jaranowski >Assignee: Slawomir Jaranowski >Priority: Major > Fix For: 3.4.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HADOOP-19116) update to zookeeper client 3.8.4 due to CVE-2024-23944
[
https://issues.apache.org/jira/browse/HADOOP-19116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853254#comment-17853254
]
ASF GitHub Bot commented on HADOOP-19116:
-
hadoop-yetus commented on PR #6675:
URL: https://github.com/apache/hadoop/pull/6675#issuecomment-2155363184
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 7m 59s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +0 :ok: | xmllint | 0m 0s | | xmllint was not available. |
| +0 :ok: | shelldocs | 0m 0s | | Shelldocs was not available. |
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
_ branch-3.3 Compile Tests _ |
| +0 :ok: | mvndep | 13m 55s | | Maven dependency ordering for branch |
| +1 :green_heart: | mvninstall | 29m 43s | | branch-3.3 passed |
| +1 :green_heart: | compile | 13m 25s | | branch-3.3 passed |
| +1 :green_heart: | mvnsite | 20m 24s | | branch-3.3 passed |
| +1 :green_heart: | javadoc | 5m 51s | | branch-3.3 passed |
| +1 :green_heart: | shadedclient | 33m 23s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 30s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 29m 1s | | the patch passed |
| +1 :green_heart: | compile | 12m 49s | | the patch passed |
| +1 :green_heart: | javac | 12m 49s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 16m 46s | | the patch passed |
| +1 :green_heart: | shellcheck | 0m 0s | | No new issues. |
| +1 :green_heart: | javadoc | 5m 26s | | the patch passed |
| +1 :green_heart: | shadedclient | 33m 31s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| -1 :x: | unit | 774m 41s |
[/patch-unit-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6675/5/artifact/out/patch-unit-root.txt)
| root in the patch passed. |
| -1 :x: | asflicense | 1m 4s |
[/results-asflicense.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6675/5/artifact/out/results-asflicense.txt)
| The patch generated 1 ASF License warnings. |
| | | 989m 17s | | |
| Reason | Tests |
|---:|:--|
| Failed junit tests |
hadoop.hdfs.server.namenode.TestBlockPlacementPolicyRackFaultTolerant |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.TestReconstructStripedFileWithRandomECPolicy |
| | hadoop.hdfs.TestUnsetAndChangeDirectoryEcPolicy |
| | hadoop.hdfs.server.namenode.TestFsck |
| | hadoop.hdfs.tools.TestECAdmin |
| | hadoop.hdfs.TestDFSStripedInputStream |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.TestErasureCodingExerciseAPIs |
| | hadoop.hdfs.TestReconstructStripedFile |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
| | hadoop.hdfs.TestReadStripedFileWithDNFailure |
| | hadoop.hdfs.TestSetrepIncreasing |
| |
hadoop.hdfs.server.namenode.sps.TestStoragePolicySatisfierWithStripedFile |
| | hadoop.hdfs.server.namenode.TestNetworkTopologyServlet |
| | hadoop.hdfs.TestErasureCodingPolicyWithSnapshot |
| |
hadoop.hdfs.tools.offlineImageViewer.TestOfflineImageViewerWithStripedBlocks |
| | hadoop.hdfs.TestErasureCodingPolicies |
| | hadoop.hdfs.TestDistributedFileSystemWithECFile |
| | hadoop.hdfs.server.namenode.TestNameEditsConfigs |
| | hadoop.hdfs.server.datanode.TestDataNodeReconfiguration |
| | hadoop.hdfs.server.namenode.TestAddStripedBlocks |
| | hadoop.hdfs.TestListFilesInFileContext |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.TestHFlush |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestDFSRename |
| | hadoop.hdfs.server.datanode.TestLargeBlockReport |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.TestRestartDFS
(commons-fileupload) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now b83a4c08)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-fileupload.git at b83a4c08 Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 28259ca2ee8 Updating config from bot
28259ca2ee8 is described below
commit 28259ca2ee82c0b9e947a38f48f48c5771610fa3
Author: github-actions
AuthorDate: Fri Jun 7 18:52:09 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31549.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31549.json
b/scripts/ci/pr-bot/state/pr-state/pr-31549.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31549.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922633=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922633
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:48
Start Date: 07/Jun/24 18:48
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631579053
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -134,6 +154,7 @@ private void setStatus(DagManagementStateStore
dagManagementStateStore,
for (Dag.DagNode node : dag.getNodes()) {
if (node.getValue().getId().equals(dagNodeId)) {
node.getValue().setExecutionStatus(executionStatus);
+dagManagementStateStore.addDagNodeState(node, getDagId());
Review Comment:
i realized i was not storing the dag node after modifying it.
however, i realized after making the change that we do not use execution
status from dag node, we always call jobstatusretriever to get it.
that also questions the existence of execution status inside a dag node.
Issue Time Tracking
---
Worklog Id: (was: 922633)
Time Spent: 2h 20m (was: 2h 10m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(pinot) branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 updated (fc7eb4a0a8 -> db3a4c5918)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/net.openhft-chronicle-core-2.26ea0 in repository https://gitbox.apache.org/repos/asf/pinot.git discard fc7eb4a0a8 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 add 3d07b90bd7 Fix Logging Location for CPU-Based Query Killing (#13318) add f4a4bf389a Custom configuration property reader for segment metadata files (#12440) add 6290bc001a Interning for OnHeapByteDictionary (#12342) add ba2c373637 Bump org.apache.maven.plugins:maven-checkstyle-plugin (#13321) add 2c59fbec96 Bump pulsar.version from 3.2.3 to 3.3.0 (#13322) add b602df5271 Bump software.amazon.awssdk:bom from 2.25.66 to 2.25.67 (#13323) add 1d1d25dc0f Update getValidDocIdsMetadataFromServer to make call in batches to servers and other bug fixes (#13314) add fb64ec29e9 Fix few metric rules which were affected by the database prefix handling (#13290) add 0f92742b45 Multi stage metrics (#13035) add fc358a87fc Add offset based lag metrics (#13298) add 4356583592 [Multi-stage] Clean up RelNode to Operator handling (#13325) add 1e0dc300f9 Bump com.nimbusds:nimbus-jose-jwt from 9.39.3 to 9.40 (#13334) add cae5e4e68a Bump lucene.version from 9.10.0 to 9.11.0 (#13328) add dbeeaaf25e Add a benchmark to measure multi-stage block serde cost (#13336) add b78de5375a Bump io.github.hakky54:sslcontext-kickstart-for-netty (#13329) add 0137e32de0 Bump software.amazon.awssdk:bom from 2.25.67 to 2.25.68 (#13330) add 5d1a04288c Bump com.microsoft.azure:msal4j from 1.15.0 to 1.15.1 (#13331) add 18a6a0df85 Bump com.azure:azure-core-http-netty from 1.15.0 to 1.15.1 (#13332) add dae4fbd48e Bump com.azure:azure-core from 1.49.0 to 1.49.1 (#1) add db3a4c5918 Bump net.openhft:chronicle-core from 2.25ea15 to 2.25.16 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (fc7eb4a0a8) \ N -- N -- N refs/heads/dependabot/maven/net.openhft-chronicle-core-2.26ea0 (db3a4c5918) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .../jmx_prometheus_javaagent/configs/broker.yml| 12 + .../configs/controller.yml | 7 + .../jmx_prometheus_javaagent/configs/server.yml| 25 +- .../apache/pinot/common/metrics/ServerGauge.java | 3 +- .../apache/pinot/common/metrics/ServerMeter.java | 41 +- .../apache/pinot/common/metrics/ServerTimer.java | 31 +- pinot-common/src/main/proto/expressions.proto | 6 +- pinot-common/src/main/proto/plan.proto | 229 +++ .../pinot/common/utils/FALFInternerTest.java | 136 ++-- .../api/resources/PinotTableRestletResource.java | 8 +- .../controller/util/CompletionServiceHelper.java | 24 +- .../util/ServerSegmentMetadataReader.java | 32 +- .../pinot/controller/util/TableMetadataReader.java | 4 +- .../PerQueryCPUMemAccountantFactory.java | 6 +- .../apache/pinot/core/common/MinionConstants.java | 5 + .../manager/realtime/IngestionDelayTracker.java| 92 ++- .../realtime/RealtimeSegmentDataManager.java | 16 +- .../manager/realtime/RealtimeTableDataManager.java | 10 +- .../realtime/IngestionDelayTrackerTest.java| 65 ++ .../org/apache/pinot/perf/BenchmarkDataBlock.java | 244 +++ pinot-plugins/pinot-file-system/pinot-adls/pom.xml | 6 +- .../UpsertCompactionTaskGenerator.java | 12 +- .../calcite/rel/logical/PinotLogicalExchange.java | 1 - .../rel/logical/PinotLogicalSortExchange.java | 1 - .../calcite/rel/logical/PinotRelExchangeType.java | 4 +- .../PinotAggregateExchangeNodeInsertRule.java | 8 +- .../rel/rules/PinotJoinToDynamicBroadcastRule.java | 2 +- .../org/apache/pinot/query/QueryEnvironment.java | 3 +- .../query/parser/CalciteRexExpressionParser.java | 49 +- .../explain/PhysicalExplainPlanVisitor.java| 12 +- .../query/planner/logical/LiteralValueNode.java| 7 +- .../planner/logical/PinotLogicalQueryPlanner.java | 57 +- .../query/planner/logical/PlanFragmenter.java | 25 +- .../planner/logical/RelToPlanNodeConverter.java| 257 --- .../pinot/query/planner/logical/RexExpression.java | 84 ++- .../query/planner/logical/RexExpressionUtils.java | 13 +- .../planner/logical/Shuffl
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922630=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922630
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:45
Start Date: 07/Jun/24 18:45
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631576116
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
adding in next commit, search by "Usually reevaluate dag action"
Issue Time Tracking
---
Worklog Id: (was: 922630)
Time Spent: 2h 10m (was: 2h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(doris-website) branch asf-site updated (b23f720e79 -> f0a6be3948)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch asf-site in repository https://gitbox.apache.org/repos/asf/doris-website.git discard b23f720e79 Automated deployment with doris branch @ ce6bb0b7b8e27bdc15788c270b1b055ddb69d302 new f0a6be3948 Automated deployment with doris branch @ ce6bb0b7b8e27bdc15788c270b1b055ddb69d302 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (b23f720e79) \ N -- N -- N refs/heads/asf-site (f0a6be3948) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: docs/1.2/search-index.json | 2 +- docs/2.0/search-index.json | 2 +- docs/dev/search-index.json | 2 +- search-index.json| 2 +- zh-CN/docs/1.2/search-index.json | 2 +- zh-CN/docs/2.0/search-index.json | 2 +- zh-CN/docs/dev/search-index.json | 2 +- zh-CN/search-index.json | 2 +- 8 files changed, 8 insertions(+), 8 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@doris.apache.org For additional commands, e-mail: commits-h...@doris.apache.org
(commons-fileupload) branch dependabot/maven/org.apache.maven.plugins-maven-javadoc-plugin-3.7.0 created (now d53b3469)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/maven/org.apache.maven.plugins-maven-javadoc-plugin-3.7.0 in repository https://gitbox.apache.org/repos/asf/commons-fileupload.git at d53b3469 Bump org.apache.maven.plugins:maven-javadoc-plugin from 3.6.3 to 3.7.0 No new revisions were added by this update.
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922626=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922626
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:14
Start Date: 07/Jun/24 18:14
Worklog Time Spent: 10m
Work Description: arjun4084346 commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631547273
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/DagProcUtils.java:
##
@@ -55,6 +56,36 @@
*/
@Slf4j
public class DagProcUtils {
+
+ /**
+ * If there is a single job to run next, it runs it. If there are multiple
jobs to run, it creates a
+ * {@link
org.apache.gobblin.service.modules.orchestration.DagActionStore.DagActionType#REEVALUATE}
dag action for
+ * each of them and those jobs will be launched in respective Reevaluate dag
proc.
+ */
+ public static void submitNextNodes(DagManagementStateStore
dagManagementStateStore, Dag dag,
+ DagManager.DagId dagId) throws IOException {
+Set> nextNodes =
DagManagerUtils.getNext(dag);
+
+if (nextNodes.size() > 1) {
+ handleMultipleJobs(dagManagementStateStore, nextNodes);
+}
+
+//Submit jobs from the dag ready for execution.
+for (Dag.DagNode dagNode : nextNodes) {
+ DagProcUtils.submitJobToExecutor(dagManagementStateStore, dagNode,
dagId);
+ log.info("Submitted job {} for dagId {}",
DagManagerUtils.getJobName(dagNode), dagId);
+}
Review Comment:
oh i missed the return statement from line 71 during the refactoring
Issue Time Tracking
---
Worklog Id: (was: 922626)
Time Spent: 2h (was: 1h 50m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2aaa8666344 Updating config from bot
2aaa8666344 is described below
commit 2aaa86663442aa87246c2018f2f3eca83d88d487
Author: github-actions
AuthorDate: Fri Jun 7 18:08:11 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index c16fa320d30..b19cb2330f8 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1717680538242,
-"Abacn": 1717470420388
+"Abacn": 1717783688131
}
}
\ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 84009227969 Updating config from bot 84009227969 is described below commit 84009227969716c21fa8cfb9996d8ba45b2ad7e7 Author: github-actions AuthorDate: Fri Jun 7 18:08:10 2024 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json b/scripts/ci/pr-bot/state/reviewers-for-label-java.json index 633c5575121..36989f83f2f 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json @@ -6,7 +6,7 @@ "robertwb": 1717560462967, "kileys": 1674428678843, "apilloud": 1678822446183, -"Abacn": 1717519044263, +"Abacn": 1717783688131, "bvolpato": 1712595969392, "m-trieu": 1717680538242, "damondouglas": 1717589630200
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b4a40663695 Updating config from bot
b4a40663695 is described below
commit b4a4066369547e589586eb84bfe7d2b813eb9d0e
Author: github-actions
AuthorDate: Fri Jun 7 18:08:09 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31490.json | 9 ++---
1 file changed, 6 insertions(+), 3 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31490.json
b/scripts/ci/pr-bot/state/pr-state/pr-31490.json
index 242a48d7d3b..8ea654a1e14 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-31490.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31490.json
@@ -1,7 +1,10 @@
{
- "commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "Abacn",
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[jira] [Commented] (TIKA-4243) tika configuration overhaul
[ https://issues.apache.org/jira/browse/TIKA-4243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853238#comment-17853238 ] ASF GitHub Bot commented on TIKA-4243: -- tballison opened a new pull request, #1805: URL: https://github.com/apache/tika/pull/1805 Thanks for your contribution to [Apache Tika](https://tika.apache.org/)! Your help is appreciated! Before opening the pull request, please verify that * there is an open issue on the [Tika issue tracker](https://issues.apache.org/jira/projects/TIKA) which describes the problem or the improvement. We cannot accept pull requests without an issue because the change wouldn't be listed in the release notes. * the issue ID (`TIKA-`) - is referenced in the title of the pull request - and placed in front of your commit messages surrounded by square brackets (`[TIKA-] Issue or pull request title`) * commits are squashed into a single one (or few commits for larger changes) * Tika is successfully built and unit tests pass by running `mvn clean test` * there should be no conflicts when merging the pull request branch into the *recent* `main` branch. If there are conflicts, please try to rebase the pull request branch on top of a freshly pulled `main` branch * if you add new module that downstream users will depend upon add it to relevant group in `tika-bom/pom.xml`. We will be able to faster integrate your pull request if these conditions are met. If you have any questions how to fix your problem or about using Tika in general, please sign up for the [Tika mailing list](http://tika.apache.org/mail-lists.html). Thanks! > tika configuration overhaul > --- > > Key: TIKA-4243 > URL: https://issues.apache.org/jira/browse/TIKA-4243 > Project: Tika > Issue Type: New Feature > Components: config >Affects Versions: 3.0.0 >Reporter: Nicholas DiPiazza >Priority: Major > Fix For: 3.0.0 > > > In 3.0.0 when dealing with Tika, it would greatly help to have a Typed > Configuration schema. > In 3.x can we remove the old way of doing configs and replace with Json > Schema? > Json Schema can be converted to Pojos using a maven plugin > [https://github.com/joelittlejohn/jsonschema2pojo] > This automatically creates a Java Pojo model we can use for the configs. > This can allow for the legacy tika-config XML to be read and converted to the > new pojos easily using an XML mapper so that users don't have to use JSON > configurations yet if they do not want. > When complete, configurations can be set as XML, JSON or YAML > tika-config.xml > tika-config.json > tika-config.yaml > Replace all instances of tika config annotations that used the old syntax, > and replace with the Pojo model serialized from the xml/json/yaml. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922625=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922625
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:03
Start Date: 07/Jun/24 18:03
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631514247
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
it may be too subtle (and even potentially mask errors) to designate that a
REEVALUATE DagAction with no job status actually happens to encode the need to
launch the job.
to step back for a moment: direct recursive handling would merely decompose
multi-job LAUNCH into multiple LAUNCH DagActions and multi-job REEVALUATE into
multiple REEVALUATE DagActions. why not transparently handle the multi-job
cases in this way, rather than pressing REEVALUATE into service for
multi-LAUNCH?
Issue Time Tracking
---
Worklog Id: (was: 922625)
Time Spent: 1h 50m (was: 1h 40m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922624=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922624
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:03
Start Date: 07/Jun/24 18:03
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631506035
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
somewhere let's add a comment describing the circumstance of a dag node w/o
job status. it's especially important as it basically reflects a "special" /
"sentinel" value.
where should that live?
Issue Time Tracking
---
Worklog Id: (was: 922624)
Time Spent: 1h 40m (was: 1.5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(commons-jxpath) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 050775a)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-jxpath.git at 050775a Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922623=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922623
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:01
Start Date: 07/Jun/24 18:01
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631530614
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/LaunchDagProcTest.java:
##
@@ -94,29 +93,50 @@ public void launchDag()
flowCompilationValidationHelper);
launchDagProc.process(this.dagManagementStateStore);
-int expectedNumOfSavingDagNodeStates = 1; // = number of start nodes
-Assert.assertEquals(expectedNumOfSavingDagNodeStates,
+int numOfLaunchedJobs = 1; // = number of start nodes
+Assert.assertEquals(numOfLaunchedJobs,
Mockito.mockingDetails(this.dagManagementStateStore).getInvocations().stream()
.filter(a ->
a.getMethod().getName().equals("addDagNodeState")).count());
-Mockito.verify(this.dagManagementStateStore, Mockito.times(1))
+Mockito.verify(this.dagManagementStateStore,
Mockito.times(numOfLaunchedJobs))
.addFlowDagAction(any(), any(), any(),
eq(DagActionStore.DagActionType.ENFORCE_FLOW_FINISH_DEADLINE));
}
+ @Test
+ public void launchDagWithMultipleParallelJobs() throws IOException,
InterruptedException, URISyntaxException {
Review Comment:
I see validation of spec executors in `ReevaluateDagProcTest`, but not
here... shouldn't this also?
(it would be a self-documenting means to indicate whether
[this](https://github.com/apache/gobblin/pull/3965#discussion_r1631491883) was
intended.)
Issue Time Tracking
---
Worklog Id: (was: 922623)
Time Spent: 1.5h (was: 1h 20m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922622=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922622
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:00
Start Date: 07/Jun/24 18:00
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631527206
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProcTest.java:
##
@@ -206,4 +207,84 @@ public void testNoNextJobToRun() throws Exception {
Assert.assertEquals(Mockito.mockingDetails(this.dagManagementStateStore).getInvocations().stream()
.filter(a ->
a.getMethod().getName().equals("deleteDagAction")).count(), 1);
}
+
+ @Test
+ public void testCurrentJobToRun() throws Exception {
+String flowName = "fn3";
+Dag dag = DagManagerTest.buildDag("1", flowExecutionId,
DagManager.FailureOption.FINISH_ALL_POSSIBLE.name(),
+2, "user5", ConfigFactory.empty()
+.withValue(ConfigurationKeys.FLOW_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+.withValue(ConfigurationKeys.FLOW_NAME_KEY,
ConfigValueFactory.fromAnyRef(flowName))
+.withValue(ConfigurationKeys.JOB_GROUP_KEY,
ConfigValueFactory.fromAnyRef(flowGroup))
+);
+List> startDagNodes = dag.getStartNodes();
+List> specProducers = dag.getNodes().stream().map(n -> {
+ try {
+return DagManagerUtils.getSpecProducer(n);
+ } catch (ExecutionException | InterruptedException e) {
+throw new RuntimeException(e);
+ }
+}).collect(Collectors.toList());
+
+doReturn(Optional.of(dag)).when(dagManagementStateStore).getDag(any());
+doReturn(new ImmutablePair<>(Optional.of(startDagNodes.get(0)),
Optional.empty()))
+.when(dagManagementStateStore).getDagNodeWithJobStatus(any());
+
+ReevaluateDagProc
+reEvaluateDagProc = new ReevaluateDagProc(new ReevaluateDagTask(new
DagActionStore.DagAction(flowGroup, flowName,
+String.valueOf(flowExecutionId), "job0",
DagActionStore.DagActionType.REEVALUATE), null,
+dagManagementStateStore));
+reEvaluateDagProc.process(dagManagementStateStore);
+
+// only the current job should have run
+Mockito.verify(specProducers.get(0), Mockito.times(1)).addSpec(any());
+Mockito.verify(specProducers.get(1), Mockito.never()).addSpec(any());
Review Comment:
being a list, shouldn't you verify that *all* elems after `0` are never
called?
Issue Time Tracking
---
Worklog Id: (was: 922622)
Time Spent: 1h 20m (was: 1h 10m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922621=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922621
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 18:00
Start Date: 07/Jun/24 18:00
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631519026
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -134,6 +154,7 @@ private void setStatus(DagManagementStateStore
dagManagementStateStore,
for (Dag.DagNode node : dag.getNodes()) {
if (node.getValue().getId().equals(dagNodeId)) {
node.getValue().setExecutionStatus(executionStatus);
+dagManagementStateStore.addDagNodeState(node, getDagId());
Review Comment:
was this merely omitted previously or is there only now a need for it after
having `ReevaluateDagProc` do double-duty of launching remaining multi-jobs in
addition to reevaluation of completed jobs?
Issue Time Tracking
---
Worklog Id: (was: 922621)
Time Spent: 1h 10m (was: 1h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922620=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922620
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 17:59
Start Date: 07/Jun/24 17:59
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631514247
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
it may be too subtle (and even potentially mask errors) to designate that a
REEVALUATE DagAction with no job status actually happens to encode the need to
launch the job.
direct recursive handling would merely decompose multi-job LAUNCH into
multiple LAUNCH DagActions and multi-job REEVALUATE into multiple REEVALUATE
DagActions. why not transparently handle the multi-job cases in this way,
rather than pressing REEVALUATE for multi-LAUNCH?
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
it may be too subtle (and even potentially mask errors) to designate that a
REEVALUATE DagAction with no job status actually happens to encode the need to
launch the job.
direct recursive handling would merely decompose multi-job LAUNCH into
multiple LAUNCH DagActions and multi-job REEVALUATE into multiple REEVALUATE
DagActions. why not transparently handle the multi-job cases in this way,
rather than pressing REEVALUATE into service for multi-LAUNCH?
Issue Time Tracking
---
Worklog Id: (was: 922620)
Time Spent: 1h (was: 50m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 1h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922618
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 17:58
Start Date: 07/Jun/24 17:58
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631514247
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
it may be too subtle (and even potentially mask errors) to designate that a
REEVALUATE DagAction with no job status actually happens to encode the need to
launch the job.
directly recursive handling would decompose multi-job LAUNCH into multiple
LAUNCH DagActions and multi-job REEVALUATE into multiple REEVALUATE DagActions.
why not transparently handle the multi-job cases in this way?
Issue Time Tracking
---
Worklog Id: (was: 922618)
Time Spent: 40m (was: 0.5h)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 40m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922619=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922619
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 17:58
Start Date: 07/Jun/24 17:58
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631514247
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -89,11 +86,29 @@ protected void act(DagManagementStateStore
dagManagementStateStore, Pair dagNode =
dagNodeWithJobStatus.getLeft().get();
+
+if (!dagNodeWithJobStatus.getRight().isPresent()) {
+ // if the job status is not present, this job was never launched, submit
it now
+ submitJobForThisDagNode(dagManagementStateStore, dagNode);
+ return;
+}
Review Comment:
it may be too subtle (and even potentially mask errors) to designate that a
REEVALUATE DagAction with no job status actually happens to encode the need to
launch the job.
direct recursive handling would merely decompose multi-job LAUNCH into
multiple LAUNCH DagActions and multi-job REEVALUATE into multiple REEVALUATE
DagActions. why not transparently handle the multi-job cases in this way?
Issue Time Tracking
---
Worklog Id: (was: 922619)
Time Spent: 50m (was: 40m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 50m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922615=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922615
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 17:57
Start Date: 07/Jun/24 17:57
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631506035
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
somewhere let's add a comment describing circumstance of a dag node w/o job
status. that's because it basically represents a "special" / "sentinel" value.
where should that live?
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
+ ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
+ if
(FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
+setStatus(dagManagementStateStore,
dagNodeWithJobStatus.getLeft().get(), executionStatus);
Review Comment:
seems better to perform `setStatus` in `act`, so it's not spread out between
two methods
Issue Time Tracking
---
Worklog Id: (was: 922615)
Time Spent: 20m (was: 10m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922617=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922617
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 17:57
Start Date: 07/Jun/24 17:57
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631510375
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
+ ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
+ if
(FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
+setStatus(dagManagementStateStore,
dagNodeWithJobStatus.getLeft().get(), executionStatus);
Review Comment:
seems better to perform `setStatus` in `act`, so it's not spread between two
methods
Issue Time Tracking
---
Worklog Id: (was: 922617)
Time Spent: 0.5h (was: 20m)
> divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
> --
>
> Key: GOBBLIN-2017
> URL: https://issues.apache.org/jira/browse/GOBBLIN-2017
> Project: Apache Gobblin
> Issue Type: Task
>Reporter: Arjun Singh Bora
>Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> divide multiple job launches in a LaunchDagProc into multiple
> LaunchDagActions for two reasons
> 1) it will then spend less time in each dag proc processing and have more
> chance to complete the operation in the lease time
> 2) handling partial job submissions in one LaunchDagProc sending N jobs is
> difficult
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Work logged] (GOBBLIN-2017) divide multiple job launches in a LaunchDagProc into multiple LaunchDagActions
[
https://issues.apache.org/jira/browse/GOBBLIN-2017?focusedWorklogId=922614=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-922614
]
ASF GitHub Bot logged work on GOBBLIN-2017:
---
Author: ASF GitHub Bot
Created on: 07/Jun/24 17:56
Start Date: 07/Jun/24 17:56
Worklog Time Spent: 10m
Work Description: phet commented on code in PR #3965:
URL: https://github.com/apache/gobblin/pull/3965#discussion_r1631510375
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
+ ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
+ if
(FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
+setStatus(dagManagementStateStore,
dagNodeWithJobStatus.getLeft().get(), executionStatus);
Review Comment:
seems better to perform this in `act`, so it's not spread out between two
methods
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/DagManagementTaskStreamImpl.java:
##
@@ -66,7 +66,7 @@
* the flow action using {@link DagActionReminderScheduler} to reattempt the
lease after the current leaseholder's grant
* would have expired. The {@link DagActionReminderScheduler} is used in the
non multi-active execution configuration as
* well to utilize reminders for a single {@link DagManagementTaskStreamImpl}
case as well.
- * Note that if multi-active execution is NOT enabled, then all flow action
events are selected by
+ * Note that if multi-active execution is NOT enabsled, then all flow action
events are selected by
Review Comment:
misspelling
##
gobblin-service/src/main/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProc.java:
##
@@ -60,21 +59,19 @@ protected Pair>,
Optional> ini
Pair>, Optional>
dagNodeWithJobStatus =
dagManagementStateStore.getDagNodeWithJobStatus(this.dagNodeId);
-if (!dagNodeWithJobStatus.getLeft().isPresent() ||
!dagNodeWithJobStatus.getRight().isPresent()) {
+if (!dagNodeWithJobStatus.getLeft().isPresent()) {
// this is possible when MALA malfunctions and a duplicated reevaluate
dag proc is launched for a dag node that is
// already "reevaluated" and cleaned up.
return ImmutablePair.of(Optional.empty(), Optional.empty());
}
-ExecutionStatus executionStatus =
ExecutionStatus.valueOf(dagNodeWithJobStatus.getRight().get().getEventName());
-if
(!FlowStatusGenerator.FINISHED_STATUSES.contains(executionStatus.name())) {
- log.warn("Job status for dagNode {} is {}. Re-evaluate dag action should
have been created only for finished status - {}",
- dagNodeId, executionStatus, FlowStatusGenerator.FINISHED_STATUSES);
- // this may happen if adding job status in the store failed after adding
a ReevaluateDagAction in KafkaJobStatusMonitor
- throw new RuntimeException(String.format("Job status %s is not final for
job %s", executionStatus, getDagId()));
+if (dagNodeWithJobStatus.getRight().isPresent()) {
Review Comment:
somewhere let's add a comment describing circumstance of a dag node w/o job
status. where should that live?
##
gobblin-service/src/test/java/org/apache/gobblin/service/modules/orchestration/proc/ReevaluateDagProcTest.java:
##
@@ -206,4 +207,84 @@ public void testNoNextJobToRun() throws Exception {
Assert.asser
[jira] [Commented] (TINKERPOP-3028) Convert translators to make use of the grammar
[
https://issues.apache.org/jira/browse/TINKERPOP-3028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853233#comment-17853233
]
ASF GitHub Bot commented on TINKERPOP-3028:
---
spmallette merged PR #2640:
URL: https://github.com/apache/tinkerpop/pull/2640
> Convert translators to make use of the grammar
> --
>
> Key: TINKERPOP-3028
> URL: https://issues.apache.org/jira/browse/TINKERPOP-3028
> Project: TinkerPop
> Issue Type: Improvement
> Components: translator
>Affects Versions: 3.6.6
>Reporter: Stephen Mallette
>Assignee: Stephen Mallette
>Priority: Major
>
> {{Translator}} infrastructure uses bytecode and reflection which has a number
> of technical imperfections and exemptions for it to work properly. Switching
> to the grammar for translation simplifies the code, removes friction and
> complexity when generating GLV tests, and should perform better. The focus
> for this issue is the Java series of translators which are relied on quite
> heavily.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(beam) branch asf-site updated: Publishing website 2024/06/07 17:37:49 at commit fe54c21
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new f66e82e0082 Publishing website 2024/06/07 17:37:49 at commit fe54c21 f66e82e0082 is described below commit f66e82e0082deba912c46496909d0f43f187b88a Author: runner AuthorDate: Fri Jun 7 17:37:50 2024 + Publishing website 2024/06/07 17:37:49 at commit fe54c21 --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index 3cd466cee18..163d9d7ebb4 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-06-07T00:54:23-04:00/categories/blog/2024-06-07T00:54:23-04:00/blog/2024-06-07T00:54:23-04:00/categories/2024-06-07T00:54:23-04:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.56.0/2024-06-07T11:21:02-04:00/categories/blog/2024-06-07T11:21:02-04:00/blog/2024-06-07T11:21:02-04:00/categories/2024-06-07T11:21:02-04:00/catego [...] \ No newline at end of file
(beam) branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 5a16267ee89 Updating config from bot
5a16267ee89 is described below
commit 5a16267ee89a25ee5f693e9e3499607e82a78161
Author: github-actions
AuthorDate: Fri Jun 7 17:34:19 2024 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-31547.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-31547.json
b/scripts/ci/pr-bot/state/pr-state/pr-31547.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-31547.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[jira] [Commented] (MNG-8141) Model Builder should report if not sure about "fully correct" outcome
[
https://issues.apache.org/jira/browse/MNG-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853226#comment-17853226
]
ASF GitHub Bot commented on MNG-8141:
-
cstamas commented on code in PR #1568:
URL: https://github.com/apache/maven/pull/1568#discussion_r1631500181
##
maven-model-builder/src/main/java/org/apache/maven/model/building/ModelBuildingException.java:
##
@@ -137,7 +137,7 @@ private static String toMessage(String modelId,
List problems) {
writer.print(problems.size());
writer.print((problems.size() == 1) ? " problem was " : " problems
were ");
writer.print("encountered while building the effective model");
-if (modelId != null && !modelId.isEmpty()) {
+if (modelId != null && modelId.length() > 0) {
Review Comment:
I just rolled back unrelated change, this is how it is on `maven-3.9.x` to
make (now) unrelated change gone.
> Model Builder should report if not sure about "fully correct" outcome
> -
>
> Key: MNG-8141
> URL: https://issues.apache.org/jira/browse/MNG-8141
> Project: Maven
> Issue Type: Improvement
> Components: Core
>Reporter: Tamas Cservenak
>Assignee: Tamas Cservenak
>Priority: Major
> Fix For: 4.0.0, 3.9.8, 4.0.0-beta-4
>
>
> ModelBuilder is component building models (POM + interpolating + parent
> inheritance and many many more things), but it should not rely that built
> model "was validated", as it MAY NOT been validated: for "furthest" models it
> builds, like a parent of a some-level-dependency we use MIN level of
> validation (minimal validation).
> Still, while the model builder builds, it relies on several aspects of the
> model, and it should ensure that the "output" (built model) is correct. Model
> Builder hence must be changed in way, that IF it detects any issue _during
> building_ of the model, and IF it appears with even slightest possibility
> that it cannot deliver "correct output", it must add WARNs to model building
> result with proper messages.
> One typical case is when model building injects activated profiles (as they
> can deliver properties and extra plugins and what not) and activation code
> detects a "problem", like for example duplicated profile IDs being used (this
> IS catched by validation, but not on MIN level!), hence, model builder cannot
> guarantee that built model IS correct.
> This change is really only to make Maven emit WARNINGs if project being built
> has some "far POMs" (like parent pom of a dependency of a first level
> dependency, as in reproducer). If model builder cannot be "100% sure" it
> built model correctly, it should be reported. Moreover, WARNs of model
> building result were simply neglected so fat (lost). Having warnings like
> these would reveal "invalid parent POM" early, as it is case in issue
> MNG-8131 for example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (MNG-8141) Model Builder should report if not sure about "fully correct" outcome
[
https://issues.apache.org/jira/browse/MNG-8141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17853225#comment-17853225
]
ASF GitHub Bot commented on MNG-8141:
-
michael-o commented on code in PR #1568:
URL: https://github.com/apache/maven/pull/1568#discussion_r1631493948
##
maven-model-builder/src/main/java/org/apache/maven/model/building/ModelBuildingException.java:
##
@@ -137,7 +137,7 @@ private static String toMessage(String modelId,
List problems) {
writer.print(problems.size());
writer.print((problems.size() == 1) ? " problem was " : " problems
were ");
writer.print("encountered while building the effective model");
-if (modelId != null && !modelId.isEmpty()) {
+if (modelId != null && modelId.length() > 0) {
Review Comment:
That is weird
> Model Builder should report if not sure about "fully correct" outcome
> -
>
> Key: MNG-8141
> URL: https://issues.apache.org/jira/browse/MNG-8141
> Project: Maven
> Issue Type: Improvement
> Components: Core
>Reporter: Tamas Cservenak
>Assignee: Tamas Cservenak
>Priority: Major
> Fix For: 4.0.0, 3.9.8, 4.0.0-beta-4
>
>
> ModelBuilder is component building models (POM + interpolating + parent
> inheritance and many many more things), but it should not rely that built
> model "was validated", as it MAY NOT been validated: for "furthest" models it
> builds, like a parent of a some-level-dependency we use MIN level of
> validation (minimal validation).
> Still, while the model builder builds, it relies on several aspects of the
> model, and it should ensure that the "output" (built model) is correct. Model
> Builder hence must be changed in way, that IF it detects any issue _during
> building_ of the model, and IF it appears with even slightest possibility
> that it cannot deliver "correct output", it must add WARNs to model building
> result with proper messages.
> One typical case is when model building injects activated profiles (as they
> can deliver properties and extra plugins and what not) and activation code
> detects a "problem", like for example duplicated profile IDs being used (this
> IS catched by validation, but not on MIN level!), hence, model builder cannot
> guarantee that built model IS correct.
> This change is really only to make Maven emit WARNINGs if project being built
> has some "far POMs" (like parent pom of a dependency of a first level
> dependency, as in reproducer). If model builder cannot be "100% sure" it
> built model correctly, it should be reported. Moreover, WARNs of model
> building result were simply neglected so fat (lost). Having warnings like
> these would reveal "invalid parent POM" early, as it is case in issue
> MNG-8131 for example.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
(commons-exec) branch dependabot/github_actions/github/codeql-action-3.25.8 created (now 43f60aff)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/github/codeql-action-3.25.8 in repository https://gitbox.apache.org/repos/asf/commons-exec.git at 43f60aff Bump github/codeql-action from 3.25.7 to 3.25.8 No new revisions were added by this update.
