Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
>From Ian Maxon :
Ian Maxon has submitted this change. (
https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 )
Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
..
[ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
- user model changes: no
- storage format changes: no
- interface changes: no
Details:
In JDK17, we are encountering module access errors that
appear in the log. This is due to not having the right
access flags added to the JVM
Additionally, the NCService integration tests are failing
due to memory overuse. This patch tweaks the default allocation
to be 50% instead of 60% of the host memory.
Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5
Reviewed-on: https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271
Reviewed-by: Ian Maxon
Reviewed-by: Michael Blow
Integration-Tests: Jenkins
Tested-by: Jenkins
---
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
M
hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/test/java/org/apache/hyracks/control/nc/service/NCServiceTest.java
2 files changed, 45 insertions(+), 3 deletions(-)
Approvals:
Ian Maxon: Looks good to me, but someone else must approve
Michael Blow: Looks good to me, approved
Jenkins: Verified; Verified
Objections:
Anon. E. Moose #1000171: Violations found
diff --git
a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
index 97dc344..96d3ecb 100644
---
a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
+++
b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java
@@ -31,6 +31,7 @@
import java.net.ServerSocket;
import java.net.Socket;
import java.util.ArrayList;
+import java.util.Arrays;
import java.util.Date;
import java.util.List;
import java.util.Map;
@@ -85,6 +86,12 @@
*/
private static OperatingSystemMXBean osMXBean;
+public static final String[] DEFAULT_ADD_OPENS =
+{
"--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED",
+"--add-opens=java.management/sun.management=ALL-UNNAMED",
+"--add-opens=java.base/java.lang=ALL-UNNAMED",
"--add-opens=java.base/java.nio=ALL-UNNAMED",
+
"--add-opens=java.base/java.util.concurrent.locks=ALL-UNNAMED" };
+
private static List buildCommand() throws IOException {
List cList = new ArrayList<>();
@@ -134,12 +141,20 @@
// Sets up memory parameter if it is not specified.
if (!jvmargs.contains("-Xmx")) {
long ramSize = ((com.sun.management.OperatingSystemMXBean)
osMXBean).getTotalPhysicalMemorySize();
-int proportionalRamSize = (int) Math.ceil(0.6 * ramSize / (1024 *
1024));
+int proportionalRamSize = (int) Math.ceil(0.5 * ramSize / (1024 *
1024));
//if under 32bit JVM, use less than 1GB heap by default. otherwise
use proportional ramsize.
int heapSize =
"32".equals(System.getProperty("sun.arch.data.model"))
? (proportionalRamSize <= 1024 ? proportionalRamSize :
1024) : proportionalRamSize;
jvmargs = jvmargs + " -Xmx" + heapSize + "m";
}
+
+// Squelch some module access warnings and errors from JDK9+
+if (!jvmargs.contains("-add-opens")) {
+StringBuilder jvmArgsBuilder = new StringBuilder(jvmargs);
+Arrays.stream(DEFAULT_ADD_OPENS).map(s ->
jvmArgsBuilder.append(s));
+jvmargs = jvmArgsBuilder.toString();
+}
+
env.put("JAVA_OPTS", jvmargs.trim());
LOGGER.info("Setting JAVA_OPTS to " + jvmargs);
}
diff --git
a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/test/java/org/apache/hyracks/control/nc/service/NCServiceTest.java
b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/test/java/org/apache/hyracks/control/nc/service/NCServiceTest.java
index 00d7ca4..bc04320 100644
---
a/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/test/java/org/apache/hyracks/control/nc/service/NCServiceTest.java
+++
b/hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/test/java/org/apache/hyracks/control/nc/service/NCServiceTest.java
@@ -42,7 +42,7 @@
String sizeStr = prefix.substring(0, prefix.length() - 1);
int size = Integer.parseInt(sizeStr);
long ramSize = ((com.sun.management.OperatingSystemMXBean)
osMXBean).getTotalPhysicalMemorySize();
-int base = 1024 * 1024 * 5;
-Assert.assertTrue(size == ramSize * 3 / base +
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
>From Michael Blow : Attention is currently required from: Ian Maxon, Wail Alkowaileet. Michael Blow has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 ) Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. Patch Set 3: Code-Review+2 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Ian Maxon Gerrit-Attention: Wail Alkowaileet Gerrit-Comment-Date: Wed, 08 May 2024 17:36:36 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
Attention is currently required from: Ian Maxon, Wail Alkowaileet, Michael Blow. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 ) Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. Patch Set 3: Contrib-2 Analytics Compatibility Tests Failed https://cbjenkins.page.link/tzn9qdNYMZNZczEN7 : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Ian Maxon Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Wed, 08 May 2024 02:53:11 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
>From Jenkins : Attention is currently required from: Ian Maxon, Wail Alkowaileet, Michael Blow. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 ) Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. Patch Set 3: Integration-Tests+1 Integration Tests Successful https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/489/ : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Ian Maxon Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Wed, 08 May 2024 02:04:28 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
>From Jenkins : Attention is currently required from: Ian Maxon, Wail Alkowaileet, Michael Blow. Jenkins has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 ) Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. Patch Set 3: Integration-Tests-1 Integration Tests Failed https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-trigger/488/ : UNSTABLE -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Ian Maxon Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Wed, 08 May 2024 01:20:54 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
>From Ian Maxon : Attention is currently required from: Wail Alkowaileet, Michael Blow. Ian Maxon has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 ) Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. Patch Set 3: Code-Review+1 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Wed, 08 May 2024 01:10:31 + Gerrit-HasComments: No Gerrit-Has-Labels: Yes Gerrit-MessageType: comment
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
Attention is currently required from: Ian Maxon, Wail Alkowaileet, Michael Blow. Anon. E. Moose #1000171 has posted comments on this change. ( https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 ) Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. Patch Set 3: Analytics Compatibility Compilation Successful https://cbjenkins.page.link/RjBVvW25S78frAXRA : SUCCESS -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Ian Maxon Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Michael Blow Gerrit-Comment-Date: Wed, 08 May 2024 00:57:57 + Gerrit-HasComments: No Gerrit-Has-Labels: No Gerrit-MessageType: comment
Change in asterixdb[trinity]: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags
>From Ian Maxon : Attention is currently required from: Ian Maxon, Wail Alkowaileet, Michael Blow. Hello Wail Alkowaileet, Jenkins, Michael Blow, Anon. E. Moose #1000171, I'd like you to reexamine a change. Please visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 to look at the new patch set (#3). Change subject: [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags .. [ASTERIXDB-3396][CONF] Tweak memory, '-add-opens' flags - user model changes: no - storage format changes: no - interface changes: no Details: In JDK17, we are encountering module access errors that appear in the log. This is due to not having the right access flags added to the JVM Additionally, the NCService integration tests are failing due to memory overuse. This patch tweaks the default allocation to be 50% instead of 60% of the host memory. Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 --- M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/main/java/org/apache/hyracks/control/nc/service/NCService.java M hyracks-fullstack/hyracks/hyracks-control/hyracks-nc-service/src/test/java/org/apache/hyracks/control/nc/service/NCServiceTest.java 2 files changed, 40 insertions(+), 3 deletions(-) git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/71/18271/3 -- To view, visit https://asterix-gerrit.ics.uci.edu/c/asterixdb/+/18271 To unsubscribe, or for help writing mail filters, visit https://asterix-gerrit.ics.uci.edu/settings Gerrit-Project: asterixdb Gerrit-Branch: trinity Gerrit-Change-Id: I13ec2c1063754b93d1fad551362134c0d1c593d5 Gerrit-Change-Number: 18271 Gerrit-PatchSet: 3 Gerrit-Owner: Ian Maxon Gerrit-Reviewer: Anon. E. Moose #1000171 Gerrit-Reviewer: Ian Maxon Gerrit-Reviewer: Jenkins Gerrit-Reviewer: Michael Blow Gerrit-Reviewer: Wail Alkowaileet Gerrit-Attention: Ian Maxon Gerrit-Attention: Wail Alkowaileet Gerrit-Attention: Michael Blow Gerrit-MessageType: newpatchset
