cloud-fan commented on code in PR #59009:
URL: https://github.com/apache/spark/pull/59009#discussion_r4106948456
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset,
int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager
reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the
map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable
this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
+ throw new IllegalStateException("BytesToBytesMap has already been
freed");
+ }
+ if (longArrayRecoveryRequired) {
+ // Only the failed-reset state is recoverable here: the map is empty, so
a fresh
+ // initial-capacity array is correct. `destructiveIterator != null`
means destructive
+ // iteration has begun, after which map operations are illegal.
+ //
+ // allocate() acquires the TaskMemoryManager monitor while this monitor
is held. The memory
+ // manager may call spill() while holding its monitor, so spill() and
anything reachable
+ // from it must not acquire this monitor.
+ synchronized (this) {
Review Comment:
Addressed in 60eea7415ee. Concurrent lazy recovery and all map
synchronization were removed, so lookup no longer allocates while holding a map
monitor and there is no new lock-order dependency.
##########
core/src/main/java/org/apache/spark/unsafe/map/BytesToBytesMap.java:
##########
@@ -587,12 +604,50 @@ public Location lookup(Object keyBase, long keyOffset,
int keyLength, int hash)
return loc;
}
+ /**
+ * Restores the hash array after a {@link #reset()} freed it but its eager
reallocation failed
+ * with an OOM, leaving {@code longArray == null}. A reset() empties the
map, so re-allocating at
+ * {@code initialCapacity} here is correct. If memory is still unavailable
this throws a
+ * SparkOutOfMemoryError instead of exposing the null array.
+ */
+ private void restoreArrayAfterFailedReset() {
+ if (freed) {
Review Comment:
Addressed in 60eea7415ee. Both volatile flags are gone. The hot path now
performs one ordinary read of a single lifecycle state; map mutation is still
prohibited during concurrent safeLookup() calls.
##########
core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala:
##########
@@ -33,27 +34,40 @@ class TestMemoryManager(conf: SparkConf)
private var available = Long.MaxValue
@GuardedBy("this")
private val memoryForTask = mutable.HashMap[Long,
Long]().withDefaultValue(0L)
+ private val beforeNextExecutionMemoryGrant = new AtomicReference[Runnable]()
+
+ /** Runs one test action before the next grant, outside this manager's
accounting lock. */
Review Comment:
Obsolete after 60eea7415ee. The TestMemoryManager hook was removed entirely
along with the concurrent-recovery test.
##########
core/src/test/scala/org/apache/spark/memory/TestMemoryManager.scala:
##########
@@ -33,27 +34,40 @@ class TestMemoryManager(conf: SparkConf)
private var available = Long.MaxValue
@GuardedBy("this")
private val memoryForTask = mutable.HashMap[Long,
Long]().withDefaultValue(0L)
+ private val beforeNextExecutionMemoryGrant = new AtomicReference[Runnable]()
+
+ /** Runs one test action before the next grant, outside this manager's
accounting lock. */
+ def runBeforeNextExecutionMemoryGrant(action: Runnable): Unit = {
Review Comment:
Addressed in 60eea7415ee by removing the shared TestMemoryManager hook
entirely. The new explicit-reset tests need no allocation interleaving hook.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with
longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale
currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later
append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key,
Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value,
Platform.LONG_ARRAY_OFFSET, 8);
+
+ // Force OOM on the reallocation inside reset(), leaving longArray ==
null.
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ // The map must recover: lookup must not NPE, it must report an empty
map, and it must be
+ // usable again (append into a fresh page, then read the value back).
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertFalse(loc.isDefined());
+ Assertions.assertEquals(0, map.numValues());
+ Assertions.assertTrue(
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value,
Platform.LONG_ARRAY_OFFSET, 8));
+ Assertions.assertEquals(1, map.getNumDataPages());
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ Assertions.assertTrue(loc.isDefined());
+ Assertions.assertEquals(42L, Platform.getLong(loc.getValueBase(),
loc.getValueOffset()));
+ Assertions.assertEquals(1, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapWithSemanticKeyOperationsAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap.KeyOperationsFactory caseInsensitiveWithConstantHash =
+ () -> new BytesToBytesMap.KeyOperations() {
+ @Override
+ public int hash(Object base, long offset, int length) {
+ return 0;
+ }
+
+ @Override
+ public boolean equals(
+ Object leftBase,
+ long leftOffset,
+ int leftLength,
+ Object rightBase,
+ long rightOffset,
+ int rightLength) {
+ return Character.toLowerCase(Platform.getByte(leftBase, leftOffset))
==
+ Character.toLowerCase(Platform.getByte(rightBase, rightOffset));
+ }
+ };
+ BytesToBytesMap map = new BytesToBytesMap(
+ taskMemoryManager,
+ blockManager,
+ serializerManager,
+ 256,
+ 0.5,
+ 4000,
+ caseInsensitiveWithConstantHash);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ byte[] storedKey = new byte[8];
+ storedKey[0] = 'A';
+ byte[] equivalentKey = new byte[16];
+ equivalentKey[0] = 'a';
+ byte[] value = new byte[8];
+ Platform.putLong(value, Platform.BYTE_ARRAY_OFFSET, 42L);
+
+ BytesToBytesMap.Location loc =
+ map.lookup(storedKey, Platform.BYTE_ARRAY_OFFSET, storedKey.length);
+ assertFalse(loc.isDefined());
+ assertTrue(loc.append(
+ storedKey,
+ Platform.BYTE_ARRAY_OFFSET,
+ storedKey.length,
+ value,
+ Platform.BYTE_ARRAY_OFFSET,
+ value.length));
+
+ loc = map.lookup(equivalentKey, Platform.BYTE_ARRAY_OFFSET,
equivalentKey.length);
+ assertTrue(loc.isDefined());
+ assertEquals(storedKey.length, loc.getKeyLength());
+ assertEquals(42L, Platform.getLong(loc.getValueBase(),
loc.getValueOffset()));
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedResetRestoresGrowthState() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ // A sub-one-key growth threshold makes canGrowArray observable through
append(): after a
+ // failed grow, no new key is accepted until reset restores the flag.
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
64, 0.01, 4000);
+ try {
+ for (long i = 0; i < 2; i++) {
+ final long[] key = new long[]{i};
+ if (i == 1) {
+ // The data page already exists, so this OOM is consumed by
growAndRehash().
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ }
+ assertTrue(map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8).append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, key, Platform.LONG_ARRAY_OFFSET,
8));
+ }
+
+ final long[] blockedKey = new long[]{2L};
+ assertFalse(map.lookup(blockedKey, Platform.LONG_ARRAY_OFFSET, 8).append(
+ blockedKey,
+ Platform.LONG_ARRAY_OFFSET,
+ 8,
+ blockedKey,
+ Platform.LONG_ARRAY_OFFSET,
+ 8));
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ for (long i = 10; i < 12; i++) {
+ final long[] key = new long[]{i};
+ final long[] value = new long[]{i * 10};
+ BytesToBytesMap.Location loc = map.lookup(key,
Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertTrue(loc.append(
+ key, Platform.LONG_ARRAY_OFFSET, 8, value,
Platform.LONG_ARRAY_OFFSET, 8));
+ loc = map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8);
+ assertTrue(loc.isDefined());
+ assertEquals(i * 10, Platform.getLong(loc.getValueBase(),
loc.getValueOffset()));
+ }
+ assertEquals(2, map.numKeys());
+ assertEquals(2, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedResetPropagatesRepeatedOom() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ final long[] key = new long[]{1L};
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(
+ SparkOutOfMemoryError.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap.Location loc = map.lookup(key,
Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertEquals(0, map.numKeys());
+ assertEquals(0, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterRepeatedFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ final long[] key = new long[]{1L};
+ BytesToBytesMap.Location loc = map.lookup(key,
Platform.LONG_ARRAY_OFFSET, 8);
+ assertFalse(loc.isDefined());
+ assertEquals(0, map.numKeys());
+ assertEquals(0, map.numValues());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void safeLookupAfterFailedReset() throws Exception {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ ExecutorService executor = Executors.newFixedThreadPool(2);
+ CountDownLatch releaseRecoveryAllocation = new CountDownLatch(1);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ final long[] key = new long[]{1L};
+ CountDownLatch recoveryAllocationEntered = new CountDownLatch(1);
+ memoryManager.runBeforeNextExecutionMemoryGrant(() -> {
+ recoveryAllocationEntered.countDown();
+ try {
+ if (!releaseRecoveryAllocation.await(30, TimeUnit.SECONDS)) {
+ throw new AssertionError("Recovery allocation was not released");
+ }
+ } catch (InterruptedException e) {
+ Thread.currentThread().interrupt();
+ throw new AssertionError("Recovery allocation was interrupted", e);
+ }
+ });
+
+ Future<Boolean> firstLookup = executor.submit(() -> {
+ BytesToBytesMap.Location threadLocation = map.new Location();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, threadLocation);
+ return threadLocation.isDefined();
+ });
+ assertTrue(recoveryAllocationEntered.await(10, TimeUnit.SECONDS));
+
+ AtomicReference<Thread> secondWorker = new AtomicReference<>();
+ CountDownLatch secondStarted = new CountDownLatch(1);
+ Future<Boolean> secondLookup = executor.submit(() -> {
+ BytesToBytesMap.Location threadLocation = map.new Location();
+ secondWorker.set(Thread.currentThread());
+ secondStarted.countDown();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, threadLocation);
+ return threadLocation.isDefined();
+ });
+ assertTrue(secondStarted.await(10, TimeUnit.SECONDS));
+
+ // The first lookup is held during allocation. Verify that the second
has reached the
+ // synchronized recovery gate before allowing the first to publish the
replacement array.
+ boolean contended = false;
+ long deadline = System.nanoTime() + TimeUnit.SECONDS.toNanos(10);
+ ThreadMXBean threadBean = ManagementFactory.getThreadMXBean();
+ while (System.nanoTime() < deadline && !secondLookup.isDone()) {
+ ThreadInfo info = threadBean.getThreadInfo(secondWorker.get().getId());
+ if (info != null && info.getThreadState() == Thread.State.BLOCKED &&
+ info.getLockInfo() != null &&
+
info.getLockInfo().getClassName().equals(BytesToBytesMap.class.getName()) &&
+ info.getLockInfo().getIdentityHashCode() ==
System.identityHashCode(map)) {
+ contended = true;
+ break;
+ }
+ TimeUnit.MILLISECONDS.sleep(1);
+ }
+ assertTrue(contended, "Second lookup never contended for recovery");
+ assertFalse(firstLookup.isDone());
+ releaseRecoveryAllocation.countDown();
+
+ assertFalse(firstLookup.get(10, TimeUnit.SECONDS));
+ assertFalse(secondLookup.get(10, TimeUnit.SECONDS));
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(
+ map.getTotalMemoryConsumption(),
taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ BytesToBytesMap.Location location = map.new Location();
+ map.safeLookup(key, Platform.LONG_ARRAY_OFFSET, 8, location, 42);
+ assertFalse(location.isDefined());
+ } finally {
+ releaseRecoveryAllocation.countDown();
+ executor.shutdownNow();
Review Comment:
Obsolete after 60eea7415ee. The executor-based concurrent recovery test was
removed with the lazy-recovery design, so the shutdown race no longer exists.
##########
core/src/test/java/org/apache/spark/unsafe/map/AbstractBytesToBytesMapSuite.java:
##########
@@ -875,4 +879,362 @@ public void freeAfterFailedReset() {
}
}
+ @Test
+ public void lookupAfterFreeDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.free();
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ map.reset();
+ assertEquals(0, map.getNumDataPages());
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void lookupAfterDestructiveIterationDoesNotRestoreArray() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ final long[] key = new long[]{1L};
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ map.destructiveIterator();
+ memoryManager.limit(PAGE_SIZE_BYTES);
+
+ assertThrows(
+ IllegalStateException.class,
+ () -> map.lookup(key, Platform.LONG_ARRAY_OFFSET, 8));
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void resetAfterDestructiveIterationFails() {
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ map.destructiveIterator();
+ assertThrows(IllegalStateException.class, map::reset);
+ assertEquals(0L, taskMemoryManager.getMemoryConsumptionForThisTask());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void arrayAccessorsRecoverAfterFailedReset() {
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ memoryManager.limit(PAGE_SIZE_BYTES);
+ assertEquals(512L, map.getArray().size());
+
+ memoryManager.markExecutionAsOutOfMemoryOnce();
+ assertThrows(SparkOutOfMemoryError.class, map::reset);
+ assertEquals(256, map.maxNumKeysIndex());
+ } finally {
+ map.free();
+ }
+ }
+
+ @Test
+ public void reuseMapAfterFailedReset() {
+ // A reset() whose eager longArray reallocation OOMs leaves the map with
longArray == null (and,
+ // because the trailing reset() bookkeeping never runs, a stale
currentPage). A subsequent
+ // lookup then NPEs in safeLookup (this.longArray is null), and a later
append would write into
+ // a freed page. The map must instead recover to a usable empty state.
+ memoryManager.limit(5000);
+ BytesToBytesMap map =
+ new BytesToBytesMap(taskMemoryManager, blockManager, serializerManager,
256, 0.5, 4000);
+ try {
+ final long[] key = new long[]{1L};
+ final long[] value = new long[]{42L};
+ BytesToBytesMap.Location loc = map.lookup(key,
Platform.LONG_ARRAY_OFFSET, 8);
+ loc.append(key, Platform.LONG_ARRAY_OFFSET, 8, value,
Platform.LONG_ARRAY_OFFSET, 8);
Review Comment:
Addressed in 60eea7415ee. The setup append is now wrapped in assertTrue(),
preserving the stale-page regression coverage if memory-grant behavior changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
