spark-test commented on code in PR #59852:
URL: https://github.com/apache/spark/pull/59852#discussion_r4487462579
##########
python/pyspark/sql/profiler.py:
##########
@@ -119,12 +123,89 @@ def addInPlace(value1: "ProfileResultsV2", value2:
"ProfileResultsV2") -> "Profi
)
if memory is not None:
value1[key]["memory"] = memory
+ sampling = SamplingResultParam.addInPlace(
+ value1[key].get("sampling", {}), result.get("sampling", {})
+ )
+ if sampling:
+ value1[key]["sampling"] = sampling
return value1
ProfileResultsParamV2 = _ProfileResultsParamV2()
+class WorkerSamplingProfiler:
+ """
+ SamplingProfiler is a profiler for sampling profiling.
+ """
+
+ def __init__(
+ self,
+ task_context: "TaskContext",
+ sampling_interval: float = 100,
+ ) -> None:
+ self._callstack_data: dict[int, dict] = {}
+ self._task_context = task_context
+ self._sampling_interval = sampling_interval
+ self._sampling_thread: Optional[threading.Thread] = None
+ self._stop_event = threading.Event()
+
+ def __enter__(self) -> "WorkerSamplingProfiler":
+ self._start_sampling_thread()
+ return self
+
+ def __exit__(
+ self,
+ exc_type: Optional[type[BaseException]],
+ exc_val: Optional[BaseException],
+ exc_tb: Optional[TracebackType],
+ ) -> None:
+ assert self._sampling_thread is not None
+ self._stop_event.set()
+ self._sampling_thread.join()
+ self._sampling_thread = None
+
+ def save(self, accumulator: Accumulator["ProfileResultsV2"]) -> None:
+ accumulator.add({self._get_task_key(): {"sampling": {"callstack":
self._callstack_data}}})
+
+ def _start_sampling_thread(self) -> None:
+ if self._sampling_thread is not None:
+ raise PySparkRuntimeError("Sampling thread already started")
+ self._sampling_thread = threading.Thread(target=self._sample_loop)
+ self._sampling_thread.start()
+
+ def _sample_loop(self) -> None:
+ self_tid = threading.get_ident()
+ while not self._stop_event.wait(self._sampling_interval / 1000):
Review Comment:
**Non-blocking (P2):** While this `wait` sleeps, the loop variables `tid`
and `frame` from the last iteration are still bound. The sampler thread
therefore keeps a reference to the last sampled thread's innermost frame and,
through `f_back`, to its callers. Those frames' locals stay alive for up to one
interval after their function returns, for example the pandas or Arrow batch a
UDF frame holds. Their finalizers (`__del__`, weakref callbacks, generator
cleanup) then run on the sampler thread, concurrently with the task. In a
replica of this loop on CPython 3.12, a local of a sampled function was still
alive right after the function returned in 20 of 20 calls and was finalized on
the sampler thread. That raises peak memory under
`spark.executor.pyspark.memory` and perturbs the code being profiled. Clearing
the references after each pass, for example with `tid = frame = None` after the
`for` loop, frees the locals at return on their own thread.
**Verification:**
- **Regression:** While sampling is active, an object local to a sampled
function is finalized on that function's own thread when the function returns,
not later and not on the sampler thread.
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/python/ArrowPythonRunner.scala:
##########
@@ -193,6 +193,7 @@ object ArrowPythonRunner {
SQLConf.PYTHON_UDF_PANDAS_PREFER_INT_EXTENSION_DTYPE,
SQLConf.PYSPARK_BINARY_AS_BYTES,
SQLConf.PYTHON_UDF_ARROW_WORKER_OUTPUT_BATCH_MAX_BYTES,
+ SQLConf.PYTHON_UDF_PROFILER_SAMPLING_INTERVAL,
Review Comment:
**Blocking (P1):** This entry is written for every task the Arrow runners
start, not only when the sampling profiler is on. `getPythonRunnerConfMap` puts
it with the always-sent `ConfigEntry` values above `// Optional`, so the `case
c: ConfigEntry[_]` branch writes its default `"100"` whatever
`spark.sql.pyspark.udf.profiler` is. `BasePythonUDFRunner.runnerConf` in
`PythonUDFRunner.scala` also appends it unconditionally, so pickled-row UDF and
UDTF tasks carry it too.
The only reader is `RunnerConf.udf_profiler_sampling_interval`. `invoke_udf`
calls it only when the profiler is `sampling`, and it already falls back to 100
when the key is missing. So with the profiler unset (the default), `perf`, or
`memory`, every Python worker task now writes, sends, and decodes about 58
extra bytes that nothing reads. The profiler keys themselves are sent only when
they are set.
Sending the interval only when `spark.sql.pyspark.udf.profiler` is
`sampling`, in both builders, keeps sampling behavior identical and removes the
cost everywhere else.
**Verification:**
- **Regression:** With the UDF profiler unset, perf, or memory, the runner
conf built for pickled-row and Arrow Python runners contains no sampling
interval entry.
- **Behavior:** With the UDF profiler set to 'sampling', both runner paths
deliver the configured interval to the worker.
##########
python/pyspark/sql/profiler.py:
##########
@@ -119,12 +123,89 @@ def addInPlace(value1: "ProfileResultsV2", value2:
"ProfileResultsV2") -> "Profi
)
if memory is not None:
value1[key]["memory"] = memory
+ sampling = SamplingResultParam.addInPlace(
+ value1[key].get("sampling", {}), result.get("sampling", {})
+ )
+ if sampling:
+ value1[key]["sampling"] = sampling
return value1
ProfileResultsParamV2 = _ProfileResultsParamV2()
+class WorkerSamplingProfiler:
+ """
+ SamplingProfiler is a profiler for sampling profiling.
+ """
+
+ def __init__(
+ self,
+ task_context: "TaskContext",
+ sampling_interval: float = 100,
+ ) -> None:
+ self._callstack_data: dict[int, dict] = {}
+ self._task_context = task_context
+ self._sampling_interval = sampling_interval
+ self._sampling_thread: Optional[threading.Thread] = None
+ self._stop_event = threading.Event()
+
+ def __enter__(self) -> "WorkerSamplingProfiler":
+ self._start_sampling_thread()
+ return self
+
+ def __exit__(
+ self,
+ exc_type: Optional[type[BaseException]],
+ exc_val: Optional[BaseException],
+ exc_tb: Optional[TracebackType],
+ ) -> None:
+ assert self._sampling_thread is not None
+ self._stop_event.set()
+ self._sampling_thread.join()
+ self._sampling_thread = None
+
+ def save(self, accumulator: Accumulator["ProfileResultsV2"]) -> None:
+ accumulator.add({self._get_task_key(): {"sampling": {"callstack":
self._callstack_data}}})
+
+ def _start_sampling_thread(self) -> None:
+ if self._sampling_thread is not None:
+ raise PySparkRuntimeError("Sampling thread already started")
+ self._sampling_thread = threading.Thread(target=self._sample_loop)
+ self._sampling_thread.start()
+
+ def _sample_loop(self) -> None:
+ self_tid = threading.get_ident()
+ while not self._stop_event.wait(self._sampling_interval / 1000):
+ for tid, frame in sys._current_frames().items():
+ if tid != self_tid:
+ self._add_sample(tid, frame)
+
+ def _add_sample(self, tid: int, frame: Optional[FrameType]) -> None:
+ if tid not in self._callstack_data:
+ self._callstack_data[tid] = {}
+ frames: list[FrameType] = []
+ while frame is not None:
+ frames.append(frame)
+ frame = frame.f_back
+
+ d = self._callstack_data[tid]
+ for f in reversed(frames):
+ ident = self._get_ident_from_frame(f)
+ if ident not in d:
+ d[ident] = {"count": 0}
Review Comment:
**Non-blocking (P2):** Each sampled frame adds one nested dict level here,
so a thread's tree is as deep as the deepest stack ever sampled. Two recursive
consumers then hit their limits:
- `send_accumulator_updates` pickles the tree with cloudpickle after
`run_process` has finished. On CPython 3.12 this succeeds at depth 4900 and
raises `PicklingError` (wrapping `RecursionError`) at 5000, even with a raised
`sys.setrecursionlimit`. A UDF that raises its recursion limit and is sampled
that deep therefore fails its task, and its retries, although it succeeds with
the profiler off.
- Sometimes two Python workers of one task attempt report, for example an
Arrow-batched UDF and a pickled-row UDF in one stage. The second tree is then
merged into the first by `SamplingResultParam.addInPlace`, which recurses once
per level in Python. Near the default recursion limit of 1000, that raises
`RecursionError` in the driver's accumulator server (classic) or in the Spark
Connect client's handling of the query results.
Both triggers need the opt-in profiler and deep recursion, and both failures
are explicit. Still, recursive code is a common reason to reach for a profiler.
**Recommended change:** Limit how many frames one sample records. Keep a
fixed number of outermost and innermost frames and, for deeper stacks, replace
the middle with one explicit node that records how many frames were omitted, so
the stored tree depth has a small constant ceiling. Add a test that samples a
UDF that raises its recursion limit and recurses several thousand frames deep,
and checks that the task succeeds and its tree contains the omission node.
**Why this works:** Everything that recurses over the tree (the worker's
pickler, the driver-side merge, and traversals in tests or a later display) is
bounded by the stored depth. Capping depth where samples are recorded removes
the failure for all of them at once, without changing the tree for ordinary
stacks.
**Scope:** Cap the stored call-tree depth per sample, with an explicit
omitted-frames node.
**Compatibility:** Call trees for stacks within the cap keep their current
shape and counts.
**Risks:** Stacks deeper than the cap lose their middle frames; the omission
node makes that visible rather than silent.
**Constraints:** Choose the cap well below the default Python recursion
limit of 1000, for example a few hundred frames: the merge of results for the
same task recurses once per tree level in Python on the driver and in Spark
Connect clients, a stricter limit than the pickler's (about 5000 levels on
CPython 3.12). Give the omission node the same shape as other nodes, so
existing merging and traversal keep working.
**Exclusions:**
- Changing the call-tree result shape for stacks within the cap.
**Success:** A sampled UDF thousands of frames deep no longer fails its task
when the task sends its profiling results. Trees for stacks within the cap are
unchanged. Stacks deeper than the cap record an explicit omitted-frames node.
Merging two sampling results for the same task, on the driver or in a Spark
Connect client, stays within the default recursion limit for any sampled stack
depth.
**Verification:**
- **Regression:** A UDF that raises its recursion limit and recurses several
thousand frames deep while sampled completes successfully, and its sampling
result contains the omitted-frames node.
- **Behavior:** For stacks shallower than the cap, the recorded tree is
identical to the tree recorded without the cap.
- **Regression:** Two Python workers in one task, such as an Arrow-batched
UDF and a pickled-row UDF in one stage, each sampled while recursing close to
the default recursion limit, have their results merged without error in both
classic and Spark Connect sessions.
##########
python/pyspark/sql/profiler.py:
##########
@@ -119,12 +123,89 @@ def addInPlace(value1: "ProfileResultsV2", value2:
"ProfileResultsV2") -> "Profi
)
if memory is not None:
value1[key]["memory"] = memory
+ sampling = SamplingResultParam.addInPlace(
+ value1[key].get("sampling", {}), result.get("sampling", {})
+ )
+ if sampling:
+ value1[key]["sampling"] = sampling
return value1
ProfileResultsParamV2 = _ProfileResultsParamV2()
+class WorkerSamplingProfiler:
+ """
+ SamplingProfiler is a profiler for sampling profiling.
+ """
+
+ def __init__(
+ self,
+ task_context: "TaskContext",
+ sampling_interval: float = 100,
+ ) -> None:
+ self._callstack_data: dict[int, dict] = {}
+ self._task_context = task_context
+ self._sampling_interval = sampling_interval
+ self._sampling_thread: Optional[threading.Thread] = None
+ self._stop_event = threading.Event()
+
+ def __enter__(self) -> "WorkerSamplingProfiler":
+ self._start_sampling_thread()
+ return self
+
+ def __exit__(
+ self,
+ exc_type: Optional[type[BaseException]],
+ exc_val: Optional[BaseException],
+ exc_tb: Optional[TracebackType],
+ ) -> None:
+ assert self._sampling_thread is not None
+ self._stop_event.set()
+ self._sampling_thread.join()
+ self._sampling_thread = None
+
+ def save(self, accumulator: Accumulator["ProfileResultsV2"]) -> None:
+ accumulator.add({self._get_task_key(): {"sampling": {"callstack":
self._callstack_data}}})
+
+ def _start_sampling_thread(self) -> None:
+ if self._sampling_thread is not None:
+ raise PySparkRuntimeError("Sampling thread already started")
+ self._sampling_thread = threading.Thread(target=self._sample_loop)
+ self._sampling_thread.start()
+
+ def _sample_loop(self) -> None:
+ self_tid = threading.get_ident()
+ while not self._stop_event.wait(self._sampling_interval / 1000):
+ for tid, frame in sys._current_frames().items():
+ if tid != self_tid:
+ self._add_sample(tid, frame)
+
+ def _add_sample(self, tid: int, frame: Optional[FrameType]) -> None:
+ if tid not in self._callstack_data:
+ self._callstack_data[tid] = {}
+ frames: list[FrameType] = []
+ while frame is not None:
+ frames.append(frame)
+ frame = frame.f_back
+
+ d = self._callstack_data[tid]
+ for f in reversed(frames):
+ ident = self._get_ident_from_frame(f)
+ if ident not in d:
+ d[ident] = {"count": 0}
+ d[ident]["count"] += 1
+ d = d[ident]
+
+ def _get_ident_from_frame(self, frame: FrameType) -> str:
+ code = frame.f_code
+ return f"{code.co_qualname} ({code.co_filename}:{code.co_firstlineno})"
Review Comment:
**Nit (P3):** This builds a new string for every frame on every tick. A tree
node keeps the string that first created it, so a function that appears at
several nodes (through shared helpers or recursion) is stored as several equal
but distinct strings. Pickle memoizes by object identity, so every copy is
serialized in full and sent with the task's accumulator update. The driver or
Connect client then keeps each copy as a separate object, once per retained
task entry. In a replica of this sampler, caching one string per code object
shrank the pickled tree 1.6x for a shallow stack and 5.1x for a 40-deep
recursive one, with identical content. It also saves formatting work on every
sample. A small cache keyed by `id(code)` that also holds the code object, so
the id cannot be reused, gives the same keys.
**Verification:**
- **Behavior:** For a sampled workload in which one function appears at
several tree nodes, the pickled sampling payload stores that function's ident
once and is smaller than with per-node strings, while the tree content stays
identical.
##########
python/pyspark/sql/tests/test_udf_profiler.py:
##########
@@ -762,6 +762,67 @@ def test_perf_profiler_clear(self):
self.spark.profile.clear()
self.assertEqual(0, len(self.profile_results),
str(list(self.profile_results)))
+ @property
+ def sampling_profile_results(self):
+ return self.spark._profiler_collector._sampling_profile_results
+
+ def test_sampling_profiler_udf(self):
+ import time
+
+ @udf("long")
+ def busy_add1(x):
+ # Sleep so the sampling thread reliably catches this frame on the
stack.
+ time.sleep(0.05)
+ return x + 1
+
+ # Without the conf enabled, no sampling results are collected.
+ self.spark.range(5, numPartitions=1).select(busy_add1("id")).collect()
+ self.assertEqual(0, len(self.sampling_profile_results))
+
+ with self.sql_conf(
+ {
+ "spark.sql.pyspark.udf.profiler": "sampling",
+ "spark.sql.pyspark.udf.profiler.samplingInterval": "10",
Review Comment:
**Nit (P3):** Nothing in the test depends on this 10 ms value. Each task
runs five 50 ms sleeps, so the 100 ms fallback in
`RunnerConf.udf_profiler_sampling_interval` still catches `busy_add1`. Dropping
the interval from either JVM builder would therefore keep the test green.
`@udf("long")` runs Arrow-batched because
`spark.sql.execution.pythonUDF.arrow.enabled` defaults to true, so the
`BasePythonUDFRunner.runnerConf` change never runs with sampling on. Each task
key also receives a single update, so the summing branches of
`SamplingResultParam.addInPlace` never execute. Three additions would cover
this: a `useArrow=False` variant, an interval large enough that a short task
records no samples, and two Python UDFs of different eval types in one stage,
so that two workers report under one task key.
See **Shared repair plan 1** in the review body.
##########
python/pyspark/sql/tests/test_udf_profiler.py:
##########
@@ -762,6 +762,67 @@ def test_perf_profiler_clear(self):
self.spark.profile.clear()
self.assertEqual(0, len(self.profile_results),
str(list(self.profile_results)))
+ @property
+ def sampling_profile_results(self):
+ return self.spark._profiler_collector._sampling_profile_results
+
+ def test_sampling_profiler_udf(self):
+ import time
+
+ @udf("long")
+ def busy_add1(x):
+ # Sleep so the sampling thread reliably catches this frame on the
stack.
+ time.sleep(0.05)
+ return x + 1
+
+ # Without the conf enabled, no sampling results are collected.
+ self.spark.range(5, numPartitions=1).select(busy_add1("id")).collect()
+ self.assertEqual(0, len(self.sampling_profile_results))
+
+ with self.sql_conf(
+ {
+ "spark.sql.pyspark.udf.profiler": "sampling",
+ "spark.sql.pyspark.udf.profiler.samplingInterval": "10",
+ }
+ ):
+ self.spark.range(10,
numPartitions=2).select(busy_add1("id")).collect()
+
+ results = self.sampling_profile_results
+ # Two partitions -> two tasks -> two call-trees keyed by task.
+ self.assertEqual(2, len(results), str(list(results)))
+
+ # Flatten the per-thread call trees and assert the UDF frame was
sampled.
+ def collect_idents(node, idents):
+ for key, child in node.items():
+ if key == "count":
+ continue
+ idents.append(key)
+ collect_idents(child, idents)
+
+ for result in results.values():
+ idents = []
+ for thread_tree in result["callstack"].values():
+ collect_idents(thread_tree, idents)
+ self.assertTrue(
+ any("busy_add1" in ident for ident in idents),
Review Comment:
**Nit (P3):** This presence check is the test's only assertion on the call
tree. Dropping `reversed()` in `_add_sample`, leaving every `count` at 0, or
sampling the sampler's own thread would all still pass. The per-id clear below
goes through the private `spark._profiler_collector.clear_sampling_profiles`.
Meanwhile, the public `spark.profile.clear(id)` path this PR changed is never
called with an id, so if it stopped forwarding the id and wiped every sampling
result, nothing would notice. No case sets `samplingInterval` to 0 or a
negative value either. Without the `checkValue`, such a value would make
`Event.wait` return at once and spin the sampler. Five assertions would close
these gaps: root-to-leaf order, positive counts, no sampler-thread entry,
`spark.profile.clear(<task key>)` removing only that entry, and an error for a
non-positive interval.
See **Shared repair plan 1** in the review body.
##########
python/pyspark/worker.py:
##########
@@ -293,9 +298,24 @@ def _queued_iter():
serializer._flush_per_batch = True
run_process = pipelined_process if is_pipelined else process
+ sampling_profiler_enabled = runner_conf.udf_profiler == "sampling"
Review Comment:
**Non-blocking (P2):** This check turns on the task-wide sampler for every
Python worker whose runner forwards `spark.sql.pyspark.udf.profiler`, not just
for Python UDFs. `UserDefinedPythonDataSource.createMapInBatchEvaluatorFactory`
forwards it for Python data source batch reads, streaming reads, and writes.
`PythonUDTFRunner` and `ArrowEvalPythonUDTFExec` forward it for UDTFs. So with
`sampling` set, those tasks are sampled as well. Their task-keyed trees land in
the same results as UDF tasks, with nothing to tell them apart. When a stage
reads a Python data source and applies a UDF, both workers report under one
task key and their trees merge.
`perf` and `memory` behave differently. `EvalTypeHandler._with_profiler`
sends `pyspark.sql.worker.*` functions to
`spark.sql.pyspark.dataSource.profiler` and never wraps UDTFs.
`test_data_source_read_with_udf_perf_profiler` asserts that the UDF profiler
setting does not enable data source profiling.
`spark.sql.pyspark.dataSource.profiler` can neither select nor exclude
`sampling`, and the comment earlier in `invoke_udf` saying that UDTF runners
are not profiled no longer holds. Query results are unaffected. The effect is
extra overhead and mixed profiling output in this opt-in mode.
**Recommended change:** Start the task-level sampling profiler only for
tasks whose functions the UDF profiler conf governs: Python UDF tasks whose
functions are not Python data source functions, and not UDTF tasks. This
matches the routing that perf and memory already follow. Update the worker
comment about which runners are profiled and the spark.sql.pyspark.udf.profiler
documentation to say that 'sampling' samples whole Python UDF tasks. Add
PySpark tests showing that a Python data source read and a Python UDTF add no
sampling results under the UDF sampling setting, while a Python UDF still does.
**Why this works:** invoke_udf already builds the eval-type handler, which
knows whether it serves a UDTF and which functions it wraps. A handler-level
check (no UDTF and no function from pyspark.sql.worker.) therefore replaces the
bare comparison of the UDF profiler conf. Data source and UDTF tasks then never
start the sampler or add task entries, and UDF tasks keep today's behavior.
**Scope:** Gate the worker's sampling activation on the existing
UDF-versus-data-source profiler routing and exclude UDTF tasks, with matching
documentation and tests.
**Compatibility:** perf and memory keep profiling UDF functions under the
UDF setting and data source functions under the data source setting, and Python
UDF tasks stay sampled under 'sampling'.
**Risks:** UDTF tasks stop being sampled under 'sampling'. If UDTF sampling
is wanted, the check must admit UDTF handlers explicitly, and the worker
comment and conf documentation must say so.
**Constraints:** Keep perf and memory profiling unchanged, including their
per-UDF wrapping and their routing of data source functions to the data source
profiler setting.
**Exclusions:**
- Accepting 'sampling' in spark.sql.pyspark.dataSource.profiler for Python
data source tasks.
**Success:** With spark.sql.pyspark.udf.profiler set to 'sampling', Python
data source read and write tasks and Python UDTF tasks add no sampling results
to the session profile results. Python UDF tasks are still sampled and saved
per task. The worker comment and the spark.sql.pyspark.udf.profiler
documentation describe which tasks 'sampling' profiles.
**Verification:**
- **Regression:** With the UDF profiler set to 'sampling', reading from a
registered Python data source adds no sampling results, matching the existing
guarantee that the UDF profiler setting does not enable data source profiling.
- **Regression:** With the UDF profiler set to 'sampling', running a Python
UDTF adds no sampling results.
- **Behavior:** With the UDF profiler set to 'sampling', a Python UDF query
still produces one sampling result per task that contains the UDF frame, in
both classic and Spark Connect sessions.
- **Compatibility:** perf and memory profiling results for UDFs and for
Python data sources are unchanged.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala:
##########
@@ -5452,14 +5452,25 @@ object SQLConf {
val PYTHON_UDF_PROFILER =
buildConf("spark.sql.pyspark.udf.profiler")
.doc("Configure the Python/Pandas UDF profiler by enabling or disabling
it " +
- "with the option to choose between \"perf\" and \"memory\" types, " +
+ "with the option to choose between \"perf\", \"memory\" and
\"sampling\" types, " +
"or unsetting the config disables the profiler. This is disabled by
default.")
.version("4.0.0")
.stringConf
.transform(_.toLowerCase(Locale.ROOT))
- .checkValues(Set("perf", "memory"))
+ .checkValues(Set("perf", "memory", "sampling"))
.createOptional
+ val PYTHON_UDF_PROFILER_SAMPLING_INTERVAL =
+ buildConf("spark.sql.pyspark.udf.profiler.samplingInterval")
Review Comment:
**Nit (P3):** Item 3 of the naming guideline at the top of
`ConfigEntry.scala` asks a time-duration config to carry its unit in the name
as well as use `timeConf`. The nearest peers do both:
`spark.python.worker.idleTimeoutSeconds` and
`spark.python.worker.tracebackDumpIntervalSeconds`, whose
`spark.sql.execution.pyspark.udf.*` fallbacks sit a few lines below. Here a
bare number is read as milliseconds, which users learn only from the doc. Some
older public time keys, such as `spark.sql.broadcastTimeout`, omit the unit,
but the Python worker configs this one sits beside include it. Once the key
ships, renaming it needs an alternative key and a deprecation. It is easier to
settle now, for example as `spark.sql.pyspark.udf.profiler.samplingIntervalMs`,
together with the worker lookup in `worker_util.py` and the test.
**Verification:**
- **Inspection:** The sampling interval conf key ends with a millisecond
unit suffix, and no reference to the old key remains.
- **Behavior:** Setting the renamed key changes the sampling period the
worker uses.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
