It should be in the first email in this chain.
On Tue, Dec 12, 2017, 7:10 PM Ryan Blue wrote:
> Great. What's the JIRA issue?
>
> On Mon, Dec 11, 2017 at 8:12 PM, Jason White
> wrote:
>
>> Yes, the fix has been merged at should make it into the 2.3 release.
>>
>> On Mon, Dec 11, 2017, 5:50 PM R
Great. What's the JIRA issue?
On Mon, Dec 11, 2017 at 8:12 PM, Jason White
wrote:
> Yes, the fix has been merged at should make it into the 2.3 release.
>
> On Mon, Dec 11, 2017, 5:50 PM Ryan Blue wrote:
>
>> Is anyone currently working on this? I just fixed it in our Spark build
>> and can con
Yes, the fix has been merged at should make it into the 2.3 release.
On Mon, Dec 11, 2017, 5:50 PM Ryan Blue wrote:
> Is anyone currently working on this? I just fixed it in our Spark build
> and can contribute the fix if there isn't already a PR for it.
>
> On Mon, Nov 27, 2017 at 12:59 PM, Jas
Is anyone currently working on this? I just fixed it in our Spark build and
can contribute the fix if there isn't already a PR for it.
On Mon, Nov 27, 2017 at 12:59 PM, Jason White
wrote:
> It doesn't look like the insert command has any metrics in it. I don't see
> any commands with metrics, bu
It doesn't look like the insert command has any metrics in it. I don't see
any commands with metrics, but I could be missing something.
--
Sent from: http://apache-spark-developers-list.1001551.n3.nabble.com/
-
To unsubscribe e
I think the difference lies somewhere in here:
- RDD writes are done with SparkHadoopMapReduceWriter.executeTask, which
calls outputMetrics.setRecordsWritten
- DF writes are done with InsertIntoHadoopFsRelationCommand.run ? Which I'm
not entirely sure how it works.
executeTask appears to be run on
Is this due to the insert command not having metrics? It's a problem we
should fix.
On Mon, Nov 27, 2017 at 10:45 AM, Jason White
wrote:
> I'd like to use the SparkListenerInterface to listen for some metrics for
> monitoring/logging/metadata purposes. The first ones I'm interested in
> hooking
I'd like to use the SparkListenerInterface to listen for some metrics for
monitoring/logging/metadata purposes. The first ones I'm interested in
hooking into are recordsWritten and bytesWritten as a measure of throughput.
I'm using PySpark to write Parquet files from DataFrames.
I'm able to extrac
