Re: Spark (1.2) yarn allocator does not remove container request for allocated container, resulting in a bloated ask[] of containers and inefficient resource utilization of cluster resources.

[prakhar jauhari]

hey all,

Thanks in advance.
I am facing this issue in production, where due to increased container
request RM is reserving memory and hampering cluster utilization. Thus the
fix needs to be patched on spark 1.2. 

Has any one looked in the removeContainerRequest part for allocated
containers in spark 1.2?

Regards,
Prakhar. 





--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-2-yarn-allocator-does-not-remove-container-request-for-allocated-container-resulting-in-a-bl-tp13508p13518.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: unit test failure for hive query

[Michael Armbrust]

I'd suggest using org.apache.spark.sql.hive.test.TestHive as the context in
unit tests.  It takes care of creating separate directories for each
invocation automatically.

On Wed, Jul 29, 2015 at 7:02 PM, JaeSung Jun  wrote:

> Hi,
> I'm working on custom sql processing on top of Spark-SQL, and i'm
> upgrading it along with spark 1.4.1.
> I've got an error regarding multiple test suites access hive meta store at
> the same time like :
>
>  Cause: org.apache.derby.impl.jdbc.EmbedSQLException: Another instance of
> Derby may have already booted the database /Users/~~~/metastore_db.
>
>   at
> org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
> Source)
>
>   at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
> Source)
>
>   at
> org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
> Source)
>
>   at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)
>
>   at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown
> Source)
>
>   at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)
>
>   at org.apache.derby.impl.jdbc.EmbedConnection40.(Unknown Source)
>
>   at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)
>
>   at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)
>
>   at org.apache.derby.jdbc.Driver20.connect(Unknown Source)
>
>
> It was okay with spark 1.3.0.
>
> Any idea to fist this?
>
>
> thanks in advance.
>
> Jason
>

unit test failure for hive query

[JaeSung Jun]

Hi,
I'm working on custom sql processing on top of Spark-SQL, and i'm upgrading
it along with spark 1.4.1.
I've got an error regarding multiple test suites access hive meta store at
the same time like :

 Cause: org.apache.derby.impl.jdbc.EmbedSQLException: Another instance of
Derby may have already booted the database /Users/~~~/metastore_db.

  at org.apache.derby.impl.jdbc.SQLExceptionFactory.getSQLException(Unknown
Source)

  at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.wrapArgsForTransportAcrossDRDA(Unknown
Source)

  at
org.apache.derby.impl.jdbc.SQLExceptionFactory40.getSQLException(Unknown
Source)

  at org.apache.derby.impl.jdbc.Util.generateCsSQLException(Unknown Source)

  at org.apache.derby.impl.jdbc.EmbedConnection.bootDatabase(Unknown Source)

  at org.apache.derby.impl.jdbc.EmbedConnection.(Unknown Source)

  at org.apache.derby.impl.jdbc.EmbedConnection40.(Unknown Source)

  at org.apache.derby.jdbc.Driver40.getNewEmbedConnection(Unknown Source)

  at org.apache.derby.jdbc.InternalDriver.connect(Unknown Source)

  at org.apache.derby.jdbc.Driver20.connect(Unknown Source)


It was okay with spark 1.3.0.

Any idea to fist this?


thanks in advance.

Jason

Re: update on git timeouts for jenkins builds

[shane knapp]

newp.  still happening, and i'm still looking in to it:

https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/38880/console

On Wed, Jul 29, 2015 at 12:20 PM, shane knapp  wrote:
> ok, i think i found the problem and solution to the git timeouts:
>
> https://stackoverflow.com/questions/12236415/git-clone-return-result-18-code-200-on-a-specific-repository
>
> so, on each worker i've run "git config --global http.postBuffer
> 524288000" as the jenkins user and we'll see if this makes a
> difference.
>
> On Tue, Jul 28, 2015 at 11:51 AM, shane knapp  wrote:
>> hey all, i'm just back in from my wedding weekend (woot!) and am
>> working on figuring out what's happening w/the git timeouts for pull
>> request builds.
>>
>> TL;DR:  if your build fails due to a timeout, please retrigger your
>> builds.  i know this isn't the BEST solution, but until we get some
>> stuff implemented (traffic shaping, git cache for the workers) it's
>> the only thing i can recommend.
>>
>> here's a snapshot of the state of the union:
>> $ get_timeouts.sh 5
>> timeouts by date:
>> 2015-07-23 -- 3
>> 2015-07-24 -- 1
>> 2015-07-26 -- 7
>> 2015-07-27 -- 18
>> 2015-07-28 -- 9
>>
>> timeouts by project:
>>  35 SparkPullRequestBuilder
>>   3 Tachyon-Pull-Request-Builder
>> total builds (excepting aborted by a user):
>> 1908
>>
>> total percentage of builds timing out:
>> 01%
>>
>> nothing has changed on our end AFAIK, our traffic graphs look totally
>> fine, but starting sunday, we started seeing a spike in timeouts, with
>> yesterday being the worst.  today is also not looking good either.
>>
>> github is looking OK, but not "great":
>> https://status.github.com/
>>
>> as a solution, we'll be setting up some traffic shaping on our end, as
>> well as implementing a git cache on the workers so that we'll
>> (hopefully) minimize how many hits we make against github.  i was
>> planning on doing the git cache months ago, but the timeout issue
>> pretty much went away and i back-burnered that idea until today.
>>
>> other than that, i'll be posting updates as we get them.
>>
>> shane

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Broadcast variable of size 1 GB fails with negative memory exception

[Mike Hynes]

Hi Imran,
Thanks to you and Shivaram for looking into this, and opening the
JIRA/PR. I will update you once the PR is merged if there are any
other problems that arise from the broadcast.
Mike

On 7/29/15, Imran Rashid  wrote:
> Hi Mike,
>
> I dug into this a little more, and it turns out in this case there is a
> pretty trivial fix -- the problem you are seeing is just from integer
> overflow before casting to a long in SizeEstimator.  I've opened
> https://issues.apache.org/jira/browse/SPARK-9437 for this.
>
> For now, I think your workaround is to break into multiple arrays, and
> broadcast them each separately.  If its any consolation, you would have to
> do this anyway if you tried to broadcast more than Int.MAX_VALUE doubles in
> any case.  The JVM only allows arrays up to that length.  So even if spark
> didn't have this limitation, you could go up to an array of 2^31 doubles,
> which would be 16GB, before having to break into multiple arrays.
>
> There *are* a number of places in spark where things are limited to 2GB,
> and there are a handful of open issues to deal with it.  However I think
> this one is pretty easy to fix (I must have looked at this a half-dozen
> times before and never realized the fix was so simple before ...).  However
> it could be we'll run into something else after this particular issue w/
> SizeEstimator is fixed.  This certainly won't work with HttpBroadcast, but
> I think it might just work as long as you stick with TorrentBroadcast.
>
> imran
>
> On Tue, Jul 28, 2015 at 10:56 PM, Mike Hynes <91m...@gmail.com> wrote:
>
>> Hi Imran,
>>
>> Thanks for your reply. I have double-checked the code I ran to
>> generate an nxn matrix and nx1 vector for n = 2^27. There was
>> unfortunately a bug in it, where instead of having typed 134,217,728
>> for n = 2^27, I included a third '7' by mistake, making the size 10x
>> larger.
>>
>> However, even after having corrected this, my question about
>> broadcasting is still whether or not a variable >= 2G in size may be
>> transferred? In this case, for n >= 2^28, the broadcast variable
>> crashes, and an array of size MAX_INT cannot be broadcast.
>>
>> Looking at Chowdhury's "Performance and Scalability of Broadcast in
>> Spark" technical report, I realize that the results are reported only
>> for broadcast variables up to 1 GB in physical size. I was hoping,
>> however, that an Array of size MAX_INT would be transferrable via a
>> broadcast (since the previous PR I mentioned seems to have added
>> support for > 2GB variables) such that the matrix-vector
>> multiplication would scale to MAX_INT x MAX_INT matrices with a
>> broadcast variable.
>>
>> Would you or anyone on the dev list be able to comment on whether this
>> is possible? Since the (corrected) overflow I'm seeing is for > 2^31
>> physical bytes being transferred, I am guessing that there is still a
>> physical limitation on how many bytes may be sent via broadcasting, at
>> least for a primitive Array[Double]?
>>
>> Thanks,
>> Mike
>>
>> 19176&INFO&IndexedRowMatrix&Broadcasting vecArray with size 268435456&
>> 19177&INFO&MemoryStore&ensureFreeSpace(-2147483592) called with
>> curMem=6888, maxMem=92610625536&
>> 19177&INFO&MemoryStore&Block broadcast_2 stored as values in memory
>> (estimated size -2147483592.0 B, free 88.3 GB)&
>> Exception in thread "main" java.lang.IllegalArgumentException:
>> requirement failed: sizeInBytes was negative: -2147483592
>>
>> On 7/28/15, Imran Rashid  wrote:
>> > Hi Mike,
>> >
>> > are you sure there the size isn't off 2x somehow?  I just tried to
>> > reproduce with a simple test in BlockManagerSuite:
>> >
>> > test("large block") {
>> >   store = makeBlockManager(4e9.toLong)
>> >   val arr = new Array[Double](1 << 28)
>> >   println(arr.size)
>> >   val blockId = BlockId("rdd_3_10")
>> >   val result = store.putIterator(blockId, Iterator(arr),
>> > StorageLevel.MEMORY_AND_DISK)
>> >   result.foreach{println}
>> > }
>> >
>> > it fails at 1 << 28 with nearly the same message, but its fine for (1
>> > <<
>> > 28) - 1 with a reported block size of 2147483680.  Not exactly the same
>> as
>> > what you did, but I expect it to be close enough to exhibit the same
>> error.
>> >
>> >
>> > On Tue, Jul 28, 2015 at 12:37 PM, Mike Hynes <91m...@gmail.com> wrote:
>> >>
>> >> Hello Devs,
>> >>
>> >> I am investigating how matrix vector multiplication can scale for an
>> >> IndexedRowMatrix in mllib.linalg.distributed.
>> >>
>> >> Currently, I am broadcasting the vector to be multiplied on the right.
>> >> The IndexedRowMatrix is stored across a cluster with up to 16 nodes,
>> >> each with >200 GB of memory. The spark driver is on an identical node,
>> >> having more than 200 Gb of memory.
>> >>
>> >> In scaling n, the size of the vector to be broadcast, I find that the
>> >> maximum size of n that I can use is 2^26. For 2^27, the broadcast will
>> >> fail. The array being broadcast is of type Array[Double], so the
>> >> contents have size 2^30 bytes, which 

Re: update on git timeouts for jenkins builds

[shane knapp]

ok, i think i found the problem and solution to the git timeouts:

https://stackoverflow.com/questions/12236415/git-clone-return-result-18-code-200-on-a-specific-repository

so, on each worker i've run "git config --global http.postBuffer
524288000" as the jenkins user and we'll see if this makes a
difference.

On Tue, Jul 28, 2015 at 11:51 AM, shane knapp  wrote:
> hey all, i'm just back in from my wedding weekend (woot!) and am
> working on figuring out what's happening w/the git timeouts for pull
> request builds.
>
> TL;DR:  if your build fails due to a timeout, please retrigger your
> builds.  i know this isn't the BEST solution, but until we get some
> stuff implemented (traffic shaping, git cache for the workers) it's
> the only thing i can recommend.
>
> here's a snapshot of the state of the union:
> $ get_timeouts.sh 5
> timeouts by date:
> 2015-07-23 -- 3
> 2015-07-24 -- 1
> 2015-07-26 -- 7
> 2015-07-27 -- 18
> 2015-07-28 -- 9
>
> timeouts by project:
>  35 SparkPullRequestBuilder
>   3 Tachyon-Pull-Request-Builder
> total builds (excepting aborted by a user):
> 1908
>
> total percentage of builds timing out:
> 01%
>
> nothing has changed on our end AFAIK, our traffic graphs look totally
> fine, but starting sunday, we started seeing a spike in timeouts, with
> yesterday being the worst.  today is also not looking good either.
>
> github is looking OK, but not "great":
> https://status.github.com/
>
> as a solution, we'll be setting up some traffic shaping on our end, as
> well as implementing a git cache on the workers so that we'll
> (hopefully) minimize how many hits we make against github.  i was
> planning on doing the git cache months ago, but the timeout issue
> pretty much went away and i back-burnered that idea until today.
>
> other than that, i'll be posting updates as we get them.
>
> shane

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

Re: Broadcast variable of size 1 GB fails with negative memory exception

[Imran Rashid]

Hi Mike,

I dug into this a little more, and it turns out in this case there is a
pretty trivial fix -- the problem you are seeing is just from integer
overflow before casting to a long in SizeEstimator.  I've opened
https://issues.apache.org/jira/browse/SPARK-9437 for this.

For now, I think your workaround is to break into multiple arrays, and
broadcast them each separately.  If its any consolation, you would have to
do this anyway if you tried to broadcast more than Int.MAX_VALUE doubles in
any case.  The JVM only allows arrays up to that length.  So even if spark
didn't have this limitation, you could go up to an array of 2^31 doubles,
which would be 16GB, before having to break into multiple arrays.

There *are* a number of places in spark where things are limited to 2GB,
and there are a handful of open issues to deal with it.  However I think
this one is pretty easy to fix (I must have looked at this a half-dozen
times before and never realized the fix was so simple before ...).  However
it could be we'll run into something else after this particular issue w/
SizeEstimator is fixed.  This certainly won't work with HttpBroadcast, but
I think it might just work as long as you stick with TorrentBroadcast.

imran

On Tue, Jul 28, 2015 at 10:56 PM, Mike Hynes <91m...@gmail.com> wrote:

> Hi Imran,
>
> Thanks for your reply. I have double-checked the code I ran to
> generate an nxn matrix and nx1 vector for n = 2^27. There was
> unfortunately a bug in it, where instead of having typed 134,217,728
> for n = 2^27, I included a third '7' by mistake, making the size 10x
> larger.
>
> However, even after having corrected this, my question about
> broadcasting is still whether or not a variable >= 2G in size may be
> transferred? In this case, for n >= 2^28, the broadcast variable
> crashes, and an array of size MAX_INT cannot be broadcast.
>
> Looking at Chowdhury's "Performance and Scalability of Broadcast in
> Spark" technical report, I realize that the results are reported only
> for broadcast variables up to 1 GB in physical size. I was hoping,
> however, that an Array of size MAX_INT would be transferrable via a
> broadcast (since the previous PR I mentioned seems to have added
> support for > 2GB variables) such that the matrix-vector
> multiplication would scale to MAX_INT x MAX_INT matrices with a
> broadcast variable.
>
> Would you or anyone on the dev list be able to comment on whether this
> is possible? Since the (corrected) overflow I'm seeing is for > 2^31
> physical bytes being transferred, I am guessing that there is still a
> physical limitation on how many bytes may be sent via broadcasting, at
> least for a primitive Array[Double]?
>
> Thanks,
> Mike
>
> 19176&INFO&IndexedRowMatrix&Broadcasting vecArray with size 268435456&
> 19177&INFO&MemoryStore&ensureFreeSpace(-2147483592) called with
> curMem=6888, maxMem=92610625536&
> 19177&INFO&MemoryStore&Block broadcast_2 stored as values in memory
> (estimated size -2147483592.0 B, free 88.3 GB)&
> Exception in thread "main" java.lang.IllegalArgumentException:
> requirement failed: sizeInBytes was negative: -2147483592
>
> On 7/28/15, Imran Rashid  wrote:
> > Hi Mike,
> >
> > are you sure there the size isn't off 2x somehow?  I just tried to
> > reproduce with a simple test in BlockManagerSuite:
> >
> > test("large block") {
> >   store = makeBlockManager(4e9.toLong)
> >   val arr = new Array[Double](1 << 28)
> >   println(arr.size)
> >   val blockId = BlockId("rdd_3_10")
> >   val result = store.putIterator(blockId, Iterator(arr),
> > StorageLevel.MEMORY_AND_DISK)
> >   result.foreach{println}
> > }
> >
> > it fails at 1 << 28 with nearly the same message, but its fine for (1 <<
> > 28) - 1 with a reported block size of 2147483680.  Not exactly the same
> as
> > what you did, but I expect it to be close enough to exhibit the same
> error.
> >
> >
> > On Tue, Jul 28, 2015 at 12:37 PM, Mike Hynes <91m...@gmail.com> wrote:
> >>
> >> Hello Devs,
> >>
> >> I am investigating how matrix vector multiplication can scale for an
> >> IndexedRowMatrix in mllib.linalg.distributed.
> >>
> >> Currently, I am broadcasting the vector to be multiplied on the right.
> >> The IndexedRowMatrix is stored across a cluster with up to 16 nodes,
> >> each with >200 GB of memory. The spark driver is on an identical node,
> >> having more than 200 Gb of memory.
> >>
> >> In scaling n, the size of the vector to be broadcast, I find that the
> >> maximum size of n that I can use is 2^26. For 2^27, the broadcast will
> >> fail. The array being broadcast is of type Array[Double], so the
> >> contents have size 2^30 bytes, which is approximately 1 (metric) GB.
> >>
> >> I have read in PR  [SPARK-3721] [PySpark] "broadcast objects larger
> >> than 2G" that this should be supported (I assume this works for scala,
> >> as well?). However, when I increase n to 2^27 or above, the program
> >> invariably crashes at the broadcast.
> >>
> >> The problem stems from the size of the result

Re: "Spree": Live-updating web UI for Spark

[mkhaitman]

We tested this out on our dev cluster (Hadoop 2.7.1 + Spark 1.4.0), and it
looks great! I might also be interested in contributing to it when I get a
chance! Keep up the awesome work! :)

Mark.



--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Spree-Live-updating-web-UI-for-Spark-tp13456p13511.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org

RE: Two joins in GraphX Pregel implementation

[Ulanov, Alexander]

Hi Ankur,

Thank you! This looks like a nice simplification. There should be some 
performance improvement since newVerts are not chached now.
I’ve added your patch:
https://issues.apache.org/jira/browse/SPARK-9436

Best regards, Alexander

From: Ankur Dave [mailto:ankurd...@gmail.com]
Sent: Tuesday, July 28, 2015 12:05 PM
To: Ulanov, Alexander
Cc: Robin East; dev@spark.apache.org
Subject: Re: Two joins in GraphX Pregel implementation

On 27 Jul 2015, at 16:42, Ulanov, Alexander 
mailto:alexander.ula...@hp.com>> wrote:
It seems that the mentioned two joins can be rewritten as one outer join

You're right. In fact, the outer join can be streamlined further using a method 
from GraphOps:

g = g.joinVertices(messages)(vprog).cache()

Then, instead of passing newVerts as the active set for mapReduceTriplets, we 
could pass `messages`.

If you're interested in proposing a PR for this, I've attached a patch with 
these changes and updates to the comments.

On Tue, Jul 28, 2015 at 1:15 AM, Ulanov, Alexander 
mailto:alexander.ula...@hp.com>> wrote:
I’ve found two PRs (almost identical) for replacing mapReduceTriplets with 
aggregateMessages
[...]
Do you know the reason why this improvement is not pushed?

There isn't any performance benefit to switching Pregel to use 
aggregateMessages while preserving its current interface, because the interface 
uses Iterators and would require us to wrap and unwrap them anyway. The 
semantics of aggregateMessagesWithActiveSet are otherwise the same as 
mapReduceTriplets, so there isn't any functionality we are missing out on. And 
this change seems too small to justify introducing a new version of Pregel, 
though it would be worthwhile when combined with other 
improvements.

Ankur

Re: [ANNOUNCE] Nightly maven and package builds for Spark

[Bharath Ravi Kumar]

Hey Patrick,

Any update on this front please?

Thanks,
Bharath

On Fri, Jul 24, 2015 at 8:38 PM, Patrick Wendell  wrote:

> Hey Bharath,
>
> There was actually an incompatible change to the build process that
> broke several of the Jenkins builds. This should be patched up in the
> next day or two and nightly builds will resume.
>
> - Patrick
>
> On Fri, Jul 24, 2015 at 12:51 AM, Bharath Ravi Kumar
>  wrote:
> > I noticed the last (1.5) build has a timestamp of 16th July. Have nightly
> > builds been discontinued since then?
> >
> > Thanks,
> > Bharath
> >
> > On Sun, May 24, 2015 at 1:11 PM, Patrick Wendell 
> wrote:
> >>
> >> Hi All,
> >>
> >> This week I got around to setting up nightly builds for Spark on
> >> Jenkins. I'd like feedback on these and if it's going well I can merge
> >> the relevant automation scripts into Spark mainline and document it on
> >> the website. Right now I'm doing:
> >>
> >> 1. SNAPSHOT's of Spark master and release branches published to ASF
> >> Maven snapshot repo:
> >>
> >>
> >>
> https://repository.apache.org/content/repositories/snapshots/org/apache/spark/
> >>
> >> These are usable by adding this repository in your build and using a
> >> snapshot version (e.g. 1.3.2-SNAPSHOT).
> >>
> >> 2. Nightly binary package builds and doc builds of master and release
> >> versions.
> >>
> >> http://people.apache.org/~pwendell/spark-nightly/
> >>
> >> These build 4 times per day and are tagged based on commits.
> >>
> >> If anyone has feedback on these please let me know.
> >>
> >> Thanks!
> >> - Patrick
> >>
> >> -
> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
> >> For additional commands, e-mail: dev-h...@spark.apache.org
> >>
> >
>

Spark (1.2) yarn allocator does not remove container request for allocated container, resulting in a bloated ask[] of containers and inefficient resource utilization of cluster resources.

[prakhar jauhari]

This is because Yarn's AM client does not remove fulfilled container request
from its MAP until the application's AM specifically calls
removeContainerRequest for fulfilled container requests.

Spark-1.2 : Spark's yarn AM does not call removeContainerRequest for
fulfilled container request.

Spark-1.3 : yarn AM calls removeContainerRequest for the container requests
it can map to be fulfilled. Tried the same test case of killing one executor
with spark-1.3 and the ask[] in this case was for 1 container.

As long as the cluster size is large enough to allocate the bloated
container requests, containers are sent to spark yarn allocator in allocate
response, spark yarn allocator uses missing number of container to launch
new executors and release the extra allocated containers.

The problem magnifies in case of long running jobs with large executor
memory requirements. In this case when ever a executor gets killed, the next
ask to yarn Resource manager (RM) is of n+1 containers (n being count of
already requested containers), which might be served by the RM if it still
has enough resources, else RM starts reserving cluster resources for a
containers which are not even required by spark in the first place. 

This causes resource crunch for other applications, and inefficient resource
utilization of cluster resources.

I was not able to find a thread talking specifically about this. Is there
any known use case due to which removeContainerRequest is not done in spark
1.2?  





--
View this message in context: 
http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-1-2-yarn-allocator-does-not-remove-container-request-for-allocated-container-resulting-in-a-bl-tp13508.html
Sent from the Apache Spark Developers List mailing list archive at Nabble.com.

-
To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org
For additional commands, e-mail: dev-h...@spark.apache.org