I get a few warnings like this in Spark 2.0.1 when using org
.apache.spark.mllib.recommendation.ALS:
WARN org.apache.spark.executor.Executor - 1 block locks were not released
by TID = 1448:
[rdd_239_0]
What can be the reason for that?
--
[image: MagineTV]
*Mikael Ståldal*
Senior software
/O) and then be able to do recommendProducts()
on it multiple times, quickly without I/O?
--
[image: MagineTV]
*Mikael Ståldal*
Senior software developer
*Magine TV*
mikael.stal...@magine.com
Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com
Privileged and/or Confidential
anything useful because
> you're executing asynchronously but then just blocking waiting for
> completion. It seems the same as just doing all the work in processItems()
> directly.
>
> On Fri, Jul 8, 2016 at 1:56 PM, Mikael Ståldal <mikael.stal...@magine.com>
> wrote:
>
>
on on an RDD. A transformation
> doesn't actually do anything by itself, so does not block. An action
> triggers computation and blocks until the action completes. You can wait
> for it with a Future, sure.
>
> On Fri, Jul 8, 2016 at 10:43 AM, Mikael Ståldal <mikael.stal...@magine.
In a Spark Streaming job, is the operation inside foreachRDD supposed to
synchronous / blocking?
What if you do some asynchronous operation which returns a Future? Are you
then supposed to do Await on that Future?
--
[image: MagineTV]
*Mikael Ståldal*
Senior software developer
*Magine TV
/src/main/scala/org/apache/spark/streaming/kafka/KafkaUtils.scala#L248
--
[image: MagineTV]
*Mikael Ståldal*
Senior software developer
*Magine TV*
mikael.stal...@magine.com
Grev Turegatan 3 | 114 46 Stockholm, Sweden | www.magine.com
Privileged and/or Confidential Information may be contained
