Re: Training Failure in Clustering : Execution mode singlenode

[Matthias Boehm]

ok this has been fixed in master - it was essentially an issue of size
propagation for ctable with two sequence inputs.

Regards,
Matthias

On Sat, Jul 29, 2017 at 1:57 PM, Matthias Boehm 
wrote:

> Hi Krishna,
>
> I just gave Kmeans a try with your parameters and it runs fine with
> hybrid_spark (default) through spark submit. However, I'm able to reproduce
> the issue when forcing it into singlenode. Thanks for catching this - I'll
> take care of it.
>
> Regards,
> Matthias
>
>
> On Sat, Jul 29, 2017 at 9:34 AM, Krishna Kalyan 
> wrote:
>
>> Hello All,
>> I am faced with an error below while trying to train Kmeans algorithm.
>>  ERROR: Runtime error in program block generated from statement block
>> between lines 94 and 120
>>
>> Could someone please guide with the fix.
>>
>> Full Error Stack Trace :
>> https://gist.github.com/krishnakalyan3/1225efa5cc13a53a3b4f859b10aefe79
>>
>> Regards,
>> Krishna
>>
>
>

Build failed in Jenkins: SystemML-DailyTest #1141

[jenkins]

See 

Changes:

[Matthias Boehm] [SYSTEMML-1817] Fix hop construction for full standard dev 
computation

--
[...truncated 29391 lines...]
17/07/29 15:53:37 INFO scheduler.DAGScheduler: running: Set(ShuffleMapStage 1)
17/07/29 15:53:37 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2)
17/07/29 15:53:37 INFO scheduler.DAGScheduler: failed: Set()
17/07/29 15:53:37 INFO scheduler.DAGScheduler: ShuffleMapStage 1 
(parallelizePairs at SparkExecutionContext.java:699) finished in 0.123 s
17/07/29 15:53:37 INFO scheduler.DAGScheduler: looking for newly runnable stages
17/07/29 15:53:37 INFO scheduler.DAGScheduler: running: Set()
17/07/29 15:53:37 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 2)
17/07/29 15:53:37 INFO scheduler.DAGScheduler: failed: Set()
17/07/29 15:53:37 INFO scheduler.DAGScheduler: Submitting ResultStage 2 
(MapPartitionsRDD[5] at mapValues at BinarySPInstruction.java:117), which has 
no missing parents
17/07/29 15:53:37 INFO memory.MemoryStore: Block broadcast_2 stored as values 
in memory (estimated size 4.2 KB, free 1033.8 MB)
17/07/29 15:53:37 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as 
bytes in memory (estimated size 2.3 KB, free 1033.8 MB)
17/07/29 15:53:37 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in 
memory on 23.246.248.181:37857 (size: 2.3 KB, free: 1033.8 MB)
17/07/29 15:53:37 INFO spark.SparkContext: Created broadcast 2 from broadcast 
at DAGScheduler.scala:996
17/07/29 15:53:37 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from 
ResultStage 2 (MapPartitionsRDD[5] at mapValues at BinarySPInstruction.java:117)
17/07/29 15:53:37 INFO scheduler.TaskSchedulerImpl: Adding task set 2.0 with 1 
tasks
17/07/29 15:53:37 INFO scheduler.FairSchedulableBuilder: Added task set 
TaskSet_2.0 tasks to pool default
17/07/29 15:53:37 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 2.0 
(TID 2, localhost, executor driver, partition 0, PROCESS_LOCAL, 5813 bytes)
17/07/29 15:53:37 INFO executor.Executor: Running task 0.0 in stage 2.0 (TID 2)
17/07/29 15:53:37 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty 
blocks out of 1 blocks
17/07/29 15:53:37 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote 
fetches in 4 ms
17/07/29 15:53:37 INFO storage.ShuffleBlockFetcherIterator: Getting 1 non-empty 
blocks out of 1 blocks
17/07/29 15:53:37 INFO storage.ShuffleBlockFetcherIterator: Started 0 remote 
fetches in 0 ms
17/07/29 15:53:37 INFO storage.BlockManagerInfo: Removed broadcast_1_piece0 on 
23.246.248.181:37857 in memory (size: 1302.0 B, free: 1033.8 MB)
17/07/29 15:53:37 INFO storage.BlockManagerInfo: Removed broadcast_0_piece0 on 
23.246.248.181:37857 in memory (size: 1305.0 B, free: 1033.8 MB)
17/07/29 15:53:37 INFO executor.Executor: Finished task 0.0 in stage 2.0 (TID 
2). 2150 bytes result sent to driver
17/07/29 15:53:37 INFO scheduler.TaskSetManager: Finished task 0.0 in stage 2.0 
(TID 2) in 58 ms on localhost (executor driver) (1/1)
17/07/29 15:53:37 INFO scheduler.TaskSchedulerImpl: Removed TaskSet 2.0, whose 
tasks have all completed, from pool default
17/07/29 15:53:37 INFO scheduler.DAGScheduler: ResultStage 2 (collect at 
SparkExecutionContext.java:789) finished in 0.059 s
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Registering RDD 7 
(parallelizePairs at SparkExecutionContext.java:699)
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Registering RDD 6 
(parallelizePairs at SparkExecutionContext.java:699)
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Got job 1 (collect at 
SparkExecutionContext.java:789) with 1 output partitions
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Final stage: ResultStage 5 
(collect at SparkExecutionContext.java:789)
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Parents of final stage: 
List(ShuffleMapStage 3, ShuffleMapStage 4)
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Missing parents: 
List(ShuffleMapStage 3, ShuffleMapStage 4)
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Submitting ShuffleMapStage 3 
(ParallelCollectionRDD[7] at parallelizePairs at 
SparkExecutionContext.java:699), which has no missing parents
17/07/29 15:53:38 INFO memory.MemoryStore: Block broadcast_3 stored as values 
in memory (estimated size 2032.0 B, free 1033.8 MB)
17/07/29 15:53:38 INFO memory.MemoryStore: Block broadcast_3_piece0 stored as 
bytes in memory (estimated size 1305.0 B, free 1033.8 MB)
17/07/29 15:53:38 INFO storage.BlockManagerInfo: Added broadcast_3_piece0 in 
memory on 23.246.248.181:37857 (size: 1305.0 B, free: 1033.8 MB)
17/07/29 15:53:38 INFO spark.SparkContext: Created broadcast 3 from broadcast 
at DAGScheduler.scala:996
17/07/29 15:53:38 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from 
ShuffleMapStage 3 (ParallelCollectionRDD[7] at parallelizePairs at 
SparkExecutionContext.java:699)
17/07/29 15:53:38 INFO scheduler.TaskSchedulerImpl: Adding 

Re: Training Failure in Clustering : Execution mode singlenode

[Matthias Boehm]

Hi Krishna,

I just gave Kmeans a try with your parameters and it runs fine with
hybrid_spark (default) through spark submit. However, I'm able to reproduce
the issue when forcing it into singlenode. Thanks for catching this - I'll
take care of it.

Regards,
Matthias


On Sat, Jul 29, 2017 at 9:34 AM, Krishna Kalyan 
wrote:

> Hello All,
> I am faced with an error below while trying to train Kmeans algorithm.
>  ERROR: Runtime error in program block generated from statement block
> between lines 94 and 120
>
> Could someone please guide with the fix.
>
> Full Error Stack Trace :
> https://gist.github.com/krishnakalyan3/1225efa5cc13a53a3b4f859b10aefe79
>
> Regards,
> Krishna
>

Re: svd( ) implementation

[Imran Younus]

Hi Janardhan,

By tall-skinny matrix I mean that the number of columns should be
reasonably small (doesn't matter how many rows are there) . When you do QR
decomposition, R would a square matrix (upper triangular) of dimensions
equal to the number of columns of matrix A. But then R has to fit on
driver's memory, and thats why number of columns of A cannot be be very
large. (10k x 10k matrix will close to 1 GB). Once you've R, a local SVD
implementation will be needed to to compute SVD of R. Now, this is not a
very general method, but I think this is good enough for most of the cases.

imran

On Fri, Jul 28, 2017 at 10:37 PM, Janardhan Pulivarthi <
janardhan.pulivar...@gmail.com> wrote:

> Thanks, Imran. As per the paper, at first we perform QR decomposition of
> input matrix (A), from which we obtain R. And then, we compute the svd(R)
> using the builtin local function (??). I'll try this.
>
> Tall-skinny matrix: so, do we have problem with square matrices?. or do we
> have to partition the matrix into tall-skinny matrices if we have a square
> one?.
>
> Thanks,
>
> Janardhan
>
> On Fri, Jul 28, 2017 at 11:52 PM, Imran Younus 
> wrote:
>
> > Just to clarify one thing. For QR based, method, you can assume that R
> > matrix is small enough to fit on driver memory and them perform SVD on
> the
> > driver. That means your actual matrix has to tall-skinny matrix.
> >
> > imran
> >
> > On Fri, Jul 28, 2017 at 11:15 AM, Imran Younus 
> > wrote:
> >
> > > Janardhan,
> > >
> > > The papers you're referring may not be relevant. The first paper, as
> far
> > > as I can tell, is about updating an existing svd decomposition as new
> > data
> > > comes in. The 3rd paper in this list is the one I used, but that method
> > is
> > > not good.
> > >
> > > There is also a method that uses QR decomposition and then calculates
> SVD
> > > from R matrix. Please have a look at equation 1.3 in this paper:
> > >
> > > http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.127.115=1
> > >
> > > I think this is worth trying out. The distributed QR is already
> > > implemented in SystemlML, so it may quick to try out.
> > >
> > > imran
> > >
> > >
> > >
> > > On Fri, Jul 28, 2017 at 10:10 AM, Janardhan Pulivarthi <
> > > janardhan.pulivar...@gmail.com> wrote:
> > >
> > >> Hi Nakul & all the committers,
> > >>
> > >> Till now I am half way through the literature. But, for now a couple
> of
> > >> things to mention, in SVD there are three stages
> > >>   1. Bidiagonal reduction step
> > >>   2. Computation of the singular values
> > >>   3. Computation of the singular vectors
> > >>
> > >> of these three, The* Bidiagonal reduction* step is very expensive, so
> is
> > >> our focus on this( when considering GPU, at times where handling with
> > CPU
> > >> is infeasible).
> > >>
> > >> About literature:
> > >>
> > >>- I took some time to go through " A Stable and Fast Algorithm for
> > >>Updating the Singular Value Decomposition" by "Gu & Stanley", to
> > >> understand
> > >>the numerical stability and round-off errors when we are
> partitioning
> > >> the
> > >>matrix in this distributed algorithm. The author has assured that
> > each
> > >>component computed will be of high absolute accuracy. And also, the
> > >>properties that the resultant matrix support do not have any
> > conflicts
> > >> with
> > >>parent matrix. [pdf
> > >>]
> > >>
> > >>
> > >>- "High performance bidiagonal reduction using the tile algorithms
> on
> > >>homogeneous multicore clusters ", by "Ltaief et. al", this paper
> has
> > >>focused on the first stage mainly and has discussed a good about
> tile
> > >>algorithms and their runtime implementations.(although off-topic, I
> > >> read
> > >>this just to understand.) [pdf
> > >>]
> > >>
> > >>
> > >>-  "A distributed and incremental svd algorithm for agglomerative
> > data
> > >>analysis on large networks", by "Iwen & Ong", *Please go through*
> the
> > >>(a). TABLE 1, TABLE 2 . (b). APPENDIX A. RAW DATA FROM NUMERICAL
> > >>EXPERIMENTS. [pdf ]
> > >>
> > >> Thanks,
> > >>
> > >> Janardhan
> > >>
> > >> On Wed, Jul 26, 2017 at 12:29 AM, Nakul Jindal 
> > wrote:
> > >>
> > >> > Hi Janardhan,
> > >> >
> > >> > The images you've used as attachments haven't reached my inbox.
> > >> > Could you please send them to me directly, rather than through the
> dev
> > >> > mailing list.
> > >> > (Or upload it to a image hosting site like imgur and paste the links
> > in
> > >> the
> > >> > email)
> > >> >
> > >> > I would like to point out that my knowledge of machine learning is
> > >> limited.
> > >> > Still, how would you want to test the algorithm?
> > >> >
> > >> >
> > >> > Sparse matrices in SystemML (in Spark Execution Mode)
> > >> > 

Training Failure in Clustering : Execution mode singlenode

[Krishna Kalyan]

Hello All,
I am faced with an error below while trying to train Kmeans algorithm.
 ERROR: Runtime error in program block generated from statement block
between lines 94 and 120

Could someone please guide with the fix.

Full Error Stack Trace :
https://gist.github.com/krishnakalyan3/1225efa5cc13a53a3b4f859b10aefe79

Regards,
Krishna