rdd.flatMap(lambda x:x) maybe could solve your problem, it will convert an RDD from
[[[1,2,3],[4,5,6]], [[7,8,9,],[10,11,12]]] into: [[1,2,3], [4,5,6], [7,8,9,], [10,11,12]] On Mon, Aug 18, 2014 at 2:42 AM, Chengi Liu <chengi.liu...@gmail.com> wrote: > I have an rdd in pyspark which looks like follows: > It has two sub matrices..(array,array) > [ > array([[-13.00771575, 0.2740844 , 0.9752694 , 0.67465999, > -1.45741537, 0.546775 , 0.7900841 , -0.59473707, > -1.11752044, 0.61564356], > [ -0. , 12.20115746, -0.49016935, -0.9236129 , > -1.1693633 , -0.39135626, 1.10752864, 0.16920118, > -1.098806 , 1.10045185], > [ 0. , 0. , -11.26425992, 0.56309152, > 0.44872832, 0.69722768, 0.84200281, 0.89537327, > 0.10460865, -0.62938474], > [ -0. , -0. , 0. , 13.1112119 , > 0.39986223, -1.22218936, 0.72315955, 0.12208597, > -0.6258082 , -0.91077504], > [ 0. , -0. , 0. , 0. , > -11.04483145, -1.71948244, -0.73239228, -0.19651712, > -0.97931725, -0.43263423], > [ 0. , 0. , 0. , -0. , > 0. , -12.1996715 , -0.05580816, 0.20517336, > 0.53584998, 1.3370874 ], > [ 0. , -0. , -0. , 0. , > 0. , 0. , 12.32603631, 0.47498103, > -0.65901705, -0.85713277], > [ 0. , 0. , 0. , -0. , > 0. , -0. , 0. , 11.90030251, > 1.73036795, 0.70588443], > [ -0. , -0. , 0. , 0. , > -0. , -0. , 0. , -0. , > 13.00493769, 1.37753403], > [ 0. , -0. , 0. , 0. , > 0. , -0. , 0. , 0. , > -0. , -10.89006283]]), > > array([[-12.43375184, 1.07703619, -0.47818221, 1.65919732, > 0.96307502, -1.6322447 , -1.09409297, -0.64849112, > -1.09349557, -0.68706834], > [ 0. , -11.93616969, 0.08784614, 1.76677411, > -0.0585134 , -0.70979485, 0.29757848, 1.19562173, > -1.54176475, 1.71500862], > [ 0. , -0. , -12.42060272, 2.17060365, > -1.3212244 , 0.73742297, 0.50410937, -0.35278129, > -0.40513689, -0.81222302], > [ -0. , 0. , 0. , -11.93419851, > -1.15614929, 1.04085489, 0.69986351, -1.3615322 , > 0.43467842, -1.33041858], > [ -0. , -0. , 0. , 0. , > 11.22907137, -0.12925322, 0.46293906, -2.01577912, > -2.26566926, -0.17750339], > [ 0. , 0. , 0. , 0. , > -0. , -12.0705513 , -0.19432359, 0.41226088, > 0.79436699, -0.61288711], > [ 0. , -0. , 0. , 0. , > -0. , -0. , 11.99770753, -1.24277228, > 1.32240282, 1.5140609 ], > [ -0. , 0. , -0. , -0. , > 0. , -0. , 0. , -13.07008472, > 0.52031563, -1.56247391], > [ 0. , -0. , 0. , 0. , > -0. , -0. , -0. , -0. , > 13.16585107, 0.57741265], > [ 0. , 0. , -0. , -0. , > 0. , 0. , 0. , -0. , > -0. , -13.53719704]]) > ] > > So, basically I have sub matrices like [sub_matrix_1, sub_matrix_2.... ] > (the above has just two matrices.. > > I want to combine in one big matrix column wise > > [ sub_matrix_1 > sub_matrix_2 > .... > ] > Any suggestions? > --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
