You can create a nested struct that contains multiple columns using
struct().

Here's a pretty complete guide on working with nested data:
https://databricks.com/blog/2017/02/23/working-complex-data-formats-structured-streaming-apache-spark-2-1.html

On Wed, Aug 23, 2017 at 2:30 PM, JG Perrin <jper...@lumeris.com> wrote:

> Hi folks,
>
>
>
> I am trying to join 2 dataframes, but I would like to have the result as a
> list of rows of the right dataframe (dDf in the example) in a column of the
> left dataframe (cDf in the example). I made it work with *one column*,
> but having issues adding more columns/creating a row(?).
>
>     Seq<String> joinColumns = new Set2<>("c1", "c2").toSeq();
>
>     Dataset<Row> allDf = cDf.join(dDf, joinColumns, "inner");
>
>     allDf.printSchema();
>
>     allDf.show();
>
>
>
>     Dataset<Row> aggDf = allDf.groupBy(cDf.col("c1"), cDf.col("c2"))
>
>             .agg(collect_list(col("c50")));
>
>     aggDf.show();
>
>
>
> Output:
>
> +--------+-------+---------------------------+
>
> |c1      |c2     |collect_list(c50)          |
>
> +--------+-------+---------------------------+
>
> |    3744|1160242|         [6, 5, 4, 3, 2, 1]|
>
> |    3739|1150097|                        [1]|
>
> |    3780|1159902|            [5, 4, 3, 2, 1]|
>
> |     132|1200743|               [4, 3, 2, 1]|
>
> |    3778|1183204|                        [1]|
>
> |    3766|1132709|                        [1]|
>
> |    3835|1146169|                        [1]|
>
> +--------+-------+---------------------------+
>
>
>
> Thanks,
>
>
>
> jg
>
>
> ------------------------------
>
> This electronic transmission and any documents accompanying this
> electronic transmission contain confidential information belonging to the
> sender. This information may contain confidential health information that
> is legally privileged. The information is intended only for the use of the
> individual or entity named above. The authorized recipient of this
> transmission is prohibited from disclosing this information to any other
> party unless required to do so by law or regulation and is required to
> delete or destroy the information after its stated need has been fulfilled.
> If you are not the intended recipient, you are hereby notified that any
> disclosure, copying, distribution or the taking of any action in reliance
> on or regarding the contents of this electronically transmitted information
> is strictly prohibited. If you have received this E-mail in error, please
> notify the sender and delete this message immediately.
>

Reply via email to