you can check if the value exists by using distinct before you loop over the dataset.
man. 1. mai 2023 kl. 10:38 skrev marc nicole <mk1853...@gmail.com>: > Hello > > I want to change values of a column in a dataset according to a mapping > list that maps original values of that column to other new values. Each > element of the list (colMappingValues) is a string that separates the > original values from the new values using a ";". > > So for a given column (in the following example colName), I do the > following processing to alter the column values as described: > > for (i=0;i<colMappingValues.size();i++){ >> >> //below lists contains all distinct values of a column >> (colMappingValues[i]) and their target values) >> allValuesChanges = colMappingValues[i].toString().split(";", 2); >> >> dataset = dataset.withColumn(colName, >> when(dataset.col(colName).equalTo(allValuesChanges[0])),allValuesChanges[1]).otherwise(dataset.col(colName)); > > } > > which is working but I want it to be efficient to avoid unnecessary > iterations. Meaning that I want when the column doesn't contain the value > from the list, the call to withColumn() gets ignored. > How to do exactly that in a more efficient way using Spark in Java? > > > -- Bjørn Jørgensen Vestre Aspehaug 4, 6010 Ålesund Norge +47 480 94 297