java.lang.StackOverflow Error How to sum across rows in a data frame with a large number of columns

[Andrew Davidson]

Hi

I have a dataframe of integers. It has 10409 columns. How can I sum across each 
row?

I get a very long stack trace

rowSums BEGIN
2022-01-20 22:11:24 ERROR __main__:? - An error occurred while calling 
o93935.withColumn.
: java.lang.StackOverflowError
                        at 
scala.collection.immutable.Set$SetBuilderImpl.$plus$eq(Set.scala:349)
                        at 
scala.collection.immutable.Set$SetBuilderImpl.$plus$eq(Set.scala:329)


###############################################################################

def rowSums( self, countsSparkDF, columnNames ):

    self.logger.warn( "rowSums BEGIN" )



    # https://stackoverflow.com/a/54283997/4586180

    retDF = countsSparkDF.na.fill( 0 ).withColumn( "rowSum" , reduce( add, 
[col( x ) for x in columnNames] ) )



    self.logger.warn( "rowSums retDF numRows:{} numCols:{}"\

                         .format( retDF.count(), len( retDF.columns ) ) )



    self.logger.warn( "rowSums END\n" )

    return retDF