Re: Kind help request
It is telling you that the UI can't bind to any port. I presume that's because of container restrictions? If you don't want the UI at all, just set spark.ui.enabled to false On Sat, Mar 25, 2023 at 8:28 AM Lorenzo Ferrando < lorenzo.ferra...@edu.unige.it> wrote: > Dear Spark team, > > I am Lorenzo from University of Genoa. I am currently using (ubuntu 18.04) > the nextflow/sarek pipeline to analyse genomic data through a singularity > container. One of the step of the pipeline uses GATK4 and it implements > Spark. However, after some time I get the following error: > > > 23:27:48.112 INFO NativeLibraryLoader - Loading libgkl_compression.so from > jar:file:/gatk/gatk-package-4.2.6.1-local.jar!/com/intel/gkl/native/libgkl_compression.so > 23:27:48.523 INFO ApplyBQSRSpark - > > 23:27:48.524 INFO ApplyBQSRSpark - The Genome Analysis Toolkit (GATK) > v4.2.6.1 > 23:27:48.524 INFO ApplyBQSRSpark - For support and documentation go to > https://software.broadinstitute.org/gatk/ > 23:27:48.525 INFO ApplyBQSRSpark - Executing as ferrandl@alucard on Linux > v5.4.0-91-generic amd64 > 23:27:48.525 INFO ApplyBQSRSpark - Java runtime: OpenJDK 64-Bit Server VM > v1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 > 23:27:48.526 INFO ApplyBQSRSpark - Start Date/Time: March 24, 2023 11:27:47 > PM GMT > 23:27:48.526 INFO ApplyBQSRSpark - > > 23:27:48.526 INFO ApplyBQSRSpark - > > 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Version: 2.24.1 > 23:27:48.527 INFO ApplyBQSRSpark - Picard Version: 2.27.1 > 23:27:48.527 INFO ApplyBQSRSpark - Built for Spark Version: 2.4.5 > 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 > 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK > Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false > 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK > Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true > 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK > Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false > 23:27:48.527 INFO ApplyBQSRSpark - Deflater: IntelDeflater > 23:27:48.528 INFO ApplyBQSRSpark - Inflater: IntelInflater > 23:27:48.528 INFO ApplyBQSRSpark - GCS max retries/reopens: 20 > 23:27:48.528 INFO ApplyBQSRSpark - Requester pays: disabled > 23:27:48.528 WARN ApplyBQSRSpark - > > > >Warning: ApplyBQSRSpark is a BETA tool and is not yet ready for use in > production > > > > > 23:27:48.528 INFO ApplyBQSRSpark - Initializing engine > 23:27:48.528 INFO ApplyBQSRSpark - Done initializing engine > Using Spark's default log4j profile: > org/apache/spark/log4j-defaults.properties > 23/03/24 23:27:49 INFO SparkContext: Running Spark version 2.4.5 > 23/03/24 23:27:49 WARN NativeCodeLoader: Unable to load native-hadoop library > for your platform... using builtin-java classes where applicable > 23/03/24 23:27:50 INFO SparkContext: Submitted application: ApplyBQSRSpark > 23/03/24 23:27:50 INFO SecurityManager: Changing view acls to: ferrandl > 23/03/24 23:27:50 INFO SecurityManager: Changing modify acls to: ferrandl > 23/03/24 23:27:50 INFO SecurityManager: Changing view acls groups to: > 23/03/24 23:27:50 INFO SecurityManager: Changing modify acls groups to: > 23/03/24 23:27:50 INFO SecurityManager: SecurityManager: authentication > disabled; ui acls disabled; users with view permissions: Set(ferrandl); > groups with view permissions: Set(); users with modify permissions: > Set(ferrandl); groups with modify permissions: Set() > 23/03/24 23:27:50 INFO Utils: Successfully started service 'sparkDriver' on > port 46757. > 23/03/24 23:27:50 INFO SparkEnv: Registering MapOutputTracker > 23/03/24 23:27:50 INFO SparkEnv: Registering BlockManagerMaster > 23/03/24 23:27:50 INFO BlockManagerMasterEndpoint: Using > org.apache.spark.storage.DefaultTopologyMapper for getting topology > information > 23/03/24 23:27:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint > up > 23/03/24 23:27:50 INFO DiskBlockManager: Created local directory at > /home/ferrandl/projects/ribas_reanalysis/sarek/work/27/89b7451fcac6fd31461885b5774752/blockmgr-e76f7d59-da0b-4e62-8a99-3cdb23f11ae6 > 23/03/24 23:27:50 INFO MemoryStore: MemoryStore started with capacity 2004.6 > MB > 23/03/24 23:27:50 INFO SparkEnv: Registering OutputCommitCoordinator > 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4040. > Attempting port 4041. > 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4041. > Attempting port 4042. > 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4042. > Attempting port 4043. > 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4043. > Attempting port 4044. > 23/03/24 23:27:51 WARN Utils: Service
Kind help request
Dear Spark team, I am Lorenzo from University of Genoa. I am currently using (ubuntu 18.04) the nextflow/sarek pipeline to analyse genomic data through a singularity container. One of the step of the pipeline uses GATK4 and it implements Spark. However, after some time I get the following error: 23:27:48.112 INFO NativeLibraryLoader - Loading libgkl_compression.so from jar:file:/gatk/gatk-package-4.2.6.1-local.jar!/com/intel/gkl/native/libgkl_compression.so 23:27:48.523 INFO ApplyBQSRSpark - 23:27:48.524 INFO ApplyBQSRSpark - The Genome Analysis Toolkit (GATK) v4.2.6.1 23:27:48.524 INFO ApplyBQSRSpark - For support and documentation go to https://software.broadinstitute.org/gatk/ 23:27:48.525 INFO ApplyBQSRSpark - Executing as ferrandl@alucard on Linux v5.4.0-91-generic amd64 23:27:48.525 INFO ApplyBQSRSpark - Java runtime: OpenJDK 64-Bit Server VM v1.8.0_242-8u242-b08-0ubuntu3~18.04-b08 23:27:48.526 INFO ApplyBQSRSpark - Start Date/Time: March 24, 2023 11:27:47 PM GMT 23:27:48.526 INFO ApplyBQSRSpark - 23:27:48.526 INFO ApplyBQSRSpark - 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Version: 2.24.1 23:27:48.527 INFO ApplyBQSRSpark - Picard Version: 2.27.1 23:27:48.527 INFO ApplyBQSRSpark - Built for Spark Version: 2.4.5 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Defaults.COMPRESSION_LEVEL : 2 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Defaults.USE_ASYNC_IO_READ_FOR_SAMTOOLS : false 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_SAMTOOLS : true 23:27:48.527 INFO ApplyBQSRSpark - HTSJDK Defaults.USE_ASYNC_IO_WRITE_FOR_TRIBBLE : false 23:27:48.527 INFO ApplyBQSRSpark - Deflater: IntelDeflater 23:27:48.528 INFO ApplyBQSRSpark - Inflater: IntelInflater 23:27:48.528 INFO ApplyBQSRSpark - GCS max retries/reopens: 20 23:27:48.528 INFO ApplyBQSRSpark - Requester pays: disabled 23:27:48.528 WARN ApplyBQSRSpark - Warning: ApplyBQSRSpark is a BETA tool and is not yet ready for use in production 23:27:48.528 INFO ApplyBQSRSpark - Initializing engine 23:27:48.528 INFO ApplyBQSRSpark - Done initializing engine Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties 23/03/24 23:27:49 INFO SparkContext: Running Spark version 2.4.5 23/03/24 23:27:49 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 23/03/24 23:27:50 INFO SparkContext: Submitted application: ApplyBQSRSpark 23/03/24 23:27:50 INFO SecurityManager: Changing view acls to: ferrandl 23/03/24 23:27:50 INFO SecurityManager: Changing modify acls to: ferrandl 23/03/24 23:27:50 INFO SecurityManager: Changing view acls groups to: 23/03/24 23:27:50 INFO SecurityManager: Changing modify acls groups to: 23/03/24 23:27:50 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(ferrandl); groups with view permissions: Set(); users with modify permissions: Set(ferrandl); groups with modify permissions: Set() 23/03/24 23:27:50 INFO Utils: Successfully started service 'sparkDriver' on port 46757. 23/03/24 23:27:50 INFO SparkEnv: Registering MapOutputTracker 23/03/24 23:27:50 INFO SparkEnv: Registering BlockManagerMaster 23/03/24 23:27:50 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information 23/03/24 23:27:50 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up 23/03/24 23:27:50 INFO DiskBlockManager: Created local directory at /home/ferrandl/projects/ribas_reanalysis/sarek/work/27/89b7451fcac6fd31461885b5774752/blockmgr-e76f7d59-da0b-4e62-8a99-3cdb23f11ae6 23/03/24 23:27:50 INFO MemoryStore: MemoryStore started with capacity 2004.6 MB 23/03/24 23:27:50 INFO SparkEnv: Registering OutputCommitCoordinator 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4040. Attempting port 4041. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4041. Attempting port 4042. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4042. Attempting port 4043. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4043. Attempting port 4044. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4044. Attempting port 4045. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4045. Attempting port 4046. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4046. Attempting port 4047. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4047. Attempting port 4048. 23/03/24 23:27:51 WARN Utils: Service 'SparkUI' could not bind on port 4048. Attempting port 4049. 23/03/24
