I'm wondering why https://issues.apache.org/jira/browse/SPARK-3638
only updated the version of http client for the kinesis-asl profile and left the base dependencies unchanged. Spark built without that profile still has the same java.lang.NoSuchMethodError: org.apache.http.impl.conn.DefaultClientConnectionOperator.<init>(Lorg/apache/http/conn/scheme/SchemeRegistry;Lorg/apache/http/conn/DnsResolver;)V when using aws components in general, not just kinesis (e.g. AmazonCloudWatchAsyncClient) Re-Reading the "Dependency Hell in Spark applications" thread from september didn't shed any light on the subject. As noted in that thread, userClassPathFirst doesn't help. Is there a reason not to depend on an updated version of httpclient for all spark builds, as opposed to just kinesis-asl?