[jira] [Commented] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438509#comment-16438509 ] Sercan Karaoglu commented on SPARK-23891: - So to summarize, as a user I want to have two things one is official spark images with all kinds of tags and second is I would like to customize those images in such a way that I can add my jars into it and seperate class loader loads them so that I have no conflicts with existing spark classpath. Existing classes may be shaded or not but either way app layer and spark layer should be isolated from each other. > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > Attachments: Dockerfile > > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438500#comment-16438500 ] Sercan Karaoglu commented on SPARK-23891: - I don't know if you guys want to do this but what I would suggest would be; if you take a look at here [https://hub.docker.com/r/library/openjdk/] , they have jdk and all kinds of tags to determine the underlying platform, because spark is another layer on top of jvm, there could have been an option to choose spark version plus jdk and distro version from docker-hub as official images and I think this should not be that hard since we have cool CI/CD tools today that can automate pretty much everything. If you look at docker hub there is no official supported spark images there yet. > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > Attachments: Dockerfile > > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16438499#comment-16438499 ] Sercan Karaoglu commented on SPARK-23891: - Sure! I've just attached it, and as a reference, this is another workaround to get netty-tcnative running in docker using alpine images. [https://github.com/pires/netty-tcnative-alpine] . > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > Attachments: Dockerfile > > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sercan Karaoglu updated SPARK-23891: Attachment: (was: Dockerfile) > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > Attachments: Dockerfile > > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sercan Karaoglu updated SPARK-23891: Attachment: Dockerfile > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > Attachments: Dockerfile > > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sercan Karaoglu updated SPARK-23891: Attachment: Dockerfile > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > Attachments: Dockerfile > > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436942#comment-16436942 ] Sercan Karaoglu edited comment on SPARK-23891 at 4/13/18 7:24 AM: -- Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find from most repos like maven central, then netty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here was (Author: sercankaraoglu): Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then netty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436942#comment-16436942 ] Sercan Karaoglu edited comment on SPARK-23891 at 4/13/18 7:23 AM: -- Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then netty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here was (Author: sercankaraoglu): Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then Betty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23891) Debian based Dockerfile
[ https://issues.apache.org/jira/browse/SPARK-23891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16436942#comment-16436942 ] Sercan Karaoglu commented on SPARK-23891: - Debian and centos based linux distros are the most popular ones even if nowadays we see people tend to use alpine based jdk images because of their size, we still have most libraries like netty has their native bindings built in both for centos and debian. As far as I know spark also have netty dependency, if you want to get less gc pressure more performance from the TCP layer you can add netty linux native bindings to the classpath through the jars that you can find most repos like maven central, then Betty automatically binds to those .so's my problem was specific to SSL communication with google big table, in this case google depends on netty tcnative library in their SDK. I had two ways to solve this problem, one is rebuilt the tcnative for alpine and exclude .so from classpath coming through existing jars and add my custom built .so to the classpath, the other way was to change base image which is way easier. I solved this problem by customizing that dockerfile you provide as a reference and would like to report the issue here > Debian based Dockerfile > --- > > Key: SPARK-23891 > URL: https://issues.apache.org/jira/browse/SPARK-23891 > Project: Spark > Issue Type: New Feature > Components: Kubernetes >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > > Current dockerfile inherits from alpine linux which causes netty tcnative ssl > bindings to fail while loading which is the case when we use Google Cloud > Platforms Bigtable Client on top of spark cluster. would be better to have > another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-23897) Guava version
[ https://issues.apache.org/jira/browse/SPARK-23897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16429868#comment-16429868 ] Sercan Karaoglu commented on SPARK-23897: - What about shading it > Guava version > - > > Key: SPARK-23897 > URL: https://issues.apache.org/jira/browse/SPARK-23897 > Project: Spark > Issue Type: Dependency upgrade > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Sercan Karaoglu >Priority: Minor > > Guava dependency version 14 is pretty old, needs to be updated to at least > 16, google cloud storage connector uses newer one which causes pretty popular > error with guava; "java.lang.NoSuchMethodError: > com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List;" > and causes app to crash -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23897) Guava version
Sercan Karaoglu created SPARK-23897: --- Summary: Guava version Key: SPARK-23897 URL: https://issues.apache.org/jira/browse/SPARK-23897 Project: Spark Issue Type: Dependency upgrade Components: Spark Core Affects Versions: 2.3.0 Reporter: Sercan Karaoglu Guava dependency version 14 is pretty old, needs to be updated to at least 16, google cloud storage connector uses newer one which causes pretty popular error with guava; "java.lang.NoSuchMethodError: com.google.common.base.Splitter.splitToList(Ljava/lang/CharSequence;)Ljava/util/List;" and causes app to crash -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23891) Debian based Dockerfile
Sercan Karaoglu created SPARK-23891: --- Summary: Debian based Dockerfile Key: SPARK-23891 URL: https://issues.apache.org/jira/browse/SPARK-23891 Project: Spark Issue Type: Bug Components: Kubernetes Affects Versions: 2.3.0 Reporter: Sercan Karaoglu Current dockerfile inherits from alpine linux which causes netty tcnative ssl bindings to fail while loading which is the case when we use Google Cloud Platforms Bigtable Client on top of spark cluster. would be better to have another debian based dockerfile -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
