[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865163#comment-15865163 ] Billy Liu commented on KYLIN-2428: -- Hi [~Shaofengshi], the patch has rebased with the latest master with support on Spark IT. It passed IT test. Could you help review this patch? or check the code on the dev branch https://github.com/apache/kylin/commit/f10d9d9ba95878a97f70975547763083c000bcfe > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15865161#comment-15865161 ] Billy Liu commented on KYLIN-2428: -- [INFO] Apache Kylin ... SUCCESS [ 3.483 s] [INFO] Apache Kylin - Calcite Overrides ... SUCCESS [ 2.295 s] [INFO] Apache Kylin - Core Common . SUCCESS [ 5.559 s] [INFO] Apache Kylin - Core Metadata ... SUCCESS [ 26.885 s] [INFO] Apache Kylin - Core Dictionary . SUCCESS [01:18 min] [INFO] Apache Kylin - Core Cube ... SUCCESS [01:49 min] [INFO] Apache Kylin - Core Job SUCCESS [01:58 min] [INFO] Apache Kylin - Core Storage SUCCESS [ 35.183 s] [INFO] Apache Kylin - MapReduce Engine SUCCESS [ 22.122 s] [INFO] Apache Kylin - HBase Storage ... SUCCESS [ 14.430 s] [INFO] Apache Kylin - Spark Engine SUCCESS [ 9.362 s] [INFO] Apache Kylin - Hive Source . SUCCESS [ 4.243 s] [INFO] Apache Kylin - Kafka Source SUCCESS [ 3.351 s] [INFO] Apache Kylin - Query ... SUCCESS [ 2.604 s] [INFO] Apache Kylin - Tool SUCCESS [ 4.809 s] [INFO] Apache Kylin - REST Server Base SUCCESS [ 7.992 s] [INFO] Apache Kylin - REST Server . SUCCESS [ 39.903 s] [INFO] Apache Kylin - JDBC Driver . SUCCESS [ 4.899 s] [INFO] Apache Kylin - Assembly SUCCESS [ 6.212 s] [INFO] Apache Kylin - Tool Assembly ... SUCCESS [ 2.187 s] [INFO] Apache Kylin - Integration Test SUCCESS [ 01:03 h] [INFO] Apache Kylin - Tomcat Extension SUCCESS [ 0.633 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:12 h [INFO] Finished at: 2017-02-14T14:25:15+00:00 [INFO] Final Memory: 76M/856M > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15855491#comment-15855491 ] Billy Liu commented on KYLIN-2428: -- [INFO] Apache Kylin ... SUCCESS [ 3.045 s] [INFO] Apache Kylin - Calcite Overrides ... SUCCESS [ 2.161 s] [INFO] Apache Kylin - Core Common . SUCCESS [ 5.515 s] [INFO] Apache Kylin - Core Metadata ... SUCCESS [ 25.081 s] [INFO] Apache Kylin - Core Dictionary . SUCCESS [01:14 min] [INFO] Apache Kylin - Core Cube ... SUCCESS [01:44 min] [INFO] Apache Kylin - Core Job SUCCESS [01:58 min] [INFO] Apache Kylin - Core Storage SUCCESS [ 34.999 s] [INFO] Apache Kylin - MapReduce Engine SUCCESS [ 20.623 s] [INFO] Apache Kylin - HBase Storage ... SUCCESS [ 12.564 s] [INFO] Apache Kylin - Spark Engine SUCCESS [ 13.664 s] [INFO] Apache Kylin - Hive Source . SUCCESS [ 4.314 s] [INFO] Apache Kylin - Kafka Source SUCCESS [ 3.250 s] [INFO] Apache Kylin - Query ... SUCCESS [ 2.485 s] [INFO] Apache Kylin - Tool SUCCESS [ 4.199 s] [INFO] Apache Kylin - REST Server Base SUCCESS [ 6.802 s] [INFO] Apache Kylin - REST Server . SUCCESS [ 37.196 s] [INFO] Apache Kylin - JDBC Driver . SUCCESS [ 5.646 s] [INFO] Apache Kylin - Assembly SUCCESS [ 5.158 s] [INFO] Apache Kylin - Integration Test SUCCESS [ 01:08 h] [INFO] Apache Kylin - Tomcat Extension SUCCESS [ 2.810 s] [INFO] [INFO] BUILD SUCCESS [INFO] [INFO] Total time: 01:16 h [INFO] Finished at: 2017-02-07T15:27:23+00:00 [INFO] Final Memory: 75M/935M > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854211#comment-15854211 ] Billy Liu commented on KYLIN-2428: -- By refine the JDBC shade setting, the package has reduced from 7.6M to 5.4M, no duplicated shaded any more. > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2428) Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server
[ https://issues.apache.org/jira/browse/KYLIN-2428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15854147#comment-15854147 ] Billy Liu commented on KYLIN-2428: -- Compared with kylin 1.6: kylin-coprocessor is reduced from 2.4M to 1.9M, by removing extendedset, commons-lang3, but also introduced tduning.math kylin-jdbc increased from 7.6M to 8.5M since Avatica 1.9 upgrade. kylin-job is reduced from 9.5M to 5.8M, by removing esotericsoftware, guava, jsch, extendedset, jsr305, commons-cli, commons-io, commons-lang, curator and objenesis. Web application has reduced from 94 libraries to 80. > Cleanup unnecessary shaded libraries for job/coprocessor/jdbc/server > > > Key: KYLIN-2428 > URL: https://issues.apache.org/jira/browse/KYLIN-2428 > Project: Kylin > Issue Type: Improvement > Components: General >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu > Attachments: KYLIN-2428.patch > > > Kylin releases three libraries: kylin-coprocessor, kylin-jdbc, kylin-job and > one web application: server. > Currently, all libraries have shaded some used third party libraries into the > package. For example, guava, curator, commons, kyro in kylin-job. The > duplicate libraries in runtime classpath may have potential class loading > conflicts and waste computing resource. We should leverage the hadoop > provided libraries at runtime instead of the shaded one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
