Re: Considering deprecation and removal of XZ compression (hbase-compression-xz)
+1 same here. gzip/lzo in the past, Snappy or zstd now. On Tue, Apr 2, 2024 at 7:50 PM 张铎(Duo Zhang) wrote: > For me I've never seen people actually use the xz compression. > > For size, usually people will choose gzip, and for speed, in the past > people will choose lzo and now they choose snappy or zstd. > > So for me I prefer we just deprecated the xz compression immediately > and remove it 2.6.0. > > Thanks. > > Andrew Purtell 于2024年4月2日周二 08:02写道： > > > > Red Hat filed CVE-2024-3094 late last week on 2024-03-29. This implicates > > recent releases of the native liblzma library as a vector for malicious > > code. > > > > This is not the pure Java version that we depend upon for HBase's support > > for the LZMA algorithm ( > > > https://github.com/apache/hbase/tree/master/hbase-compression/hbase-compression-xz > ). > > We depend on version 1.9 of xz-java, which was published in 2021, well > > before maintenance changes in the project and the involvement of a person > > who is now believed to be a malicious actor. Projects like HBase that > > depend on xz-java have no reason to be concerned about the issues > affecting > > the native xz library. > > > > How the backdoor was introduced calls into question the trustworthiness > and > > viability of the XZ project. GitHub has disabled all repositories related > > to XZ and liblzma, even xz-java. The webpage for XZ and xz-java is down. > > The open source software community is responding vigorously. > CVE-2024-3094 > > has a CVSS score 10, the highest possible score. Your security team may > > become interested in HBase because of hbase-compression-xz's dependency > on > > xz-java. It is likely any discovered dependency on any LZMA > implementation > > will at least raise questions. > > > > For now xz-java remains available in Maven central. (See > > https://central.sonatype.com/artifact/org.tukaani/xz/versions) We may > have > > no choice but to immediately remove hbase-compression-xz if Maven blocks > or > > drops xz-java too, because that will break our builds. > > > > There is no immediate cause for concern. Still, we believe XZ compression > > provides little to no value over more modern alternatives, like > ZStandard, > > that can also achieve similar compression ratios. XZ, and alternatives > like > > ZStandard with the compression level set to a high value, are also > suitable > > only for archival use cases and unsuitable for compression of flush files > > or for use in minor compactions. Given how niche any use of XZ > > compression could > > be, we are wondering if there are actually any users of it. > > > > If we have no users of hbase-compression-xz, then it provides little to > no > > value and continued maintenance of hbase-compression-xz given the issues > > with its dependency does not make sense. > > > > Do you use XZ compression, or are you planning to? > > > > If we deprecate XZ compression immediately and then remove it in 2.6, > would > > this present a problem? In a private discussion we reached consensus on > > this approach, but, of course, that is not yet a plan, and something that > > could easily change based on feedback. > > > > From https://nvd.nist.gov/vuln/detail/CVE-2024-3094: > > "Malicious code was discovered in the upstream tarballs of xz, starting > > with version 5.6.0. Through a series of complex obfuscations, the liblzma > > build process extracts a prebuilt object file from a disguised test file > > existing in the source code, which is then used to modify specific > > functions in the liblzma code. This results in a modified liblzma library > > that can be used by any software linked against this library, > intercepting > > and modifying the data interaction with this library." > > > > -- > > Best regards, > > Andrew >
Re: Considering deprecation and removal of XZ compression (hbase-compression-xz)
Let's remove in 2.6.0. I will submit a PR. On Tue, Apr 2, 2024 at 7:50 PM 张铎(Duo Zhang) wrote: > For me I've never seen people actually use the xz compression. > > For size, usually people will choose gzip, and for speed, in the past > people will choose lzo and now they choose snappy or zstd. > > So for me I prefer we just deprecated the xz compression immediately > and remove it 2.6.0. > > Thanks. > > Andrew Purtell 于2024年4月2日周二 08:02写道： > > > > Red Hat filed CVE-2024-3094 late last week on 2024-03-29. This implicates > > recent releases of the native liblzma library as a vector for malicious > > code. > > > > This is not the pure Java version that we depend upon for HBase's support > > for the LZMA algorithm ( > > > https://github.com/apache/hbase/tree/master/hbase-compression/hbase-compression-xz > ). > > We depend on version 1.9 of xz-java, which was published in 2021, well > > before maintenance changes in the project and the involvement of a person > > who is now believed to be a malicious actor. Projects like HBase that > > depend on xz-java have no reason to be concerned about the issues > affecting > > the native xz library. > > > > How the backdoor was introduced calls into question the trustworthiness > and > > viability of the XZ project. GitHub has disabled all repositories related > > to XZ and liblzma, even xz-java. The webpage for XZ and xz-java is down. > > The open source software community is responding vigorously. > CVE-2024-3094 > > has a CVSS score 10, the highest possible score. Your security team may > > become interested in HBase because of hbase-compression-xz's dependency > on > > xz-java. It is likely any discovered dependency on any LZMA > implementation > > will at least raise questions. > > > > For now xz-java remains available in Maven central. (See > > https://central.sonatype.com/artifact/org.tukaani/xz/versions) We may > have > > no choice but to immediately remove hbase-compression-xz if Maven blocks > or > > drops xz-java too, because that will break our builds. > > > > There is no immediate cause for concern. Still, we believe XZ compression > > provides little to no value over more modern alternatives, like > ZStandard, > > that can also achieve similar compression ratios. XZ, and alternatives > like > > ZStandard with the compression level set to a high value, are also > suitable > > only for archival use cases and unsuitable for compression of flush files > > or for use in minor compactions. Given how niche any use of XZ > > compression could > > be, we are wondering if there are actually any users of it. > > > > If we have no users of hbase-compression-xz, then it provides little to > no > > value and continued maintenance of hbase-compression-xz given the issues > > with its dependency does not make sense. > > > > Do you use XZ compression, or are you planning to? > > > > If we deprecate XZ compression immediately and then remove it in 2.6, > would > > this present a problem? In a private discussion we reached consensus on > > this approach, but, of course, that is not yet a plan, and something that > > could easily change based on feedback. > > > > From https://nvd.nist.gov/vuln/detail/CVE-2024-3094: > > "Malicious code was discovered in the upstream tarballs of xz, starting > > with version 5.6.0. Through a series of complex obfuscations, the liblzma > > build process extracts a prebuilt object file from a disguised test file > > existing in the source code, which is then used to modify specific > > functions in the liblzma code. This results in a modified liblzma library > > that can be used by any software linked against this library, > intercepting > > and modifying the data interaction with this library." > > > > -- > > Best regards, > > Andrew > -- Best regards, Andrew Unrest, ignorance distilled, nihilistic imbeciles - It's what we’ve earned Welcome, apocalypse, what’s taken you so long? Bring us the fitting end that we’ve been counting on - A23, Welcome, Apocalypse
Re: Considering deprecation and removal of XZ compression (hbase-compression-xz)
For me I've never seen people actually use the xz compression. For size, usually people will choose gzip, and for speed, in the past people will choose lzo and now they choose snappy or zstd. So for me I prefer we just deprecated the xz compression immediately and remove it 2.6.0. Thanks. Andrew Purtell 于2024年4月2日周二 08:02写道： > > Red Hat filed CVE-2024-3094 late last week on 2024-03-29. This implicates > recent releases of the native liblzma library as a vector for malicious > code. > > This is not the pure Java version that we depend upon for HBase's support > for the LZMA algorithm ( > https://github.com/apache/hbase/tree/master/hbase-compression/hbase-compression-xz). > We depend on version 1.9 of xz-java, which was published in 2021, well > before maintenance changes in the project and the involvement of a person > who is now believed to be a malicious actor. Projects like HBase that > depend on xz-java have no reason to be concerned about the issues affecting > the native xz library. > > How the backdoor was introduced calls into question the trustworthiness and > viability of the XZ project. GitHub has disabled all repositories related > to XZ and liblzma, even xz-java. The webpage for XZ and xz-java is down. > The open source software community is responding vigorously. CVE-2024-3094 > has a CVSS score 10, the highest possible score. Your security team may > become interested in HBase because of hbase-compression-xz's dependency on > xz-java. It is likely any discovered dependency on any LZMA implementation > will at least raise questions. > > For now xz-java remains available in Maven central. (See > https://central.sonatype.com/artifact/org.tukaani/xz/versions) We may have > no choice but to immediately remove hbase-compression-xz if Maven blocks or > drops xz-java too, because that will break our builds. > > There is no immediate cause for concern. Still, we believe XZ compression > provides little to no value over more modern alternatives, like ZStandard, > that can also achieve similar compression ratios. XZ, and alternatives like > ZStandard with the compression level set to a high value, are also suitable > only for archival use cases and unsuitable for compression of flush files > or for use in minor compactions. Given how niche any use of XZ > compression could > be, we are wondering if there are actually any users of it. > > If we have no users of hbase-compression-xz, then it provides little to no > value and continued maintenance of hbase-compression-xz given the issues > with its dependency does not make sense. > > Do you use XZ compression, or are you planning to? > > If we deprecate XZ compression immediately and then remove it in 2.6, would > this present a problem? In a private discussion we reached consensus on > this approach, but, of course, that is not yet a plan, and something that > could easily change based on feedback. > > From https://nvd.nist.gov/vuln/detail/CVE-2024-3094: > "Malicious code was discovered in the upstream tarballs of xz, starting > with version 5.6.0. Through a series of complex obfuscations, the liblzma > build process extracts a prebuilt object file from a disguised test file > existing in the source code, which is then used to modify specific > functions in the liblzma code. This results in a modified liblzma library > that can be used by any software linked against this library, intercepting > and modifying the data interaction with this library." > > -- > Best regards, > Andrew
Considering deprecation and removal of XZ compression (hbase-compression-xz)
Red Hat filed CVE-2024-3094 late last week on 2024-03-29. This implicates recent releases of the native liblzma library as a vector for malicious code. This is not the pure Java version that we depend upon for HBase's support for the LZMA algorithm ( https://github.com/apache/hbase/tree/master/hbase-compression/hbase-compression-xz). We depend on version 1.9 of xz-java, which was published in 2021, well before maintenance changes in the project and the involvement of a person who is now believed to be a malicious actor. Projects like HBase that depend on xz-java have no reason to be concerned about the issues affecting the native xz library. How the backdoor was introduced calls into question the trustworthiness and viability of the XZ project. GitHub has disabled all repositories related to XZ and liblzma, even xz-java. The webpage for XZ and xz-java is down. The open source software community is responding vigorously. CVE-2024-3094 has a CVSS score 10, the highest possible score. Your security team may become interested in HBase because of hbase-compression-xz's dependency on xz-java. It is likely any discovered dependency on any LZMA implementation will at least raise questions. For now xz-java remains available in Maven central. (See https://central.sonatype.com/artifact/org.tukaani/xz/versions) We may have no choice but to immediately remove hbase-compression-xz if Maven blocks or drops xz-java too, because that will break our builds. There is no immediate cause for concern. Still, we believe XZ compression provides little to no value over more modern alternatives, like ZStandard, that can also achieve similar compression ratios. XZ, and alternatives like ZStandard with the compression level set to a high value, are also suitable only for archival use cases and unsuitable for compression of flush files or for use in minor compactions. Given how niche any use of XZ compression could be, we are wondering if there are actually any users of it. If we have no users of hbase-compression-xz, then it provides little to no value and continued maintenance of hbase-compression-xz given the issues with its dependency does not make sense. Do you use XZ compression, or are you planning to? If we deprecate XZ compression immediately and then remove it in 2.6, would this present a problem? In a private discussion we reached consensus on this approach, but, of course, that is not yet a plan, and something that could easily change based on feedback. >From https://nvd.nist.gov/vuln/detail/CVE-2024-3094: "Malicious code was discovered in the upstream tarballs of xz, starting with version 5.6.0. Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code. This results in a modified liblzma library that can be used by any software linked against this library, intercepting and modifying the data interaction with this library." -- Best regards, Andrew
