[jira] [Commented] (DRILL-7017) lz4 codec for (un)compression
[
https://issues.apache.org/jira/browse/DRILL-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16957093#comment-16957093
]
benj commented on DRILL-7017:
-
Not sure to understand because lz4 is already (by default) in
jars/3rdparty/lz4-1.3.0.jar in Apache Drill and it doesn't work.
Even with adding "org.apache.hadoop.io.compress.Lz4Codec" in
io.compression.codecs in core-site.xml and
Djava.library.path=/usr/hdp/.../lib/native/
{code:sql}
SELECT * FROM dfs.test.`a.csvh.lz4`;
Error: EXECUTION_ERROR ERROR: native lz4 library not available
{code}
> lz4 codec for (un)compression
> -
>
> Key: DRILL-7017
> URL: https://issues.apache.org/jira/browse/DRILL-7017
> Project: Apache Drill
> Issue Type: Wish
> Components: Storage - Text CSV
>Affects Versions: 1.15.0
>Reporter: benj
>Priority: Major
>
> I didn't find in the documentation what compression formats are supported.
> But as it's possible to use drill on compressed file, like
> {code:java}
> SELECT * FROM tmp.`myfile.csv.gz`;
> {code}
> It will be useful to have the possibility to use this functionality for lz4
> file ([https://github.com/lz4/lz4])
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7017) lz4 codec for (un)compression
[
https://issues.apache.org/jira/browse/DRILL-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16954676#comment-16954676
]
Arina Ielchiieva commented on DRILL-7017:
-
To support lz4 native lz4 lib should be accessible (i.e. included in Drill
classpath). Similar issues was discussed in other project -
https://issues.apache.org/jira/browse/KYLIN-3201.
> lz4 codec for (un)compression
> -
>
> Key: DRILL-7017
> URL: https://issues.apache.org/jira/browse/DRILL-7017
> Project: Apache Drill
> Issue Type: Wish
> Components: Storage - Text CSV
>Affects Versions: 1.15.0
>Reporter: benj
>Priority: Major
>
> I didn't find in the documentation what compression formats are supported.
> But as it's possible to use drill on compressed file, like
> {code:java}
> SELECT * FROM tmp.`myfile.csv.gz`;
> {code}
> It will be useful to have the possibility to use this functionality for lz4
> file ([https://github.com/lz4/lz4])
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (DRILL-7017) lz4 codec for (un)compression
[
https://issues.apache.org/jira/browse/DRILL-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16757258#comment-16757258
]
benj commented on DRILL-7017:
-
lz4 is well in the list of supported compression.
Searched a lot to try to work with .csv.lz4 but don't find any way.
I encounter problems with hadoop & Lz4 due to HADOOP-12990. But I don't know if
this problem can really prevent Drill from working with .csv.lz4 files
> lz4 codec for (un)compression
> -
>
> Key: DRILL-7017
> URL: https://issues.apache.org/jira/browse/DRILL-7017
> Project: Apache Drill
> Issue Type: Wish
> Components: Storage - Text CSV
>Affects Versions: 1.15.0
>Reporter: benj
>Priority: Major
>
> I didn't find in the documentation what compression formats are supported.
> But as it's possible to use drill on compressed file, like
> {code:java}
> SELECT * FROM tmp.`myfile.csv.gz`;
> {code}
> It will be useful to have the possibility to use this functionality for lz4
> file ([https://github.com/lz4/lz4])
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (DRILL-7017) lz4 codec for (un)compression
[
https://issues.apache.org/jira/browse/DRILL-7017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756328#comment-16756328
]
Volodymyr Vysotskyi commented on DRILL-7017:
Please see examples of usage of querying compressed files here:
[https://drill.apache.org/docs/querying-plain-text-files/#querying-compressed-files].
{{hadoop-common}} already contains lz4 compressors/decompressors. So it is
possible that this compression is already supported. Not sure about the full
list of codecs, but I think it may be extended by specifying additional codecs
in {{io.compression.codecs}} in hadoop conf file (perhaps in {{core-site.xml}}).
> lz4 codec for (un)compression
> -
>
> Key: DRILL-7017
> URL: https://issues.apache.org/jira/browse/DRILL-7017
> Project: Apache Drill
> Issue Type: Wish
> Components: Storage - Text CSV
>Affects Versions: 1.15.0
>Reporter: benj
>Priority: Major
>
> I didn't find in the documentation what compression formats are supported.
> But as it's possible to use drill on compressed file, like
> {code:java}
> SELECT * FROM tmp.`myfile.csv.gz`;
> {code}
> It will be useful to have the possibility to use this functionality for lz4
> file ([https://github.com/lz4/lz4)]
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
