[
https://issues.apache.org/jira/browse/PARQUET-41?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14351224#comment-14351224
]
Antonios Chalkiopoulos edited comment on PARQUET-41 at 3/7/15 12:38 AM:
------------------------------------------------------------------------
I don't know if it helps - but in scala , using algebird, when i want to
serialize a Bloom Filter into a file and then be able to re-read it from
another job i do:
def serialize(dataStructure: Any): Array[Byte] = {
val stream = new ByteArrayOutputStream()
val out = new ObjectOutputStream(stream)
out.writeObject(dataStructure)
out.close()
stream.close()
stream.toByteArray
}
def deserialize[T](byteArray: Array[Byte]): T = {
val is = new ObjectInputStream(new ByteArrayInputStream(byteArray))
is.readObject().asInstanceOf[T]
}
and then from i.e. Scalding
pipe.mapTo('bloom -> 'serialized) { bf:BF =>
io.scalding.approximations.Utils.serialize(bf) }.write( ... )
pipe.mapTo('bloom -> 'serialized) { bf:BF => new
String(io.scalding.approximations.Utils.serialize(bf)) }.write( ... )
Where BF is actually com.twitter.algebird.BF
was (Author: antwnis):
I don't know if it helps - but in scala , using algebird, when i want to
serialize a Bloom Filter into a file and then be able to re-read it from
another job i do:
def serialize(dataStructure: Any): Array[Byte] = {
val stream = new ByteArrayOutputStream()
val out = new ObjectOutputStream(stream)
out.writeObject(dataStructure)
out.close()
stream.close()
stream.toByteArray
}
def deserialize[T](byteArray: Array[Byte]): T = {
val is = new ObjectInputStream(new ByteArrayInputStream(byteArray))
is.readObject().asInstanceOf[T]
}
and then from i.e. Scalding
pipe.mapTo('bloom -> 'serialized) { bf:BF =>
io.scalding.approximations.Utils.serialize(bf) }.write( ... )
pipe.mapTo('bloom -> 'serialized) { bf:BF => new
String(io.scalding.approximations.Utils.serialize(bf)) }.write( ... )
> Add bloom filters to parquet statistics
> ---------------------------------------
>
> Key: PARQUET-41
> URL: https://issues.apache.org/jira/browse/PARQUET-41
> Project: Parquet
> Issue Type: New Feature
> Components: parquet-mr
> Reporter: Alex Levenson
> Assignee: ferdinand xu
> Labels: filter2
>
> For row groups with no dictionary, we could still produce a bloom filter.
> This could be very useful in filtering entire row groups.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
