I'm not sure what you are looking for but in case that can help you,
We are persisting the data from our kafka cluster in camus and map it in
hive with Camus2Hive, you can look at it here if you want to!
https://github.com/mate1/camus2hive
François Langelier
Étudiant en génie Logiciel - École
Hm, aye, I haven’t tried to write a custom partitioner, and that does look
pretty easy. I’ll put that on my backlog to think about.
The Camus team in the past has been excited to accept patches, and I think if I
Hive partitioner came with Camus it would make it much easier to use. Oh wait,
e.g File produce by the camus job: /user/[hive.user]/output/
*partition_month_utc=2015-03/partition_day_utc=2015-03-11/partition_minute_bucket=2015-03-11-02-09/*
Bhavesh, how do you get Camus to write into a directory hierarchy like this?
Is it reading the partition values from your
Hi Ad
You have to implement custom partitioner and also you will have to create
what ever path (based on message eg log line timestamp, or however you
choose to create directory hierarchy from your each message).
You will need to implement your own Partitioner class implementation:
Hi Andrew,
I would say camus is generic enough (but you can propose this to Camus
Team).
Here is sample code and methods that you can use to create any path or
directory structure and create a corresponding (Hive Table schema for it).
public class UTCLogPartitioner extends Partitioner {
Thanks,
Do you have this partitioner implemented? Perhaps it would be good to try to
get this into Camus as a build in option. HivePartitioner? :)
-Ao
On Mar 11, 2015, at 13:11, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote:
Hi Ad
You have to implement custom partitioner and also
Hive provides the ability to provide custom patterns for partitions. You
can use this in combination with MSCK REPAIR TABLE to automatically detect
and load the partitions into the metastore.
I tried this yesterday, and as far as I can tell it doesn’t work with a custom
partition layout. At
If I understood your question correctly, you want to be able to read the
output of Camus in Hive and be able to know partition values. If my
understanding is right, you can do so by using the following.
Hive provides the ability to provide custom patterns for partitions. You
can use this in
