Re: How to get the creation time of a HTable?
We have method to identify which tables will not receive read/write requests; But to be safe, we will exclude the tables that newly created (as they might be under creating), say in latest 2 days; If Hbase doesn't have such metadata, it's okay that we add that during the table creation; But if it can be implemented in the future HBase release, that is good for the users (especially the admins) I think; Thanks for the answers and comments! 2015-08-14 20:51 GMT+08:00 Ted Yu yuzhih...@gmail.com: After predetermined number of days, would your table(s) not receive any read / write requests ? Have you considered using TTL for cleaning old data ? Cheers On Aug 14, 2015, at 2:54 AM, ShaoFeng Shi shaofeng...@gmail.com wrote: Hello the community, In my case, I want to cleanup the HTables that older than certain days; so we need to get the table's creation time, is there any API to get this? If not, I may have to add such an attribute when creating the table; Thanks for any suggestion; Shaofeng Shi, Apache Kylin
Re: Replicating MySQL table to HBase
We got a bunch of datasets in our CDH cluster and want to do some data analysis on those which requires some meta data only available in MySQL. We could do a one time Sqoop and then want to setup a job to capture the changes and write to HBase. I'm looking for options to handle the MySQL changes, thanks! On Fri, Aug 14, 2015 at 5:00 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi, Before even going into that direction, why do you want to do that? It's most probably not a good idea. Is is for backup? For replication? etc. JM 2015-08-14 19:56 GMT-04:00 Buntu Dev buntu...@gmail.com: I'm looking for ways to setup an incremental update task to replicate the MySQL table to HBase. Sqoop seems like an option to import but doesn't seem to handle row updates and deletes. Hoping there might be other better ways to handle the inserts/updates/deletes. Thanks!
Re: Replicating MySQL table to HBase
Hi, Before even going into that direction, why do you want to do that? It's most probably not a good idea. Is is for backup? For replication? etc. JM 2015-08-14 19:56 GMT-04:00 Buntu Dev buntu...@gmail.com: I'm looking for ways to setup an incremental update task to replicate the MySQL table to HBase. Sqoop seems like an option to import but doesn't seem to handle row updates and deletes. Hoping there might be other better ways to handle the inserts/updates/deletes. Thanks!
Replicating MySQL table to HBase
I'm looking for ways to setup an incremental update task to replicate the MySQL table to HBase. Sqoop seems like an option to import but doesn't seem to handle row updates and deletes. Hoping there might be other better ways to handle the inserts/updates/deletes. Thanks!
Re: Replicating MySQL table to HBase
But why do you want that in HBase? Do you need random read and random write milliseconds access to those rows? Or you are going to run some analytics jobs which are going to do full scans anyway? If the later, why not just dumping that as files and doing the queries with a SQL engine? If you need HBase for user, then you might have to build a tool to migrate the data from MySQL to HBase because the MySQL schema will most probably not fit the HBase tables... 2015-08-14 20:12 GMT-04:00 Buntu Dev buntu...@gmail.com: We got a bunch of datasets in our CDH cluster and want to do some data analysis on those which requires some meta data only available in MySQL. We could do a one time Sqoop and then want to setup a job to capture the changes and write to HBase. I'm looking for options to handle the MySQL changes, thanks! On Fri, Aug 14, 2015 at 5:00 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi, Before even going into that direction, why do you want to do that? It's most probably not a good idea. Is is for backup? For replication? etc. JM 2015-08-14 19:56 GMT-04:00 Buntu Dev buntu...@gmail.com: I'm looking for ways to setup an incremental update task to replicate the MySQL table to HBase. Sqoop seems like an option to import but doesn't seem to handle row updates and deletes. Hoping there might be other better ways to handle the inserts/updates/deletes. Thanks!
Re: Replicating MySQL table to HBase
Let me clarify that our analysis involves looking at the changes over a period of time and HBase provides helps us with time based lookups and state of the things at particular point of time. On Fri, Aug 14, 2015 at 5:16 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: But why do you want that in HBase? Do you need random read and random write milliseconds access to those rows? Or you are going to run some analytics jobs which are going to do full scans anyway? If the later, why not just dumping that as files and doing the queries with a SQL engine? If you need HBase for user, then you might have to build a tool to migrate the data from MySQL to HBase because the MySQL schema will most probably not fit the HBase tables... 2015-08-14 20:12 GMT-04:00 Buntu Dev buntu...@gmail.com: We got a bunch of datasets in our CDH cluster and want to do some data analysis on those which requires some meta data only available in MySQL. We could do a one time Sqoop and then want to setup a job to capture the changes and write to HBase. I'm looking for options to handle the MySQL changes, thanks! On Fri, Aug 14, 2015 at 5:00 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi, Before even going into that direction, why do you want to do that? It's most probably not a good idea. Is is for backup? For replication? etc. JM 2015-08-14 19:56 GMT-04:00 Buntu Dev buntu...@gmail.com: I'm looking for ways to setup an incremental update task to replicate the MySQL table to HBase. Sqoop seems like an option to import but doesn't seem to handle row updates and deletes. Hoping there might be other better ways to handle the inserts/updates/deletes. Thanks!
RE: How to get the creation time of a HTable?
I think we do not have any such attribute for the table but we can implement one like we have for Snapshot. Regards, Ashish Singhi -Original Message- From: Serega Sheypak [mailto:serega.shey...@gmail.com] Sent: 14 August 2015 15:28 To: user Subject: Re: How to get the creation time of a HTable? Hm... you can check underlying table catalog on HDFS and see time properties there? 2015-08-14 11:54 GMT+02:00 ShaoFeng Shi shaofeng...@gmail.com: Hello the community, In my case, I want to cleanup the HTables that older than certain days; so we need to get the table's creation time, is there any API to get this? If not, I may have to add such an attribute when creating the table; Thanks for any suggestion; Shaofeng Shi, Apache Kylin
Re: How to get the creation time of a HTable?
After predetermined number of days, would your table(s) not receive any read / write requests ? Have you considered using TTL for cleaning old data ? Cheers On Aug 14, 2015, at 2:54 AM, ShaoFeng Shi shaofeng...@gmail.com wrote: Hello the community, In my case, I want to cleanup the HTables that older than certain days; so we need to get the table's creation time, is there any API to get this? If not, I may have to add such an attribute when creating the table; Thanks for any suggestion; Shaofeng Shi, Apache Kylin
How to get the creation time of a HTable?
Hello the community, In my case, I want to cleanup the HTables that older than certain days; so we need to get the table's creation time, is there any API to get this? If not, I may have to add such an attribute when creating the table; Thanks for any suggestion; Shaofeng Shi, Apache Kylin
Re: How to get the creation time of a HTable?
Hm... you can check underlying table catalog on HDFS and see time properties there? 2015-08-14 11:54 GMT+02:00 ShaoFeng Shi shaofeng...@gmail.com: Hello the community, In my case, I want to cleanup the HTables that older than certain days; so we need to get the table's creation time, is there any API to get this? If not, I may have to add such an attribute when creating the table; Thanks for any suggestion; Shaofeng Shi, Apache Kylin
