I see that the API has changed a bit so my old code doesn’t work anymore. Can someone direct me to some code samples?
Thanks, Ben > On Sep 20, 2016, at 1:44 PM, Todd Lipcon <t...@cloudera.com> wrote: > > On Tue, Sep 20, 2016 at 1:18 PM, Benjamin Kim <bbuil...@gmail.com > <mailto:bbuil...@gmail.com>> wrote: > Now that Kudu 1.0.0 is officially out and ready for production use, where do > we find the spark connector jar for this release? > > > It's available in the official ASF maven repository: > https://repository.apache.org/#nexus-search;quick~kudu-spark > <https://repository.apache.org/#nexus-search;quick~kudu-spark> > > <dependency> > <groupId>org.apache.kudu</groupId> > <artifactId>kudu-spark_2.10</artifactId> > <version>1.0.0</version> > </dependency> > > > -Todd > > > >> On Jun 17, 2016, at 11:08 AM, Dan Burkert <d...@cloudera.com >> <mailto:d...@cloudera.com>> wrote: >> >> Hi Ben, >> >> To your first question about `CREATE TABLE` syntax with Kudu/Spark SQL, I do >> not think we support that at this point. I haven't looked deeply into it, >> but we may hit issues specifying Kudu-specific options (partitioning, column >> encoding, etc.). Probably issues that can be worked through eventually, >> though. If you are interested in contributing to Kudu, this is an area that >> could obviously use improvement! Most or all of our Spark features have >> been completely community driven to date. >> >> I am assuming that more Spark support along with semantic changes below will >> be incorporated into Kudu 0.9.1. >> >> As a rule we do not release new features in patch releases, but the good >> news is that we are releasing regularly, and our next scheduled release is >> for the August timeframe (see JD's roadmap >> <https://lists.apache.org/thread.html/1a3b949e715a74d7f26bd9c102247441a06d16d077324ba39a662e2a@1455234076@%3Cdev.kudu.apache.org%3E> >> email about what we are aiming to include). Also, Cloudera does publish >> snapshot versions of the Spark connector here >> <https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/>, so >> the jars are available if you don't mind using snapshots. >> >> Anyone know of a better way to make unique primary keys other than using >> UUID to make every row unique if there is no unique column (or combination >> thereof) to use. >> >> Not that I know of. In general it's pretty rare to have a dataset without a >> natural primary key (even if it's just all of the columns), but in those >> cases UUID is a good solution. >> >> This is what I am using. I know auto incrementing is coming down the line >> (don’t know when), but is there a way to simulate this in Kudu using Spark >> out of curiosity? >> >> To my knowledge there is no plan to have auto increment in Kudu. >> Distributed, consistent, auto incrementing counters is a difficult problem, >> and I don't think there are any known solutions that would be fast enough >> for Kudu (happy to be proven wrong, though!). >> >> - Dan >> >> >> Thanks, >> Ben >> >>> On Jun 14, 2016, at 6:08 PM, Dan Burkert <d...@cloudera.com >>> <mailto:d...@cloudera.com>> wrote: >>> >>> I'm not sure exactly what the semantics will be, but at least one of them >>> will be upsert. These modes come from spark, and they were really designed >>> for file-backed storage and not table storage. We may want to do append = >>> upsert, and overwrite = truncate + insert. I think that may match the >>> normal spark semantics more closely. >>> >>> - Dan >>> >>> On Tue, Jun 14, 2016 at 6:00 PM, Benjamin Kim <bbuil...@gmail.com >>> <mailto:bbuil...@gmail.com>> wrote: >>> Dan, >>> >>> Thanks for the information. That would mean both “append” and “overwrite” >>> modes would be combined or not needed in the future. >>> >>> Cheers, >>> Ben >>> >>>> On Jun 14, 2016, at 5:57 PM, Dan Burkert <d...@cloudera.com >>>> <mailto:d...@cloudera.com>> wrote: >>>> >>>> Right now append uses an update Kudu operation, which requires the row >>>> already be present in the table. Overwrite maps to insert. Kudu very >>>> recently got upsert support baked in, but it hasn't yet been integrated >>>> into the Spark connector. So pretty soon these sharp edges will get a lot >>>> better, since upsert is the way to go for most spark workloads. >>>> >>>> - Dan >>>> >>>> On Tue, Jun 14, 2016 at 5:41 PM, Benjamin Kim <bbuil...@gmail.com >>>> <mailto:bbuil...@gmail.com>> wrote: >>>> I tried to use the “append” mode, and it worked. Over 3.8 million rows in >>>> 64s. I would assume that now I can use the “overwrite” mode on existing >>>> data. Now, I have to find answers to these questions. What would happen if >>>> I “append” to the data in the Kudu table if the data already exists? What >>>> would happen if I “overwrite” existing data when the DataFrame has data in >>>> it that does not exist in the Kudu table? I need to evaluate the best way >>>> to simulate the UPSERT behavior in HBase because this is what our use case >>>> is. >>>> >>>> Thanks, >>>> Ben >>>> >>>> >>>> >>>>> On Jun 14, 2016, at 5:05 PM, Benjamin Kim <bbuil...@gmail.com >>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>> >>>>> Hi, >>>>> >>>>> Now, I’m getting this error when trying to write to the table. >>>>> >>>>> import scala.collection.JavaConverters._ >>>>> val key_seq = Seq(“my_id") >>>>> val key_list = List(“my_id”).asJava >>>>> kuduContext.createTable(tableName, df.schema, key_seq, new >>>>> CreateTableOptions().setNumReplicas(1).addHashPartitions(key_list, 100)) >>>>> >>>>> df.write >>>>> .options(Map("kudu.master" -> kuduMaster,"kudu.table" -> tableName)) >>>>> .mode("overwrite") >>>>> .kudu >>>>> >>>>> java.lang.RuntimeException: failed to write 1000 rows from DataFrame to >>>>> Kudu; sample errors: Not found: key not found (error 0)Not found: key not >>>>> found (error 0)Not found: key not found (error 0)Not found: key not found >>>>> (error 0)Not found: key not found (error 0) >>>>> >>>>> Does the key field need to be first in the DataFrame? >>>>> >>>>> Thanks, >>>>> Ben >>>>> >>>>>> On Jun 14, 2016, at 4:28 PM, Dan Burkert <d...@cloudera.com >>>>>> <mailto:d...@cloudera.com>> wrote: >>>>>> >>>>>> >>>>>> >>>>>> On Tue, Jun 14, 2016 at 4:20 PM, Benjamin Kim <bbuil...@gmail.com >>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>> Dan, >>>>>> >>>>>> Thanks! It got further. Now, how do I set the Primary Key to be a >>>>>> column(s) in the DataFrame and set the partitioning? Is it like this? >>>>>> >>>>>> kuduContext.createTable(tableName, df.schema, Seq(“my_id"), new >>>>>> CreateTableOptions().setNumReplicas(1).addHashPartitions(“my_id")) >>>>>> >>>>>> java.lang.IllegalArgumentException: Table partitioning must be specified >>>>>> using setRangePartitionColumns or addHashPartitions >>>>>> >>>>>> Yep. The `Seq("my_id")` part of that call is specifying the set of >>>>>> primary key columns, so in this case you have specified the single PK >>>>>> column "my_id". The `addHashPartitions` call adds hash partitioning to >>>>>> the table, in this case over the column "my_id" (which is good, it must >>>>>> be over one or more PK columns, so in this case "my_id" is the one and >>>>>> only valid combination). However, the call to `addHashPartition` also >>>>>> takes the number of buckets as the second param. You shouldn't get the >>>>>> IllegalArgumentException as long as you are specifying either >>>>>> `addHashPartitions` or `setRangePartitionColumns`. >>>>>> >>>>>> - Dan >>>>>> >>>>>> >>>>>> Thanks, >>>>>> Ben >>>>>> >>>>>> >>>>>>> On Jun 14, 2016, at 4:07 PM, Dan Burkert <d...@cloudera.com >>>>>>> <mailto:d...@cloudera.com>> wrote: >>>>>>> >>>>>>> Looks like we're missing an import statement in that example. Could >>>>>>> you try: >>>>>>> >>>>>>> import org.kududb.client._ >>>>>>> and try again? >>>>>>> >>>>>>> - Dan >>>>>>> >>>>>>> On Tue, Jun 14, 2016 at 4:01 PM, Benjamin Kim <bbuil...@gmail.com >>>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>>> I encountered an error trying to create a table based on the >>>>>>> documentation from a DataFrame. >>>>>>> >>>>>>> <console>:49: error: not found: type CreateTableOptions >>>>>>> kuduContext.createTable(tableName, df.schema, Seq("key"), >>>>>>> new CreateTableOptions().setNumReplicas(1)) >>>>>>> >>>>>>> Is there something I’m missing? >>>>>>> >>>>>>> Thanks, >>>>>>> Ben >>>>>>> >>>>>>>> On Jun 14, 2016, at 3:00 PM, Jean-Daniel Cryans <jdcry...@apache.org >>>>>>>> <mailto:jdcry...@apache.org>> wrote: >>>>>>>> >>>>>>>> It's only in Cloudera's maven repo: >>>>>>>> https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/kudu-spark_2.10/0.9.0/ >>>>>>>> >>>>>>>> <https://repository.cloudera.com/cloudera/cloudera-repos/org/kududb/kudu-spark_2.10/0.9.0/> >>>>>>>> >>>>>>>> J-D >>>>>>>> >>>>>>>> On Tue, Jun 14, 2016 at 2:59 PM, Benjamin Kim <bbuil...@gmail.com >>>>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>>>> Hi J-D, >>>>>>>> >>>>>>>> I installed Kudu 0.9.0 using CM, but I can’t find the kudu-spark jar >>>>>>>> for spark-shell to use. Can you show me where to find it? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Ben >>>>>>>> >>>>>>>> >>>>>>>>> On Jun 8, 2016, at 1:19 PM, Jean-Daniel Cryans <jdcry...@apache.org >>>>>>>>> <mailto:jdcry...@apache.org>> wrote: >>>>>>>>> >>>>>>>>> What's in this doc is what's gonna get released: >>>>>>>>> https://github.com/cloudera/kudu/blob/master/docs/developing.adoc#kudu-integration-with-spark >>>>>>>>> >>>>>>>>> <https://github.com/cloudera/kudu/blob/master/docs/developing.adoc#kudu-integration-with-spark> >>>>>>>>> >>>>>>>>> J-D >>>>>>>>> >>>>>>>>> On Tue, Jun 7, 2016 at 8:52 PM, Benjamin Kim <bbuil...@gmail.com >>>>>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>> Will this be documented with examples once 0.9.0 comes out? >>>>>>>>> >>>>>>>>> Thanks, >>>>>>>>> Ben >>>>>>>>> >>>>>>>>> >>>>>>>>>> On May 28, 2016, at 3:22 PM, Jean-Daniel Cryans <jdcry...@apache.org >>>>>>>>>> <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>> >>>>>>>>>> It will be in 0.9.0. >>>>>>>>>> >>>>>>>>>> J-D >>>>>>>>>> >>>>>>>>>> On Sat, May 28, 2016 at 8:31 AM, Benjamin Kim <bbuil...@gmail.com >>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>> Hi Chris, >>>>>>>>>> >>>>>>>>>> Will all this effort be rolled into 0.9.0 and be ready for use? >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> Ben >>>>>>>>>> >>>>>>>>>> >>>>>>>>>>> On May 18, 2016, at 9:01 AM, Chris George >>>>>>>>>>> <christopher.geo...@rms.com <mailto:christopher.geo...@rms.com>> >>>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> There is some code in review that needs some more refinement. >>>>>>>>>>> It will allow upsert/insert from a dataframe using the datasource >>>>>>>>>>> api. It will also allow the creation and deletion of tables from a >>>>>>>>>>> dataframe >>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2992/ >>>>>>>>>>> <http://gerrit.cloudera.org:8080/#/c/2992/> >>>>>>>>>>> >>>>>>>>>>> Example usages will look something like: >>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2992/5/docs/developing.adoc >>>>>>>>>>> <http://gerrit.cloudera.org:8080/#/c/2992/5/docs/developing.adoc> >>>>>>>>>>> >>>>>>>>>>> -Chris George >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On 5/18/16, 9:45 AM, "Benjamin Kim" <bbuil...@gmail.com >>>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>>> >>>>>>>>>>> Can someone tell me what the state is of this Spark work? >>>>>>>>>>> >>>>>>>>>>> Also, does anyone have any sample code on how to update/insert data >>>>>>>>>>> in Kudu using DataFrames? >>>>>>>>>>> >>>>>>>>>>> Thanks, >>>>>>>>>>> Ben >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>>> On Apr 13, 2016, at 8:22 AM, Chris George >>>>>>>>>>>> <christopher.geo...@rms.com <mailto:christopher.geo...@rms.com>> >>>>>>>>>>>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> SparkSQL cannot support these type of statements but we may be >>>>>>>>>>>> able to implement similar functionality through the api. >>>>>>>>>>>> -Chris >>>>>>>>>>>> >>>>>>>>>>>> On 4/12/16, 5:19 PM, "Benjamin Kim" <bbuil...@gmail.com >>>>>>>>>>>> <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>>>> >>>>>>>>>>>> It would be nice to adhere to the SQL:2003 standard for an >>>>>>>>>>>> “upsert” if it were to be implemented. >>>>>>>>>>>> >>>>>>>>>>>> MERGE INTO table_name USING table_reference ON (condition) >>>>>>>>>>>> WHEN MATCHED THEN >>>>>>>>>>>> UPDATE SET column1 = value1 [, column2 = value2 ...] >>>>>>>>>>>> WHEN NOT MATCHED THEN >>>>>>>>>>>> INSERT (column1 [, column2 ...]) VALUES (value1 [, value2 …]) >>>>>>>>>>>> >>>>>>>>>>>> Cheers, >>>>>>>>>>>> Ben >>>>>>>>>>>> >>>>>>>>>>>>> On Apr 11, 2016, at 12:21 PM, Chris George >>>>>>>>>>>>> <christopher.geo...@rms.com <mailto:christopher.geo...@rms.com>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> I have a wip kuduRDD that I made a few months ago. I pushed it >>>>>>>>>>>>> into gerrit if you want to take a look. >>>>>>>>>>>>> http://gerrit.cloudera.org:8080/#/c/2754/ >>>>>>>>>>>>> <http://gerrit.cloudera.org:8080/#/c/2754/> >>>>>>>>>>>>> It does pushdown predicates which the existing input formatter >>>>>>>>>>>>> based rdd does not. >>>>>>>>>>>>> >>>>>>>>>>>>> Within the next two weeks I’m planning to implement a datasource >>>>>>>>>>>>> for spark that will have pushdown predicates and insertion/update >>>>>>>>>>>>> functionality (need to look more at cassandra and the hbase >>>>>>>>>>>>> datasource for best way to do this) I agree that server side >>>>>>>>>>>>> upsert would be helpful. >>>>>>>>>>>>> Having a datasource would give us useful data frames and also >>>>>>>>>>>>> make spark sql usable for kudu. >>>>>>>>>>>>> >>>>>>>>>>>>> My reasoning for having a spark datasource and not using Impala >>>>>>>>>>>>> is: 1. We have had trouble getting impala to run fast with high >>>>>>>>>>>>> concurrency when compared to spark 2. We interact with >>>>>>>>>>>>> datasources which do not integrate with impala. 3. We have custom >>>>>>>>>>>>> sql query planners for extended sql functionality. >>>>>>>>>>>>> >>>>>>>>>>>>> -Chris George >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On 4/11/16, 12:22 PM, "Jean-Daniel Cryans" <jdcry...@apache.org >>>>>>>>>>>>> <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>> >>>>>>>>>>>>> You guys make a convincing point, although on the upsert side >>>>>>>>>>>>> we'll need more support from the servers. Right now all you can >>>>>>>>>>>>> do is an INSERT then, if you get a dup key, do an UPDATE. I guess >>>>>>>>>>>>> we could at least add an API on the client side that would manage >>>>>>>>>>>>> it, but it wouldn't be atomic. >>>>>>>>>>>>> >>>>>>>>>>>>> J-D >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Apr 11, 2016 at 9:34 AM, Mark Hamstra >>>>>>>>>>>>> <m...@clearstorydata.com <mailto:m...@clearstorydata.com>>wrote: >>>>>>>>>>>>> It's pretty simple, actually. I need to support versioned >>>>>>>>>>>>> datasets in a Spark SQL environment. Instead of a hack on top of >>>>>>>>>>>>> a Parquet data store, I'm hoping (among other reasons) to be able >>>>>>>>>>>>> to use Kudu's write and timestamp-based read operations to >>>>>>>>>>>>> support not only appending data, but also updating existing data, >>>>>>>>>>>>> and even some schema migration. The most typical use case is a >>>>>>>>>>>>> dataset that is updated periodically (e.g., weekly or monthly) in >>>>>>>>>>>>> which the the preliminary data in the previous window (week or >>>>>>>>>>>>> month) is updated with values that are expected to remain >>>>>>>>>>>>> unchanged from then on, and a new set of preliminary values for >>>>>>>>>>>>> the current window need to be added/appended. >>>>>>>>>>>>> >>>>>>>>>>>>> Using Kudu's Java API and developing additional functionality on >>>>>>>>>>>>> top of what Kudu has to offer isn't too much to ask, but the ease >>>>>>>>>>>>> of integration with Spark SQL will gate how quickly we would move >>>>>>>>>>>>> to using Kudu and how seriously we'd look at alternatives before >>>>>>>>>>>>> making that decision. >>>>>>>>>>>>> >>>>>>>>>>>>> On Mon, Apr 11, 2016 at 8:14 AM, Jean-Daniel Cryans >>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>>wrote: >>>>>>>>>>>>> Mark, >>>>>>>>>>>>> >>>>>>>>>>>>> Thanks for taking some time to reply in this thread, glad it >>>>>>>>>>>>> caught the attention of other folks! >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:33 PM, Mark >>>>>>>>>>>>> Hamstra<m...@clearstorydata.com <mailto:m...@clearstorydata.com>> >>>>>>>>>>>>> wrote: >>>>>>>>>>>>> Do they care being able to insert into Kudu with SparkSQL >>>>>>>>>>>>> >>>>>>>>>>>>> I care about insert into Kudu with Spark SQL. I'm currently >>>>>>>>>>>>> delaying a refactoring of some Spark SQL-oriented insert >>>>>>>>>>>>> functionality while trying to evaluate what to expect from Kudu. >>>>>>>>>>>>> Whether Kudu does a good job supporting inserts with Spark SQL >>>>>>>>>>>>> will be a key consideration as to whether we adopt Kudu. >>>>>>>>>>>>> >>>>>>>>>>>>> I'd like to know more about why SparkSQL inserts in necessary for >>>>>>>>>>>>> you. Is it just that you currently do it that way into some >>>>>>>>>>>>> database or parquet so with minimal refactoring you'd be able to >>>>>>>>>>>>> use Kudu? Would re-writing those SQL lines into Scala and >>>>>>>>>>>>> directly use the Java API's KuduSession be too much work? >>>>>>>>>>>>> >>>>>>>>>>>>> Additionally, what do you expect to gain from using Kudu VS your >>>>>>>>>>>>> current solution? If it's not completely clear, I'd love to help >>>>>>>>>>>>> you think through it. >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:23 PM, Jean-Daniel Cryans >>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>> Yup, starting to get a good idea. >>>>>>>>>>>>> >>>>>>>>>>>>> What are your DS folks looking for in terms of functionality >>>>>>>>>>>>> related to Spark? A SparkSQL integration that's as fully featured >>>>>>>>>>>>> as Impala's? Do they care being able to insert into Kudu with >>>>>>>>>>>>> SparkSQL or just being able to query real fast? Anything more >>>>>>>>>>>>> specific to Spark that I'm missing? >>>>>>>>>>>>> >>>>>>>>>>>>> FWIW the plan is to get to 1.0 in late Summer/early Fall. At >>>>>>>>>>>>> Cloudera all our resources are committed to making things happen >>>>>>>>>>>>> in time, and a more fully featured Spark integration isn't in our >>>>>>>>>>>>> plans during that period. I'm really hoping someone in the >>>>>>>>>>>>> community will help with Spark, the same way we got a big >>>>>>>>>>>>> contribution for the Flume sink. >>>>>>>>>>>>> >>>>>>>>>>>>> J-D >>>>>>>>>>>>> >>>>>>>>>>>>> On Sun, Apr 10, 2016 at 11:29 AM, Benjamin Kim >>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>>wrote: >>>>>>>>>>>>> Yes, we took Kudu for a test run using 0.6 and 0.7 versions. But, >>>>>>>>>>>>> since it’s not “production-ready”, upper management doesn’t want >>>>>>>>>>>>> to fully deploy it yet. They just want to keep an eye on it >>>>>>>>>>>>> though. Kudu was so much simpler and easier to use in every >>>>>>>>>>>>> aspect compared to HBase. Impala was great for the report writers >>>>>>>>>>>>> and analysts to experiment with for the short time it was up. >>>>>>>>>>>>> But, once again, the only blocker was the lack of Spark support >>>>>>>>>>>>> for our Data Developers/Scientists. So, production-level data >>>>>>>>>>>>> population won’t happen until then. >>>>>>>>>>>>> >>>>>>>>>>>>> I hope this helps you get an idea where I am coming from… >>>>>>>>>>>>> >>>>>>>>>>>>> Cheers, >>>>>>>>>>>>> Ben >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>>> On Apr 10, 2016, at 11:08 AM, Jean-Daniel Cryans >>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>>> >>>>>>>>>>>>>> On Sun, Apr 10, 2016 at 12:30 AM, Benjamin Kim >>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>>>>>> J-D, >>>>>>>>>>>>>> >>>>>>>>>>>>>> The main thing I hear that Cassandra is being used as an >>>>>>>>>>>>>> updatable hot data store to ensure that duplicates are taken >>>>>>>>>>>>>> care of and idempotency is maintained. Whether data was directly >>>>>>>>>>>>>> retrieved from Cassandra for analytics, reports, or searches, it >>>>>>>>>>>>>> was not clear as to what was its main use. Some also just used >>>>>>>>>>>>>> it for a staging area to populate downstream tables in parquet >>>>>>>>>>>>>> format. The last thing I heard was that CQL was terrible, so >>>>>>>>>>>>>> that rules out much use of direct queries against it. >>>>>>>>>>>>>> >>>>>>>>>>>>>> I'm no C* expert, but I don't think CQL is meant for real >>>>>>>>>>>>>> analytics, just ease of use instead of plainly using the APIs. >>>>>>>>>>>>>> Even then, Kudu should beat it easily on big scans. Same for >>>>>>>>>>>>>> HBase. We've done benchmarks against the latter, not the former. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> As for our company, we have been looking for an updatable data >>>>>>>>>>>>>> store for a long time that can be quickly queried directly >>>>>>>>>>>>>> either using Spark SQL or Impala or some other SQL engine and >>>>>>>>>>>>>> still handle TB or PB of data without performance degradation >>>>>>>>>>>>>> and many configuration headaches. For now, we are using HBase to >>>>>>>>>>>>>> take on this role with Phoenix as a fast way to directly query >>>>>>>>>>>>>> the data. I can see Kudu as the best way to fill this gap >>>>>>>>>>>>>> easily, especially being the closest thing to other relational >>>>>>>>>>>>>> databases out there in familiarity for the many SQL analytics >>>>>>>>>>>>>> people in our company. The other alternative would be to go with >>>>>>>>>>>>>> AWS Redshift for the same reasons, but it would come at a cost, >>>>>>>>>>>>>> of course. If we went with either solutions, Kudu or Redshift, >>>>>>>>>>>>>> it would get rid of the need to extract from HBase to parquet >>>>>>>>>>>>>> tables or export to PostgreSQL to support more of the SQL >>>>>>>>>>>>>> language using by analysts or the reporting software we use.. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Ok, the usual then *smile*. Looks like we're not too far off >>>>>>>>>>>>>> with Kudu. Have you folks tried Kudu with Impala yet with those >>>>>>>>>>>>>> use cases? >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> I hope this helps. >>>>>>>>>>>>>> >>>>>>>>>>>>>> It does, thanks for nice reply. >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> Ben >>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Apr 9, 2016, at 2:00 PM, Jean-Daniel Cryans >>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Ha first time I'm hearing about SMACK. Inside Cloudera we like >>>>>>>>>>>>>>> to refer to "Impala + Kudu" as Kimpala, but yeah it's not as >>>>>>>>>>>>>>> sexy. My colleagues who were also there did say that the hype >>>>>>>>>>>>>>> around Spark isn't dying down. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> There's definitely an overlap in the use cases that Cassandra, >>>>>>>>>>>>>>> HBase, and Kudu cater to. I wouldn't go as far as saying that >>>>>>>>>>>>>>> C* is just an interim solution for the use case you describe. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Nothing significant happened in Kudu over the past month, it's >>>>>>>>>>>>>>> a storage engine so things move slowly *smile*. I'd love to see >>>>>>>>>>>>>>> more contributions on the Spark front. I know there's code out >>>>>>>>>>>>>>> there that could be integrated in kudu-spark, it just needs to >>>>>>>>>>>>>>> land in gerrit. I'm sure folks will happily review it. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Do you have relevant experiences you can share? I'd love to >>>>>>>>>>>>>>> learn more about the use cases for which you envision using >>>>>>>>>>>>>>> Kudu as a C* replacement. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> On Fri, Apr 8, 2016 at 12:45 PM, Benjamin Kim >>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>>>>>>> Hi J-D, >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> My colleagues recently came back from Strata in San Jose. They >>>>>>>>>>>>>>> told me that everything was about Spark and there is a big buzz >>>>>>>>>>>>>>> about the SMACK stack (Spark, Mesos, Akka, Cassandra, Kafka). I >>>>>>>>>>>>>>> still think that Cassandra is just an interim solution as a >>>>>>>>>>>>>>> low-latency, easily queried data store. I was wondering if >>>>>>>>>>>>>>> anything significant happened in regards to Kudu, especially on >>>>>>>>>>>>>>> the Spark front. Plus, can you come up with your own proposed >>>>>>>>>>>>>>> stack acronym to promote? >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Mar 1, 2016, at 12:20 PM, Jean-Daniel Cryans >>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Ben, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> AFAIK no one in the dev community committed to any timeline. I >>>>>>>>>>>>>>>> know of one person on the Kudu Slack who's working on a better >>>>>>>>>>>>>>>> RDD, but that's about it. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> On Tue, Mar 1, 2016 at 11:00 AM, Benjamin Kim <b...@amobee.com >>>>>>>>>>>>>>>> <mailto:b...@amobee.com>> wrote: >>>>>>>>>>>>>>>> Hi J-D, >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Quick question… Is there an ETA for KUDU-1214? I want to >>>>>>>>>>>>>>>> target a version of Kudu to begin real testing of Spark >>>>>>>>>>>>>>>> against it for our devs. At least, I can tell them what >>>>>>>>>>>>>>>> timeframe to anticipate. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Just curious, >>>>>>>>>>>>>>>> Benjamin Kim >>>>>>>>>>>>>>>> Data Solutions Architect >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> [a•mo•bee] (n.) the company defining digital marketing. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Mobile: +1 818 635 2900 <tel:%2B1%20818%20635%202900> >>>>>>>>>>>>>>>> 3250 Ocean Park Blvd, Suite 200 | Santa Monica, CA 90405 | >>>>>>>>>>>>>>>> www.amobee.com <http://www.amobee.com/> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Feb 24, 2016, at 3:51 PM, Jean-Daniel Cryans >>>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The DStream stuff isn't there at all. I'm not sure if it's >>>>>>>>>>>>>>>>> needed either. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The kuduRDD is just leveraging the MR input format, ideally >>>>>>>>>>>>>>>>> we'd use scans directly. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The SparkSQL stuff is there but it doesn't do any sort of >>>>>>>>>>>>>>>>> pushdown. It's really basic. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The goal was to provide something for others to contribute >>>>>>>>>>>>>>>>> to. We have some basic unit tests that others can easily >>>>>>>>>>>>>>>>> extend. None of us on the team are Spark experts, but we'd be >>>>>>>>>>>>>>>>> really happy to assist one improve the kudu-spark code. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> On Wed, Feb 24, 2016 at 3:41 PM, Benjamin Kim >>>>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>>>>>>>>> J-D, >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> It looks like it fulfills most of the basic requirements >>>>>>>>>>>>>>>>> (kudu RDD, kudu DStream) in KUDU-1214. Am I right? Besides >>>>>>>>>>>>>>>>> shoring up more Spark SQL functionality (Dataframes) and >>>>>>>>>>>>>>>>> doing the documentation, what more needs to be done? >>>>>>>>>>>>>>>>> Optimizations? >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> I believe that it’s a good place to start using Spark with >>>>>>>>>>>>>>>>> Kudu and compare it to HBase with Spark (not clean). >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Feb 24, 2016, at 3:10 PM, Jean-Daniel Cryans >>>>>>>>>>>>>>>>>> <jdcry...@apache.org <mailto:jdcry...@apache.org>> wrote: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> AFAIK no one is working on it, but we did manage to get this >>>>>>>>>>>>>>>>>> in for 0.7.0: https://issues.cloudera.org/browse/KUDU-1321 >>>>>>>>>>>>>>>>>> <https://issues.cloudera.org/browse/KUDU-1321> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> It's a really simple wrapper, and yes you can use SparkSQL >>>>>>>>>>>>>>>>>> on Kudu, but it will require a lot more work to make it >>>>>>>>>>>>>>>>>> fast/useful. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hope this helps, >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> J-D >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> On Wed, Feb 24, 2016 at 3:08 PM, Benjamin Kim >>>>>>>>>>>>>>>>>> <bbuil...@gmail.com <mailto:bbuil...@gmail.com>> wrote: >>>>>>>>>>>>>>>>>> I see this KUDU-1214 >>>>>>>>>>>>>>>>>> <https://issues.cloudera.org/browse/KUDU-1214> targeted for >>>>>>>>>>>>>>>>>> 0.8.0, but I see no progress on it. When this is complete, >>>>>>>>>>>>>>>>>> will this mean that Spark will be able to work with Kudu >>>>>>>>>>>>>>>>>> both programmatically and as a client via Spark SQL? Or is >>>>>>>>>>>>>>>>>> there more work that needs to be done on the Spark side for >>>>>>>>>>>>>>>>>> it to work? >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Just curious. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>> Ben >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>> >>> >> >> > > > > > -- > Todd Lipcon > Software Engineer, Cloudera
