Hi John, Can't we spin up Kudu docker [1] instance for testing purposes? We have used Test containers [2] some data stores like couch DB. Gora build should work in both Linux and Non Linux environments. Eg:- Windows. Is classifier [3] depend on the environment the build is running?
Kudu is based on C/ C++, so to spin up a server instance, we need check a approach like docker, using such approach allow us to avoid these OS, dependency related stuff come in to play in builds. [1] https://hub.docker.com/r/usuresearch/apache-kudu/ [2] https://www.testcontainers.org/ [3] <classifier>linux-x86_64</classifier> Regards Kevin On Tue, Aug 6, 2019 at 9:56 AM John Mora <jhnmora...@gmail.com> wrote: > Hi Alfonso, > > Unfortunately, I have not been able to reproduce the issue. Maybe it is > related with my Java version (Oracle), I will try with OpenJDK. > Some details about my development environment: > > os.detected.name: linux > os.detected.arch: x86_64 > os.detected.version: 4.10 > os.detected.version.major: 4 > os.detected.version.minor: 10 > os.detected.release: linuxmint > os.detected.release.version: 18.3 > os.detected.release.like.linuxmint: true > os.detected.release.like.ubuntu: true > os.detected.classifier: linux-x86_64 > > Java > java version "1.8.0_171" > Java(TM) SE Runtime Environment (build 1.8.0_171-b11) > Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) > > Maven > Apache Maven 3.3.9 > Maven home: /usr/share/maven > Java version: 1.8.0_171, vendor: Oracle Corporation > Java home: /usr/lib/jvm/java-8-oracle/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family: > "unix" > > > Best, > John. > > El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (< > alfonso.nishik...@gmail.com>) escribió: > >> Hi, >> >> I am using now the following pom configuration I got from executing `mvn >> dependency:tree`: >> >> <dependency> >> <groupId>org.apache.kudu</groupId> >> <artifactId>kudu-binary</artifactId> >> <classifier>linux-x86_64</classifier> >> <version>1.9.0</version> >> <scope>test</scope> >> </dependency> >> >> When I execute `mvn clen package` on gora-kudu I find that it spawns the >> following command: >> >> kudu-master >> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal >> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data >> --block_manager=log --webserver_interface=localhost --ipki_ca_key_size=1024 >> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535 >> --webserver_interface=*127.26.116.190* --webserver_port=0 --never_fsync >> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none >> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 >> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs >> --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb >> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports >> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true >> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913, >> *127.26.116.188*:42253 >> >> >> I highlight the IP addresses because they clearly are not my computer, >> and I guess that is why the tests can't connect to the the database. >> >> Any idea on how to solve this? >> >> Thank you! >> >> >> Best Regards, >> >> Alfonso Nishikawa >> >> >> >> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (< >> alfonso.nishik...@gmail.com>) escribió: >> >>> Hi, John. >>> >>> I get a core dump from the binary kudu server when trying to run the >>> tests. Didn't find a log file, but will search thoroughly later. Happened >>> anytime to you? Does it happens to anyone? >>> >>> I am using Ubuntu 18.04 >>> >>> Thank you! >>> >>> Regards, >>> >>> Alfonso Nishikawa >>> >>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <furkankam...@gmail.com> >>> escribió: >>> >>>> Hi John, >>>> >>>> I've already made my comments at your PR. Please check them carefully >>>> and ask me if you need help. >>>> >>>> For the documentation, I've checked what you've done. On the other >>>> hand, I would want to encourage you to write a blog post about your Kudu >>>> implementation and demonstrate an example of Kudu integration with Gora as >>>> like a tutorial. >>>> >>>> Kind Regards, >>>> Furkan KAMACI >>>> >>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jhnmora...@gmail.com> wrote: >>>> >>>>> Hi all. >>>>> >>>>> I have updated my report in the Wiki[1]. >>>>> >>>>> Also, I have sent a PR with my last commits for review [2]. Please >>>>> give it a look if you have time. >>>>> >>>>> This week, I will continue working on the documentation of the kudu >>>>> datastore. >>>>> >>>>> Please let me know if you have suggestions. >>>>> >>>>> [1] >>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>> [2] https://github.com/apache/gora/pull/178 >>>>> >>>>> Best, >>>>> John. >>>>> >>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (<carlosr...@gmail.com>) >>>>> escribió: >>>>> >>>>>> Hi John, >>>>>> >>>>>> Thanks for the update. I reviewed your code a little bit, it is >>>>>> looking good. I think tha you should send a PR in order to receive >>>>>> feedback >>>>>> from other community members. >>>>>> >>>>>> Best, >>>>>> Carlos >>>>>> >>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jhnmora...@gmail.com>) >>>>>> escribió: >>>>>> >>>>>>> Hi all. >>>>>>> >>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits >>>>>>> to my branch [2]. Please give it a look if you have time. >>>>>>> >>>>>>> This week, I will give a look to the documentation of datastores. >>>>>>> >>>>>>> Please let me know if you have suggestions. >>>>>>> >>>>>>> [1] >>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>> >>>>>>> Cheers, >>>>>>> John >>>>>>> >>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (<jhnmora...@gmail.com>) >>>>>>> escribió: >>>>>>> >>>>>>>> Hi Alfonso, >>>>>>>> >>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem I >>>>>>>> will find an alternative, it is only an utilitary class. >>>>>>>> >>>>>>>> Thanks, >>>>>>>> John >>>>>>>> >>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (< >>>>>>>> alfonso.nishik...@gmail.com>) escribió: >>>>>>>> >>>>>>>>> Hi, John. >>>>>>>>> >>>>>>>>> I checked out your code and it looks good :) >>>>>>>>> I found that you use javafx, but that is not present in OpenJDK >>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I would >>>>>>>>> suggest to change it. >>>>>>>>> >>>>>>>>> Good job, keep it going :) >>>>>>>>> >>>>>>>>> Regards, >>>>>>>>> >>>>>>>>> Alfonso Nishikawa >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (< >>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>> >>>>>>>>>> Hi. >>>>>>>>>> >>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last >>>>>>>>>> commits to my branch [2]. Please give it a look if you have time. >>>>>>>>>> >>>>>>>>>> This week, I will give a look to the map reduce tests for >>>>>>>>>> DataStores. >>>>>>>>>> >>>>>>>>>> Please let me know if you have suggestions. >>>>>>>>>> >>>>>>>>>> [1] >>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>>>>> >>>>>>>>>> Thanks, >>>>>>>>>> John >>>>>>>>>> >>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (< >>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>> >>>>>>>>>>> Hi all >>>>>>>>>>> >>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last >>>>>>>>>>> commits to my branch [2]. Please give it a look if you have time. >>>>>>>>>>> >>>>>>>>>>> This week, I will be working in the getPartitions and >>>>>>>>>>> deleteByQuery methods and testing the other tests in the >>>>>>>>>>> DataStoreTestBase >>>>>>>>>>> class. >>>>>>>>>>> >>>>>>>>>>> Please let me know if you have suggestions. >>>>>>>>>>> >>>>>>>>>>> [1] >>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> John. >>>>>>>>>>> >>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (< >>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>> >>>>>>>>>>>> Hi Alfonso, >>>>>>>>>>>> >>>>>>>>>>>> Thanks so much for your time and support for this project. I >>>>>>>>>>>> will work on your comments. Responses inline :) >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (< >>>>>>>>>>>> alfonso.nishik...@gmail.com>) escribió: >>>>>>>>>>>> >>>>>>>>>>>>> Hi, John. >>>>>>>>>>>>> >>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very >>>>>>>>>>>>> busy :( I will try to answer your questions :) >>>>>>>>>>>>> >>>>>>>>>>>>> *> In the Employee example there is a field called >>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the UNIXTIME_MICROS >>>>>>>>>>>>> datatype >>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in the >>>>>>>>>>>>> java world >>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is a >>>>>>>>>>>>> Timestamp. >>>>>>>>>>>>> So, I was wondering whether I should force the usage of the >>>>>>>>>>>>> UNIXTIME_MICROS >>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.* >>>>>>>>>>>>> >>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a >>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first time I >>>>>>>>>>>>> read about >>>>>>>>>>>>> because until the last version upgrade of Avro this weren't >>>>>>>>>>>>> there. I would >>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since in >>>>>>>>>>>>> any case >>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first approach, >>>>>>>>>>>>> a design >>>>>>>>>>>>> improvement would be great, though :) >>>>>>>>>>>>> >>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so >>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu >>>>>>>>>>>>> timestamp storage? >>>>>>>>>>>>> - Is there any other approach? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion >>>>>>>>>>>> that the best alternative right now. Because, I would add more >>>>>>>>>>>> compatible >>>>>>>>>>>> datatypes to the mapping parameters which users can use. And this >>>>>>>>>>>> conversion should not be dificult to implement in my opinion. >>>>>>>>>>>> Also, the new >>>>>>>>>>>> Date datatype of avro could be implemented in newer versions >>>>>>>>>>>> because it >>>>>>>>>>>> would need further analysis in other datastores too. I will work >>>>>>>>>>>> on that. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? * >>>>>>>>>>>>> *> KuduClient has multiple flushing modes >>>>>>>>>>>>> <https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html>and >>>>>>>>>>>>> also can set time interval >>>>>>>>>>>>> <https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int-> >>>>>>>>>>>>> for automatic flush.* >>>>>>>>>>>>> *> Should theses behaviors be configurable using >>>>>>>>>>>>> gora.properties file? or just use the default configurations.* >>>>>>>>>>>>> >>>>>>>>>>>>> What we do in HBase is configure an autoflush option in >>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, but >>>>>>>>>>>>> at the same >>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. I >>>>>>>>>>>>> would >>>>>>>>>>>>> suggest to follow that example, but adding the flushing options >>>>>>>>>>>>> of Kudu. >>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you >>>>>>>>>>>>> suggest? >>>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do >>>>>>>>>>>> the job for most use cases. But I will add a configuration in >>>>>>>>>>>> gora.properties for selecting the other modes and specifying a >>>>>>>>>>>> autoflush >>>>>>>>>>>> time if needed by the user. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed >>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the >>>>>>>>>>>>> expected >>>>>>>>>>>>> behavior of this method?, should I use the partition definition >>>>>>>>>>>>> in the xml >>>>>>>>>>>>> mapping file for this?.* >>>>>>>>>>>>> >>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache >>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and Reduce >>>>>>>>>>>>> that >>>>>>>>>>>>> allows to get/write Entities directly. >>>>>>>>>>>>> You can take a look at HBase's implementation [4], which >>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to >>>>>>>>>>>>> compute the splits (start key---end key) with the location of the >>>>>>>>>>>>> split to >>>>>>>>>>>>> create a colection of partitions [6]. >>>>>>>>>>>>> >>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local kudu >>>>>>>>>>>>> splits, then this method does the needed preparation to allow to >>>>>>>>>>>>> "send the >>>>>>>>>>>>> computation to where the data is locally". >>>>>>>>>>>>> >>>>>>>>>>>>> In any case, you can see that: >>>>>>>>>>>>> >>>>>>>>>>>>> - MongoDB store implementation does not implement >>>>>>>>>>>>> splitting [7] >>>>>>>>>>>>> - Cassandra store implementation does not implement >>>>>>>>>>>>> splitting [8] >>>>>>>>>>>>> - Aerospike store implementation does not implement >>>>>>>>>>>>> splitting [9] >>>>>>>>>>>>> - Accumulo store implementation* does* implement splitting >>>>>>>>>>>>> [10] >>>>>>>>>>>>> >>>>>>>>>>>>> If Kudu has a method to get the different splits for a table >>>>>>>>>>>>> and its locations, then you will be able to implement the full >>>>>>>>>>>>> feature. >>>>>>>>>>>>> >>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't >>>>>>>>>>>>> elaborated much, so if you find you need more information let me >>>>>>>>>>>>> know :) >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> I will check whether Kudu has these features in order to >>>>>>>>>>>> implement this method. If not I will use the default >>>>>>>>>>>> implementation found >>>>>>>>>>>> in other backends. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements >>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: "get" >>>>>>>>>>>>> and "scan", >>>>>>>>>>>>> and the querying is for "scan" operation, were you want an >>>>>>>>>>>>> interval (or >>>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality? >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with >>>>>>>>>>>> conditional predicates for filtering. I am using those classes. >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone >>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do you >>>>>>>>>>>>> have a >>>>>>>>>>>>> standalone installation? How do you do it? I found some >>>>>>>>>>>>> instructions, but >>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for something >>>>>>>>>>>>> like >>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start". >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled >>>>>>>>>>>> binaries and can be used with maven[1] for testing my code. Once I >>>>>>>>>>>> get it >>>>>>>>>>>> mature enough I think I will be testing the datastore with a docker >>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either and >>>>>>>>>>>> I am >>>>>>>>>>>> kinda noob for compiling it myself. >>>>>>>>>>>> >>>>>>>>>>>> [1] >>>>>>>>>>>> https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing >>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/ >>>>>>>>>>>> >>>>>>>>>>>> >>>>>>>>>>>>> Good job and thank you!! :) >>>>>>>>>>>>> >>>>>>>>>>>>> Regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Alfonso Nishikawa >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> [1] - >>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types >>>>>>>>>>>>> [2] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175 >>>>>>>>>>>>> [3] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458 >>>>>>>>>>>>> [4] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472 >>>>>>>>>>>>> [5] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479 >>>>>>>>>>>>> [6] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517 >>>>>>>>>>>>> [7] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533 >>>>>>>>>>>>> [8] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292 >>>>>>>>>>>>> [9] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369 >>>>>>>>>>>>> [10] - >>>>>>>>>>>>> https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902 >>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html >>>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (< >>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>> >>>>>>>>>>>>>> Hi all. >>>>>>>>>>>>>> >>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I >>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a look >>>>>>>>>>>>>> if you have >>>>>>>>>>>>>> time. >>>>>>>>>>>>>> >>>>>>>>>>>>>> This week, I will be continue working in the Queries >>>>>>>>>>>>>> implementation, please reach me out if you have any suggestions. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this >>>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the expected >>>>>>>>>>>>>> behavior of >>>>>>>>>>>>>> this method?, should I use the partition definition in the xml >>>>>>>>>>>>>> mapping file >>>>>>>>>>>>>> for this?. >>>>>>>>>>>>>> >>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>> John. >>>>>>>>>>>>>> >>>>>>>>>>>>>> [1] >>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>>>>>>>>> >>>>>>>>>>>>>> >>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (< >>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>>> >>>>>>>>>>>>>>> Hi all. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of >>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for your >>>>>>>>>>>>>>> support and >>>>>>>>>>>>>>> confidence to the project and me. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I >>>>>>>>>>>>>>> pushed my last commits to my branch [2]. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/ >>>>>>>>>>>>>>> deserialization process in order to identify optimizations >>>>>>>>>>>>>>> specific for >>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends which >>>>>>>>>>>>>>> probably >>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working on >>>>>>>>>>>>>>> the Queries >>>>>>>>>>>>>>> implementation. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types. Please >>>>>>>>>>>>>>> give it a look if you have time. >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>> John >>>>>>>>>>>>>>> >>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (< >>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Hi Carlos. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. :D >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>> John. >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (< >>>>>>>>>>>>>>>> carlosr...@gmail.com>) escribió: >>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Hi John >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June >>>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' evaluation on >>>>>>>>>>>>>>>>> time. >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>> Carlos >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (< >>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Hi all. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. >>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2]. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how >>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always manually >>>>>>>>>>>>>>>>>> executed?. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object >>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, >>>>>>>>>>>>>>>>>> delete, exists. Do >>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this task?. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the >>>>>>>>>>>>>>>>>> problem. >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>> John >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (< >>>>>>>>>>>>>>>>>> carlosr...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Hi John >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I >>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for datatypes, >>>>>>>>>>>>>>>>>>> which is very >>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should >>>>>>>>>>>>>>>>>>> replace [1] for [2] >>>>>>>>>>>>>>>>>>> in order to avoid code duplication. >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76 >>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>> https://kudu.apache.org/apidocs/org/apache/kudu/Type.html >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>> Carlos >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (< >>>>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Hi all. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I >>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc documentation >>>>>>>>>>>>>>>>>>>> I think I will >>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to enable >>>>>>>>>>>>>>>>>>>> and check schema >>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.). >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> Cheers, >>>>>>>>>>>>>>>>>>>> John. >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (< >>>>>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Hi Alfonso. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on your >>>>>>>>>>>>>>>>>>>>> comments. >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>> John >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa (< >>>>>>>>>>>>>>>>>>>>> alfonso.nishik...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Hi, John. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on >>>>>>>>>>>>>>>>>>>>>> the mapping file. >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin >>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it a >>>>>>>>>>>>>>>>>>>>>> problem for Apache Gora? >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with >>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses Maven >>>>>>>>>>>>>>>>>>>>>> 3 in a quite new >>>>>>>>>>>>>>>>>>>>>> version :) >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa >>>>>>>>>>>>>>>>>>>>>> (<alfonso.nishik...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Hi, John. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Thank you! >>>>>>>>>>>>>>>>>>>>>>> Things I have seen: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go on >>>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2]. Same for >>>>>>>>>>>>>>>>>>>>>>> [3] and from there, >>>>>>>>>>>>>>>>>>>>>>> should not set the version there. >>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and >>>>>>>>>>>>>>>>>>>>>>> from there. >>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5] >>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6]. >>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC >>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it. The >>>>>>>>>>>>>>>>>>>>>>> default port for kudu >>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it? >>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to >>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's store >>>>>>>>>>>>>>>>>>>>>>> [7] in you >>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your >>>>>>>>>>>>>>>>>>>>>>> readMapping at [9], so >>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10] >>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid >>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have the >>>>>>>>>>>>>>>>>>>>>>> checked ones) like in >>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An example is >>>>>>>>>>>>>>>>>>>>>>> [12] >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> And nothing more :) >>>>>>>>>>>>>>>>>>>>>>> Keep going, good job. >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> [1] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98 >>>>>>>>>>>>>>>>>>>>>>> [2] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890 >>>>>>>>>>>>>>>>>>>>>>> [3] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121 >>>>>>>>>>>>>>>>>>>>>>> [4] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180 >>>>>>>>>>>>>>>>>>>>>>> [5] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml >>>>>>>>>>>>>>>>>>>>>>> [6] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18 >>>>>>>>>>>>>>>>>>>>>>> [7] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92 >>>>>>>>>>>>>>>>>>>>>>> [8] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53 >>>>>>>>>>>>>>>>>>>>>>> [9] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81 >>>>>>>>>>>>>>>>>>>>>>> [10] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822 >>>>>>>>>>>>>>>>>>>>>>> [11] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141 >>>>>>>>>>>>>>>>>>>>>>> [12] - >>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268 >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Regards, >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (< >>>>>>>>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: >>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Hi all. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki [1]. >>>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on the >>>>>>>>>>>>>>>>>>>>>>>> create schema operation >>>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning >>>>>>>>>>>>>>>>>>>>>>>> configurations in the mapping >>>>>>>>>>>>>>>>>>>>>>>> file. >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my last >>>>>>>>>>>>>>>>>>>>>>>> commits are available here [2] >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>> https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports >>>>>>>>>>>>>>>>>>>>>>>> [2] >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485 >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>> Best, >>>>>>>>>>>>>>>>>>>>>>>> John >>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>
