Hi Kevin. KuduTestHarness, theoretically should detect environment through this plugin "os-maven-plugin" and download the corresponding kudu binaries [1], and it have worked fine for me.
Nonetheless, docker is a good idea. I will give a look to testcontainers and docker. [1] https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing Regards, John El lun., 5 ago. 2019 a las 23:58, Kevin Ratnasekera (< djkevincr1...@gmail.com>) escribió: > Hi John, > > Can't we spin up Kudu docker [1] instance for testing purposes? We have > used Test containers [2] some data stores like couch DB. Gora build should > work in both Linux and Non Linux environments. Eg:- Windows. Is classifier > [3] depend on the environment the build is running? > > Kudu is based on C/ C++, so to spin up a server instance, we need check a > approach like docker, using such approach allow us to avoid these OS, > dependency related stuff come in to play in builds. > > [1] https://hub.docker.com/r/usuresearch/apache-kudu/ > [2] https://www.testcontainers.org/ > [3] <classifier>linux-x86_64</classifier> > > Regards > Kevin > > On Tue, Aug 6, 2019 at 9:56 AM John Mora <jhnmora...@gmail.com> wrote: > > > Hi Alfonso, > > > > Unfortunately, I have not been able to reproduce the issue. Maybe it is > > related with my Java version (Oracle), I will try with OpenJDK. > > Some details about my development environment: > > > > os.detected.name: linux > > os.detected.arch: x86_64 > > os.detected.version: 4.10 > > os.detected.version.major: 4 > > os.detected.version.minor: 10 > > os.detected.release: linuxmint > > os.detected.release.version: 18.3 > > os.detected.release.like.linuxmint: true > > os.detected.release.like.ubuntu: true > > os.detected.classifier: linux-x86_64 > > > > Java > > java version "1.8.0_171" > > Java(TM) SE Runtime Environment (build 1.8.0_171-b11) > > Java HotSpot(TM) 64-Bit Server VM (build 25.171-b11, mixed mode) > > > > Maven > > Apache Maven 3.3.9 > > Maven home: /usr/share/maven > > Java version: 1.8.0_171, vendor: Oracle Corporation > > Java home: /usr/lib/jvm/java-8-oracle/jre > > Default locale: en_US, platform encoding: UTF-8 > > OS name: "linux", version: "4.10.0-38-generic", arch: "amd64", family: > > "unix" > > > > > > Best, > > John. > > > > El lun., 5 ago. 2019 a las 16:48, Alfonso Nishikawa (< > > alfonso.nishik...@gmail.com>) escribió: > > > >> Hi, > >> > >> I am using now the following pom configuration I got from executing `mvn > >> dependency:tree`: > >> > >> <dependency> > >> <groupId>org.apache.kudu</groupId> > >> <artifactId>kudu-binary</artifactId> > >> <classifier>linux-x86_64</classifier> > >> <version>1.9.0</version> > >> <scope>test</scope> > >> </dependency> > >> > >> When I execute `mvn clen package` on gora-kudu I find that it spawns the > >> following command: > >> > >> kudu-master > >> --fs_wal_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/wal > >> --fs_data_dirs=/tmp/mini-kudu-cluster8989984398759938222/master-0/data > >> --block_manager=log --webserver_interface=localhost > --ipki_ca_key_size=1024 > >> --tsk_num_rsa_bits=512 --rpc_bind_addresses=*127.26.116.190*:39535 > >> --webserver_interface=*127.26.116.190* --webserver_port=0 --never_fsync > >> --ipki_server_key_size=1024 --enable_minidumps=false --redact=none > >> --metrics_log_interval_ms=1000 --logtostderr --logbuflevel=-1 > >> --log_dir=/tmp/mini-kudu-cluster8989984398759938222/master-0/logs > >> > --server_dump_info_path=/tmp/mini-kudu-cluster8989984398759938222/master-0/data/info.pb > >> --server_dump_info_format=pb --rpc_server_allow_ephemeral_ports > >> --unlock_experimental_flags --unlock_unsafe_flags --rpc_reuseport=true > >> --master_addresses=*127.26.116.190*:39535,*127.26.116.189*:33913, > >> *127.26.116.188*:42253 > >> > >> > >> I highlight the IP addresses because they clearly are not my computer, > >> and I guess that is why the tests can't connect to the the database. > >> > >> Any idea on how to solve this? > >> > >> Thank you! > >> > >> > >> Best Regards, > >> > >> Alfonso Nishikawa > >> > >> > >> > >> El lun., 5 ago. 2019 a las 8:39, Alfonso Nishikawa (< > >> alfonso.nishik...@gmail.com>) escribió: > >> > >>> Hi, John. > >>> > >>> I get a core dump from the binary kudu server when trying to run the > >>> tests. Didn't find a log file, but will search thoroughly later. > Happened > >>> anytime to you? Does it happens to anyone? > >>> > >>> I am using Ubuntu 18.04 > >>> > >>> Thank you! > >>> > >>> Regards, > >>> > >>> Alfonso Nishikawa > >>> > >>> El dom., 4 ago. 2019 20:10, Furkan KAMACI <furkankam...@gmail.com> > >>> escribió: > >>> > >>>> Hi John, > >>>> > >>>> I've already made my comments at your PR. Please check them carefully > >>>> and ask me if you need help. > >>>> > >>>> For the documentation, I've checked what you've done. On the other > >>>> hand, I would want to encourage you to write a blog post about your > Kudu > >>>> implementation and demonstrate an example of Kudu integration with > Gora as > >>>> like a tutorial. > >>>> > >>>> Kind Regards, > >>>> Furkan KAMACI > >>>> > >>>> On Sun, Aug 4, 2019 at 1:59 AM John Mora <jhnmora...@gmail.com> > wrote: > >>>> > >>>>> Hi all. > >>>>> > >>>>> I have updated my report in the Wiki[1]. > >>>>> > >>>>> Also, I have sent a PR with my last commits for review [2]. Please > >>>>> give it a look if you have time. > >>>>> > >>>>> This week, I will continue working on the documentation of the kudu > >>>>> datastore. > >>>>> > >>>>> Please let me know if you have suggestions. > >>>>> > >>>>> [1] > >>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>> [2] https://github.com/apache/gora/pull/178 > >>>>> > >>>>> Best, > >>>>> John. > >>>>> > >>>>> El mié., 31 jul. 2019 a las 11:17, carlos muñoz (< > carlosr...@gmail.com>) > >>>>> escribió: > >>>>> > >>>>>> Hi John, > >>>>>> > >>>>>> Thanks for the update. I reviewed your code a little bit, it is > >>>>>> looking good. I think tha you should send a PR in order to receive > feedback > >>>>>> from other community members. > >>>>>> > >>>>>> Best, > >>>>>> Carlos > >>>>>> > >>>>>> El dom., 28 jul. 2019 a las 23:20, John Mora (<jhnmora...@gmail.com > >) > >>>>>> escribió: > >>>>>> > >>>>>>> Hi all. > >>>>>>> > >>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last commits > >>>>>>> to my branch [2]. Please give it a look if you have time. > >>>>>>> > >>>>>>> This week, I will give a look to the documentation of datastores. > >>>>>>> > >>>>>>> Please let me know if you have suggestions. > >>>>>>> > >>>>>>> [1] > >>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>> > >>>>>>> Cheers, > >>>>>>> John > >>>>>>> > >>>>>>> El mié., 24 jul. 2019 a las 11:34, John Mora (< > jhnmora...@gmail.com>) > >>>>>>> escribió: > >>>>>>> > >>>>>>>> Hi Alfonso, > >>>>>>>> > >>>>>>>> Yes, I was using this class javafx.util.Pair. It is not a problem > I > >>>>>>>> will find an alternative, it is only an utilitary class. > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> John > >>>>>>>> > >>>>>>>> El mar., 23 jul. 2019 a las 12:36, Alfonso Nishikawa (< > >>>>>>>> alfonso.nishik...@gmail.com>) escribió: > >>>>>>>> > >>>>>>>>> Hi, John. > >>>>>>>>> > >>>>>>>>> I checked out your code and it looks good :) > >>>>>>>>> I found that you use javafx, but that is not present in OpenJDK > >>>>>>>>> and fails to compile, and since we don't stick to Oracle JVM I > would > >>>>>>>>> suggest to change it. > >>>>>>>>> > >>>>>>>>> Good job, keep it going :) > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> > >>>>>>>>> Alfonso Nishikawa > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> El sáb., 20 jul. 2019 a las 22:25, John Mora (< > >>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>> > >>>>>>>>>> Hi. > >>>>>>>>>> > >>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last > >>>>>>>>>> commits to my branch [2]. Please give it a look if you have > time. > >>>>>>>>>> > >>>>>>>>>> This week, I will give a look to the map reduce tests for > >>>>>>>>>> DataStores. > >>>>>>>>>> > >>>>>>>>>> Please let me know if you have suggestions. > >>>>>>>>>> > >>>>>>>>>> [1] > >>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>>>>> > >>>>>>>>>> Thanks, > >>>>>>>>>> John > >>>>>>>>>> > >>>>>>>>>> El sáb., 13 jul. 2019 a las 19:31, John Mora (< > >>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>> > >>>>>>>>>>> Hi all > >>>>>>>>>>> > >>>>>>>>>>> I updated my report in the Wiki[1]. Also, I pushed my last > >>>>>>>>>>> commits to my branch [2]. Please give it a look if you have > time. > >>>>>>>>>>> > >>>>>>>>>>> This week, I will be working in the getPartitions and > >>>>>>>>>>> deleteByQuery methods and testing the other tests in the > DataStoreTestBase > >>>>>>>>>>> class. > >>>>>>>>>>> > >>>>>>>>>>> Please let me know if you have suggestions. > >>>>>>>>>>> > >>>>>>>>>>> [1] > >>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>>>>>> > >>>>>>>>>>> Best, > >>>>>>>>>>> John. > >>>>>>>>>>> > >>>>>>>>>>> El mié., 10 jul. 2019 a las 16:17, John Mora (< > >>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>> > >>>>>>>>>>>> Hi Alfonso, > >>>>>>>>>>>> > >>>>>>>>>>>> Thanks so much for your time and support for this project. I > >>>>>>>>>>>> will work on your comments. Responses inline :) > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> El mar., 9 jul. 2019 a las 16:38, Alfonso Nishikawa (< > >>>>>>>>>>>> alfonso.nishik...@gmail.com>) escribió: > >>>>>>>>>>>> > >>>>>>>>>>>>> Hi, John. > >>>>>>>>>>>>> > >>>>>>>>>>>>> Sorry for the delay, I am changing work and I have been very > >>>>>>>>>>>>> busy :( I will try to answer your questions :) > >>>>>>>>>>>>> > >>>>>>>>>>>>> *> In the Employee example there is a field called > >>>>>>>>>>>>> 'dateOfBirth'. I tried to map that field with the > UNIXTIME_MICROS datatype > >>>>>>>>>>>>> of Kudu (I intuitively assumed this is a date.). However, in > the java world > >>>>>>>>>>>>> the Employee field is a Long value and the kudu datatype is > a Timestamp. > >>>>>>>>>>>>> So, I was wondering whether I should force the usage of the > UNIXTIME_MICROS > >>>>>>>>>>>>> datatype for this field or just use a LONG datatype in Kudu.* > >>>>>>>>>>>>> > >>>>>>>>>>>>> In Avro 1.8 were introduced "Logical Types" so there is a > >>>>>>>>>>>>> "date" type with an underlying "int" [1]. It's the first > time I read about > >>>>>>>>>>>>> because until the last version upgrade of Avro this weren't > there. I would > >>>>>>>>>>>>> suggest to ignore "dates" and map dateOfBirth as long, since > in any case > >>>>>>>>>>>>> -in avro- the value is the unix epoch. After this first > approach, a design > >>>>>>>>>>>>> improvement would be great, though :) > >>>>>>>>>>>>> > >>>>>>>>>>>>> - Would be good to have in the mapping a "timestamp" type so > >>>>>>>>>>>>> KuduStore converts between the Entity long field <-> Kudu > timestamp storage? > >>>>>>>>>>>>> - Is there any other approach? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> I think that Entity long field <-> Kudu timestamp conversion > >>>>>>>>>>>> that the best alternative right now. Because, I would add > more compatible > >>>>>>>>>>>> datatypes to the mapping parameters which users can use. And > this > >>>>>>>>>>>> conversion should not be dificult to implement in my opinion. > Also, the new > >>>>>>>>>>>> Date datatype of avro could be implemented in newer versions > because it > >>>>>>>>>>>> would need further analysis in other datastores too. I will > work on that. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> *> What is the Gora's policy regarding flush()? * > >>>>>>>>>>>>> *> KuduClient has multiple flushing modes > >>>>>>>>>>>>> < > https://kudu.apache.org/apidocs/org/apache/kudu/client/SessionConfiguration.FlushMode.html > >and > >>>>>>>>>>>>> also can set time interval > >>>>>>>>>>>>> < > https://kudu.apache.org/releases/1.2.0/apidocs/org/apache/kudu/client/KuduSession.html#setFlushInterval-int- > > > >>>>>>>>>>>>> for automatic flush.* > >>>>>>>>>>>>> *> Should theses behaviors be configurable using > >>>>>>>>>>>>> gora.properties file? or just use the default > configurations.* > >>>>>>>>>>>>> > >>>>>>>>>>>>> What we do in HBase is configure an autoflush option in > >>>>>>>>>>>>> gora.properties [2] which is used when instanced the Table, > but at the same > >>>>>>>>>>>>> time we implement the flush() method to force the flush [3]. > I would > >>>>>>>>>>>>> suggest to follow that example, but adding the flushing > options of Kudu. > >>>>>>>>>>>>> What flushing mode (and time interval if it applies) do you > suggest? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Well, IMHO the default flush mode (auto flush sync) will do > >>>>>>>>>>>> the job for most use cases. But I will add a configuration in > >>>>>>>>>>>> gora.properties for selecting the other modes and specifying > a autoflush > >>>>>>>>>>>> time if needed by the user. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> *> Also, while reviewing the datastore interface I noticed > >>>>>>>>>>>>> this method 'getPartitions(Query<K, T> query)'. What is the > expected > >>>>>>>>>>>>> behavior of this method?, should I use the partition > definition in the xml > >>>>>>>>>>>>> mapping file for this?.* > >>>>>>>>>>>>> > >>>>>>>>>>>>> The method getPartitions(Query) is related to Hadoop. Apache > >>>>>>>>>>>>> Gora integrates with Hadoop implementing a custom Map and > Reduce that > >>>>>>>>>>>>> allows to get/write Entities directly. > >>>>>>>>>>>>> You can take a look at HBase's implementation [4], which > >>>>>>>>>>>>> relies o.a.h.hbase.mapreduce.TableInputFormatBase [5] to > >>>>>>>>>>>>> compute the splits (start key---end key) with the location > of the split to > >>>>>>>>>>>>> create a colection of partitions [6]. > >>>>>>>>>>>>> > >>>>>>>>>>>>> So, if Kudu is allowed to perform computation using local > kudu > >>>>>>>>>>>>> splits, then this method does the needed preparation to > allow to "send the > >>>>>>>>>>>>> computation to where the data is locally". > >>>>>>>>>>>>> > >>>>>>>>>>>>> In any case, you can see that: > >>>>>>>>>>>>> > >>>>>>>>>>>>> - MongoDB store implementation does not implement > >>>>>>>>>>>>> splitting [7] > >>>>>>>>>>>>> - Cassandra store implementation does not implement > >>>>>>>>>>>>> splitting [8] > >>>>>>>>>>>>> - Aerospike store implementation does not implement > >>>>>>>>>>>>> splitting [9] > >>>>>>>>>>>>> - Accumulo store implementation* does* implement splitting > >>>>>>>>>>>>> [10] > >>>>>>>>>>>>> > >>>>>>>>>>>>> If Kudu has a method to get the different splits for a table > >>>>>>>>>>>>> and its locations, then you will be able to implement the > full feature. > >>>>>>>>>>>>> > >>>>>>>>>>>>> This is Hadoop related and it is not trivial. I haven't > >>>>>>>>>>>>> elaborated much, so if you find you need more information > let me know :) > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> I will check whether Kudu has these features in order to > >>>>>>>>>>>> implement this method. If not I will use the default > implementation found > >>>>>>>>>>>> in other backends. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> About Queries, what I can tell is that Hbase only implements > >>>>>>>>>>>>> "Start key" + "End key" because it has only 2 operations: > "get" and "scan", > >>>>>>>>>>>>> and the querying is for "scan" operation, were you want an > interval (or > >>>>>>>>>>>>> all) of the rows. Does Kudu have more querying functionality? > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> Yes, Kudu implements a Scanner for querying data among with > >>>>>>>>>>>> conditional predicates for filtering. I am using those > classes. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> About other topic, I am trying to install Kudu in standalone > >>>>>>>>>>>>> (all in 1 node). Do you use a Cloudera installation or do > you have a > >>>>>>>>>>>>> standalone installation? How do you do it? I found some > instructions, but > >>>>>>>>>>>>> they talk about compiling Kudu [11]. I was looking for > something like > >>>>>>>>>>>>> HBase, that it is unzip + execute "hbase start". > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>> I am using an embedded mini-cluster which comes with compiled > >>>>>>>>>>>> binaries and can be used with maven[1] for testing my code. > Once I get it > >>>>>>>>>>>> mature enough I think I will be testing the datastore with a > docker > >>>>>>>>>>>> container [2]. I could not find a unzip+execute bundle either > and I am > >>>>>>>>>>>> kinda noob for compiling it myself. > >>>>>>>>>>>> > >>>>>>>>>>>> [1] > >>>>>>>>>>>> > https://kudu.apache.org/docs/developing.html#_jvm_based_integration_testing > >>>>>>>>>>>> [2] https://hub.docker.com/r/usuresearch/apache-kudu/ > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>>> Good job and thank you!! :) > >>>>>>>>>>>>> > >>>>>>>>>>>>> Regards, > >>>>>>>>>>>>> > >>>>>>>>>>>>> Alfonso Nishikawa > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> [1] - > >>>>>>>>>>>>> https://avro.apache.org/docs/1.8.0/spec.html#Logical+Types > >>>>>>>>>>>>> [2] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L175 > >>>>>>>>>>>>> [3] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L458 > >>>>>>>>>>>>> [4] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L472 > >>>>>>>>>>>>> [5] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L479 > >>>>>>>>>>>>> [6] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L517 > >>>>>>>>>>>>> [7] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-mongodb/src/main/java/org/apache/gora/mongodb/store/MongoStore.java#L533 > >>>>>>>>>>>>> [8] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-cassandra/src/main/java/org/apache/gora/cassandra/store/CassandraStore.java#L292 > >>>>>>>>>>>>> [9] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-aerospike/src/main/java/org/apache/gora/aerospike/store/AerospikeStore.java#L369 > >>>>>>>>>>>>> [10] - > >>>>>>>>>>>>> > https://github.com/apache/gora/blob/apache-gora-0.9/gora-accumulo/src/main/java/org/apache/gora/accumulo/store/AccumuloStore.java#L902 > >>>>>>>>>>>>> [11] - https://kudu.apache.org/docs/installation.html > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> El lun., 8 jul. 2019 a las 3:42, John Mora (< > >>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>> > >>>>>>>>>>>>>> Hi all. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> As every week I updated my report in the Wiki[1]. Also, I > >>>>>>>>>>>>>> pushed my last commits to my branch [2]. Please give it a > look if you have > >>>>>>>>>>>>>> time. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> This week, I will be continue working in the Queries > >>>>>>>>>>>>>> implementation, please reach me out if you have any > suggestions. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Also, while reviewing the datastore interface I noticed this > >>>>>>>>>>>>>> method 'getPartitions(Query<K, T> query)'. What is the > expected behavior of > >>>>>>>>>>>>>> this method?, should I use the partition definition in the > xml mapping file > >>>>>>>>>>>>>> for this?. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>> John. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> El dom., 30 jun. 2019 a las 16:56, John Mora (< > >>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hi all. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I received my first evaluation from the Google Summer of > >>>>>>>>>>>>>>> Code program with a positive result. Thanks so much for > your support and > >>>>>>>>>>>>>>> confidence to the project and me. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I updated my report of this week in the Wiki[1]. Also, I > >>>>>>>>>>>>>>> pushed my last commits to my branch [2]. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This week, I will be reviewing my the serialization/ > >>>>>>>>>>>>>>> deserialization process in order to identify optimizations > specific for > >>>>>>>>>>>>>>> Kudu. Because I used a generic methods of other backends > which probably > >>>>>>>>>>>>>>> could be better tuned for kudu. Also, I will start working > on the Queries > >>>>>>>>>>>>>>> implementation. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> BTW, I added a question to the wiki about Date types. > Please > >>>>>>>>>>>>>>> give it a look if you have time. > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>> John > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 21:02, John Mora (< > >>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Hi Carlos. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Thanks for the reminder. I submitted the form yesterday. > :D > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>> John. > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>> El jue., 27 jun. 2019 a las 17:34, carlos muñoz (< > >>>>>>>>>>>>>>>> carlosr...@gmail.com>) escribió: > >>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Hi John > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> The first Google Summer of Code evaluation is due on June > >>>>>>>>>>>>>>>>> 28th. Please make sure you submit your Mentors' > evaluation on time. > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>> Carlos > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>> El dom., 23 jun. 2019 a las 18:29, John Mora (< > >>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Hi all. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> FYI, I updated my report of this week on the Wiki[1]. > >>>>>>>>>>>>>>>>>> Also, I pushed my last commits to my branch [2]. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> As I mentioned in the reports I would like to know how > >>>>>>>>>>>>>>>>>> datastores deal with flush(), should it work always > manually executed?. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Finally, This week I will be implementing object > >>>>>>>>>>>>>>>>>> serialization/deserialization in the methods put, get, > delete, exists. Do > >>>>>>>>>>>>>>>>>> you have any suggestions on how to proceed with this > task?. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Footnote: Thanks for the feedback Carlos, I fixed the > >>>>>>>>>>>>>>>>>> problem. > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>>>>>>>>> [2] https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>> John > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>> El lun., 17 jun. 2019 a las 22:58, carlos muñoz (< > >>>>>>>>>>>>>>>>>> carlosr...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Hi John > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Your last changes look good to me. Keep it up. But, I > >>>>>>>>>>>>>>>>>>> noticed that you have created an Enumeration for > datatypes, which is very > >>>>>>>>>>>>>>>>>>> similar to the kudu-client's [2]. Probably you should > replace [1] for [2] > >>>>>>>>>>>>>>>>>>> in order to avoid code duplication. > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/Column.java#L76 > >>>>>>>>>>>>>>>>>>> [2] > >>>>>>>>>>>>>>>>>>> > https://kudu.apache.org/apidocs/org/apache/kudu/Type.html > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>> Carlos > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>> El sáb., 15 jun. 2019 a las 12:01, John Mora (< > >>>>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Hi all. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> I updated my report of this week on the Wiki[1]. I > >>>>>>>>>>>>>>>>>>>> noticed that my code is lacking some javadoc > documentation I think I will > >>>>>>>>>>>>>>>>>>>> be working on that this week, also I would like to > enable and check schema > >>>>>>>>>>>>>>>>>>>> management tests (createSchema, existsSchema, etc.). > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> Cheers, > >>>>>>>>>>>>>>>>>>>> John. > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>> El mar., 11 jun. 2019 a las 0:11, John Mora (< > >>>>>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Hi Alfonso. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Thanks so much for your feedback. I am working on > your > >>>>>>>>>>>>>>>>>>>>> comments. > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>> John > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 16:11, Alfonso Nishikawa > (< > >>>>>>>>>>>>>>>>>>>>> alfonso.nishik...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Hi, John. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Regarding your questions at the report [1]: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> - How to represent partitioning configurations on > >>>>>>>>>>>>>>>>>>>>>> the mapping file. > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> This was discussed in other emails, isn't it? :) > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> - KuduTestHarness requires the Maven plugin > >>>>>>>>>>>>>>>>>>>>>> os-maven-plugin, which needs Maven 3.1.1+, is it > a problem for Apache Gora? > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> I believe it is not a problem. My Ubuntu comes with > >>>>>>>>>>>>>>>>>>>>>> 3.6.0, far from 3.1.1, and I assume everyone uses > Maven 3 in a quite new > >>>>>>>>>>>>>>>>>>>>>> version :) > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> [1] - > >>>>>>>>>>>>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>> El lun., 10 jun. 2019 a las 21:07, Alfonso Nishikawa > >>>>>>>>>>>>>>>>>>>>>> (<alfonso.nishik...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Hi, John. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Thank you! > >>>>>>>>>>>>>>>>>>>>>>> Things I have seen: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> - The version of a maven dependency [1] should go > on > >>>>>>>>>>>>>>>>>>>>>>> the Dependency Management of the root pom [2]. > Same for [3] and from there, > >>>>>>>>>>>>>>>>>>>>>>> should not set the version there. > >>>>>>>>>>>>>>>>>>>>>>> - Set test dependencies' scope to test, at [4] and > >>>>>>>>>>>>>>>>>>>>>>> from there. > >>>>>>>>>>>>>>>>>>>>>>> - Set the indentation to 2 spaces for the pom [5] > >>>>>>>>>>>>>>>>>>>>>>> - Missing "t" in "localhost" at [6]. > >>>>>>>>>>>>>>>>>>>>>>> - Port 13 for Kudu? That is "Daytime Protocol" RFC > >>>>>>>>>>>>>>>>>>>>>>> 867 and you will need root permission to run it. > The default port for kudu > >>>>>>>>>>>>>>>>>>>>>>> is 7051, isn't it? > >>>>>>>>>>>>>>>>>>>>>>> - I would ask you to add the same functionality to > >>>>>>>>>>>>>>>>>>>>>>> load the mapping from configuration as in HBase's > store [7] in you > >>>>>>>>>>>>>>>>>>>>>>> KuduStore [8]. This will have implications on your > readMapping at [9], so > >>>>>>>>>>>>>>>>>>>>>>> take a look at the one for HBase at [10] > >>>>>>>>>>>>>>>>>>>>>>> - I know it is in other backends, but avoid > >>>>>>>>>>>>>>>>>>>>>>> RuntimeExceptions (at least in Java since we have > the checked ones) like in > >>>>>>>>>>>>>>>>>>>>>>> [11]. You can wrap them in GoraException. An > example is [12] > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> And nothing more :) > >>>>>>>>>>>>>>>>>>>>>>> Keep going, good job. > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> [1] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L98 > >>>>>>>>>>>>>>>>>>>>>>> [2] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/pom.xml#L890 > >>>>>>>>>>>>>>>>>>>>>>> [3] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L121 > >>>>>>>>>>>>>>>>>>>>>>> [4] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml#L180 > >>>>>>>>>>>>>>>>>>>>>>> [5] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/pom.xml > >>>>>>>>>>>>>>>>>>>>>>> [6] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/test/resources/gora.properties#L18 > >>>>>>>>>>>>>>>>>>>>>>> [7] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L92 > >>>>>>>>>>>>>>>>>>>>>>> [8] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/store/KuduStore.java#L53 > >>>>>>>>>>>>>>>>>>>>>>> [9] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L81 > >>>>>>>>>>>>>>>>>>>>>>> [10] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L822 > >>>>>>>>>>>>>>>>>>>>>>> [11] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/GORA-485/gora-kudu/src/main/java/org/apache/gora/kudu/mapping/KuduMappingBuilder.java#L141 > >>>>>>>>>>>>>>>>>>>>>>> [12] - > >>>>>>>>>>>>>>>>>>>>>>> > https://github.com/jhnmora000/gora/blob/master/gora-hbase/src/main/java/org/apache/gora/hbase/store/HBaseStore.java#L268 > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Regards, > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> Alfonso Nishikawa > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>> El sáb., 8 jun. 2019 a las 20:26, John Mora (< > >>>>>>>>>>>>>>>>>>>>>>> jhnmora...@gmail.com>) escribió: > >>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Hi all. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> I have just updated my weekly reports on Cwiki > [1]. > >>>>>>>>>>>>>>>>>>>>>>>> This next week I think I should be focusing on > the create schema operation > >>>>>>>>>>>>>>>>>>>>>>>> and solving the issue of the partitioning > configurations in the mapping > >>>>>>>>>>>>>>>>>>>>>>>> file. > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Please let me know if you have suggestions, my > last > >>>>>>>>>>>>>>>>>>>>>>>> commits are available here [2] > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> [1] > >>>>>>>>>>>>>>>>>>>>>>>> > https://cwiki.apache.org/confluence/display/GORA/GORA-485+Apache+Kudu+datastore+for+Gora+Reports > >>>>>>>>>>>>>>>>>>>>>>>> [2] > >>>>>>>>>>>>>>>>>>>>>>>> https://github.com/jhnmora000/gora/tree/GORA-485 > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> Best, > >>>>>>>>>>>>>>>>>>>>>>>> John > >>>>>>>>>>>>>>>>>>>>>>>> > >>>>>>>>>>>>>>>>>>>>>>>> >
