GitHub user b-slim opened a pull request:
https://github.com/apache/hive/pull/425
Kafkahandler hive 20377
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/b-slim/hive kafkahandler_HIVE-20377
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/hive/pull/425.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #425
----
commit 3996b5b2e86f7b3cd4ca319227163852e268aec6
Author: Slim Bouguerra <bslim@...>
Date: 2018-07-19T16:30:48Z
Working version of kafka storage handler
Change-Id: Ief161074d151917c3a7ed443cf78374bcaf7bcfc
trying random things
fix typo
working version for demo only change might be getRecordreader call
adding kafka Trimmer first cut
Change-Id: If9bf7f561b867e80ab31f5c8c3c40730128986af
link the code and add some docs
Change-Id: I7e15d90de772fffef8ee0930352069742af12ac7
add static function to avoid dep on hive exec
Change-Id: Ib61901cd45027d1469d72a890e26f73997402974
working version of filter pushdown partition and offsets
Change-Id: I3e3d157438fcc965491380e3f1afa9c81a6cd75b
fix case when offset filter is out of range and add some code comments
Change-Id: Ic3321a14cc9a06b8eeb85cc20ea89f21e2765a93
add filter push down based on ts
Change-Id: I5a1da7634cfc80990036add3463a28810ce642c2
clean code
Change-Id: I753cfe4e9f0a69396635b7a9fe16c29f7bb0675e
case filter is null can imply no filter
Change-Id: I68fa9afe09009329ebe8ac9e5d3f7923a2003ebb
some refactor for how to build consumer
Change-Id: If007a89d93c215542027aec7ea0aff2fa6dc83fb
major refactor:
removed and/or methods
switch logic of ANDing filters
Change-Id: Id36f45842db7c4579edaa1e6062bddd291d53bc9
adding tests and refactor fetch by time
Change-Id: I6329ca3fa8c6f29e90d0001034064d2a603d2580
adding more tests
Change-Id: Ic9d1f1a86ffd0f99b298965db5c5d1f36670b15b
more tests
Change-Id: I4aada643b67e3e1207a504267a50dc75bc0176e3
small refactor to where column names are stored
Change-Id: I0635c9627520f786e4f45e501eb63158f915d2a0
Start working on GenericSerde
Change-Id: I1adddfb639da7d6a64dde06c46bc67c072806469
first cut that uses default_json
Change-Id: Icad9eb9eb36b76eea4ba1342193c649a9d419287
cleaning some tests stuff
Change-Id: I889c4788250590f6cc7f0d9d7a1756f71e9e5cc5
setting the default serde for storage handler
Change-Id: I00179ec97d43d0a955500a23ebdab770d26930a7
fix how reading the property
Change-Id: I1bba7e19defda3316b4ddb0e0721e45cf11be063
working version of generic serde
Change-Id: I370a87aaf55f599db1695775ec2737e54af81270
fixup squash with previous
Change-Id: I6adfc93efed84c38aa1ad7092660e4cca49bc29a
use nanos for logging
Change-Id: I9055e0813b3b4bcbf72db0900ab9d2cb480c8f8b
major refactor plus tests working q files tests
Change-Id: I5ffc1cfcb4708e7a89163c371027e92782f2e4fc
adding q files test
Change-Id: I887fff5e3fdcdb0322770e52f9a8ab732a8dbe86
commit d28f1c94956b65374d58f3cda94fbbee5ed3e6b4
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-13T18:00:55Z
re activate the test
Change-Id: I7f8ef8a44271286abbd5a36a92ccde87d2ef8839
commit f71f68d8f7458409d5f341c1ab262d6a894ebcc0
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-13T20:02:07Z
refactor names and added copy jars
Change-Id: I11ec3aa4f9e96efc81ca8e9994c7409625384764
commit 5c442137e3f4b02086fd2c42ce354e4d4dbe4cd3
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-13T20:11:35Z
fix headers
Change-Id: I478c0709ba2ca77a1139011006170e4ad0683617
commit dce3a0f8b4eb77a9929ff2b2edf9fde44d364244
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-13T22:25:32Z
clean code and add comments
Change-Id: I1f8b3b748b5ab4f8f7e594bf85433affca83b50e
commit 80d9d5bbd55ea6d1a5d2de8ddffa393e495008ed
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-14T00:10:11Z
fix hive package
Change-Id: I2f255590aab7cd0897583ab11ea02961d1114bbd
commit 172019a6ab06d6026a7ba69f4b2c719f2afb2408
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-14T00:35:57Z
added more tests
Change-Id: I59c8bee67877bb54a4dcf5803b9e15ab2c8f0c42
commit 77700b20b7e2aa604caaad89bcd030dfc8b8e925
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-14T01:01:09Z
intelliJ friendly warning supress
Change-Id: Ib6fb5a8a2fdc2e7cb13dde7f8386bf39b86a8926
commit 8cfbce6e3cd70a472ebb818f126d72596b39bd91
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-14T20:29:04Z
fix style issue
Change-Id: I36be0353de253e46fbe16e35b52c258c7784a63a
commit e99cac11432dcd0471a784e7d7eb389fda2309d4
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-15T01:45:33Z
adding more tests
Change-Id: I72a8a9083d4a0c387fede5ee572d831bc5db7ec8
commit df2b7774c086bd6359890c19886ff64bb42af028
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-15T16:14:04Z
adding empty iterator
Change-Id: Ibe29a536a689250211385ef2297f254701572fd5
commit 49edead90167b7b84ff9bde474f185078e3ab2dd
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-15T16:45:09Z
filter empty splits anyway
Change-Id: I31a947946efb68718951e6cb0fe08bc4c38e8f5e
commit f103ec8f148a3529b26435dfa5f54c5d77e42c67
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-15T16:53:43Z
fix typo in functions names
Change-Id: I712f09cfb8eeb16a1246df01190483eaa9a86004
commit 9591091a4ae07b1556c125c16f9b5002f81141b8
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-15T23:18:57Z
adding some tests with filter on timestamp
Change-Id: I133bc72117571dfb1c1f69bcecbda3c1c8be050b
commit 9b7f1ba8d69c1c49a9918c200310575e4700e688
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-17T18:11:46Z
return -1 from getPos to skip Header footer check
Change-Id: I051b3559998849a23112f6597e30a2006704c615
commit 41deffeedf8a0b616973e64933504ded7a2b29da
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-17T20:15:45Z
use byteswritabe instead of bytesref to use lazybinary sede
Change-Id: Ib20537c52ea1154b7d2bf2e92e57ddc79414d0c4
commit 9d603a8c4e46c5e2ef1cf3d66e6b0f9a548505fb
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-17T22:41:56Z
refactor names and add docs
Change-Id: Ia769a3e8af5f07a4f3625c2d4ba1c5b03f864a63
commit f764f8adff37704bf81c284d7b48914264de9b94
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-17T22:56:11Z
remove unsed kafka propreties
Change-Id: Idc1bd85d8fc88df6ab0238bb8dfe50344264d78c
commit 5ddac6e0c026294785d26f4f92bfa608caf6f35c
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-17T22:59:23Z
more cleanup
Change-Id: I2b493ef799ebb9fcf4d5c3847fb5c375337f6075
commit 3cdeb41ce04b75631b8832440bd945ef5ef6c852
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-17T23:14:02Z
refactor iterator test
Change-Id: Iff2d04a62cdb0053af272679ea0530ceedea435e
commit c023e51b5ab28f8fd432e1f93a2cf1cb84656854
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-18T01:40:41Z
adding supplier for reader
Change-Id: I3bb58d13be3e32bb7e9f5df6574004a3645bbb75
commit 59f7872b95f5668f629df52104e7807a52c22b97
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-18T02:59:52Z
fix the order of columns for Avro Serd
Change-Id: I5dfc21059e530976af04771b6ea1d5d467774a0a
commit 00615766903dbf10fa8a3c8c94f55cbeacbf626f
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-18T03:34:05Z
fix the json parser to use bytewritables
Change-Id: I2e1f63b3a10fe45a4eaad04c46bbba62e6a58b3b
commit 711e84164c1f4b959d46347e8def94cfd9b79573
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-18T14:40:59Z
small refactor for generic serde pulling inline code to fns
Change-Id: I90381220de28f8ba0da87ad2b7ddf2372c7288ed
commit 71a292b3fefc05fdff16ea0a5d35482fd858fdcb
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-20T04:40:26Z
added avro to kafka tests
Change-Id: I17e1641061c5a5b816aa8be4d761ead2064b946e
commit 3f86ad79ad2b6cae9a4b87d094a497c0c6fc9fbd
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-20T20:19:09Z
more docs reframe the TODOS
Change-Id: Ib84d603da833da1c76fbd14522e5c070849c6751
commit 4cfd31f006657ea764d921bde1d5983d1adbf3bb
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-22T22:28:53Z
refactor clean KafkaScanTrimmer.java
Change-Id: I2d37ab7fb5ed51bb68a010f0fa41ea33ef9e5fbc
commit dd9f22fece43057baa9d2b483e83352ab7460b46
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-22T22:57:43Z
Addressing review comments
Change-Id: Ide0513e4c36d4294fbbd65934f6671f78a8479eb
commit d36477521599c599c0191041977f9f6a7c082439
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-23T00:09:58Z
clean up code after code review
Change-Id: I99a7a487a8948d88700e8edd13dcc041ee2f4cc9
commit 6bff3b24ccaf7936e3880e4838bb70035488a8ce
Author: Slim Bouguerra <bslim@...>
Date: 2018-08-23T01:16:11Z
added start and end offset markers
Change-Id: I3a5f72a0a39003db4cb63f3f98909afded13d1cb
----
---
