GitHub user aniketadnaik opened a pull request:
https://github.com/apache/carbondata/pull/1064
[CARBONDATA-<1173>] Stream ingestion - write path framework
Description:
This is a over all streaming write path framework implementation. It is
mainly targeted for "**streaming_ingest**" branch. This framework
implementation will help other developers to leverage a common code path for
developing other pieces of the streaming ingestion feature.
- Whether new unit test cases have been added or why no new tests are
required?
- This is a just framework implementation, unit tests will be added
with individual functionality commits - What manual testing you have
done?
- mvn clean verify
[INFO]
------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO]
[INFO] Apache CarbonData :: Parent ........................ SUCCESS [
1.533 s]
[INFO] Apache CarbonData :: Common ........................ SUCCESS [
1.335 s]
[INFO] Apache CarbonData :: Core .......................... SUCCESS
[02:34 min]
[INFO] Apache CarbonData :: Processing .................... SUCCESS [
5.158 s]
[INFO] Apache CarbonData :: Hadoop ........................ SUCCESS [
5.208 s]
[INFO] Apache CarbonData :: Spark Common .................. SUCCESS [
13.989 s]
[INFO] Apache CarbonData :: Spark ......................... SUCCESS
[02:33 min]
[INFO] Apache CarbonData :: Spark Common Test ............. SUCCESS
[05:15 min]
[INFO] Apache CarbonData :: Assembly ...................... SUCCESS [
1.683 s]
[INFO] Apache CarbonData :: Spark Examples ................ SUCCESS [
5.123 s]
[INFO]
------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO]
------------------------------------------------------------------------
[INFO] Total time: 10:58 min
[INFO] Finished at: 2017-06-16T17:24:23-07:00
[INFO] Final Memory: 85M/1078M
[INFO]
------------------------------------------------------------------------
- Made sure write path class invocation happens correctly with with
spark structured streaming (2.1)
- Made sure write path execution work flow with structured
streaming(2.1) using both socket and file
sources
- Any additional information to help reviewers in testing this change.
- structured streaming now accepts carbondata as a file format in
addition to data source
- temporary functionality writes out data in text format instead of
carbondata format, will be replaced with new functionality additions.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/aniketadnaik/carbondataStreamIngest
streamIngest-1173
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1064.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1064
----
commit 5403972898dc0d62471296476d6d5603dccbda10
Author: Aniket Adnaik <aniket.adn...@gmail.com>
Date: 2017-06-15T18:57:43Z
[CARBONDATA-1173] Streaming Ingest - write path framework implementation
commit 5270952a43b5e373957682dd75edd15048cb5aa3
Author: Aniket Adnaik <aniket.adn...@gmail.com>
Date: 2017-06-15T19:11:48Z
[CARBONDATA-1173] Streaming Ingest - write path framework implementation
commit ac3f1bbbee9c33472726afe10bac73a17a607cce
Author: Aniket Adnaik <aniket.adn...@gmail.com>
Date: 2017-06-16T18:04:19Z
[CARBONDATA-1173] Streaming Ingest - write path framework implementation
commit 4fc68b6c7a986ceda2c02713884d9293d29961fc
Author: Aniket Adnaik <aniket.adn...@gmail.com>
Date: 2017-06-19T22:19:21Z
[CARBONDATA-1173] Streaming Ingest - write path framework implementation
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
