GitHub user selvaganesang opened a pull request:
https://github.com/apache/trafodion/pull/1461
[TRA[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for
text formatted hive tables
Removed the dependency on libhdfs in Trafodion code in some more portions
of the code.
Also, introduced HDFS Scan to use the java layer to read HDFS data
providing the following features:
a) Prefetch the data using double buffering concept
b) Avoid unnecessary data copy
c) Ensure that the data read is initiated to be read into the other
buffer in java layer, while it is being processed in one buffer in the native
side, without any hiccups.
This HDFS Scan is still being tested and hence it is turned off by
default.
Also changed the code to obtain millisecond-resolution modification
timestamp for HDFS files
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/selvaganesang/trafodion libhdfs_removal
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/trafodion/pull/1461.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1461
----
commit ac7066074611a09da33adf88673c2f023e7dda7d
Author: selvaganesang <selva.govindarajan@...>
Date: 2018-02-28T19:27:40Z
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text
formatted hive tables
commit e303b3a083154779bcde8a84e1e2abff12d365e8
Author: selvaganesang <selva.govindarajan@...>
Date: 2018-02-28T22:25:43Z
[TRAFODION-2917] Refactor Trafodion implementation of hdfs scan for text
formatted hive tables
Removed the dependency on libhdfs in Trafodion code in some more portions
of the code.
Also, introduced HDFS Scan to use the java layer to read HDFS data
providing the following features:
a) Prefetch the data using double buffering concept
b) Avoid unnecessary data copy
c) Ensure that the data read is initiated to be read into the other buffer
in java layer, while it is being processed in one buffer in the native side,
without any hiccups.
This HDFS Scan is still being tested and hence it is turned off by default.
Also changed the code to obtain millisecond-resolution modification
timestamp for HDFS files
----
---
