Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/376
to look at the new patch set (#2).
Change subject: Introduced Local Filesystem Feed Adapter which can be used for
continuous data ingestion from local filesystem. To determine what files to
ingest, the user can provide a regular expression for the system to match
before ingesting data.
......................................................................
Introduced Local Filesystem Feed Adapter which can be used for continuous data
ingestion from local filesystem. To determine what files to ingest, the user
can provide a regular expression for the system to match before ingesting data.
In this change, I have added filesystem based feed adapter. The adapter is
aliased push_localfs.
It works in the following way:
When it starts, it starts ingesting available data in directories passed in the
adapter
arguments. Once it is done with the existing files, it places a watch on the
directories passed.
It is push based and so will only stop using a disconnect feed statement.
Faulty records are
dropped and the feed will try to parse the next records.
Change-Id: I707756e3b4c9ffca4b55ec9817a08e5c16333010
---
A asterix-app/data/local-dir/data/even-more.txt
A asterix-app/data/local-dir/data/hello-world
A asterix-app/data/local-dir/data/ignored
A asterix-app/data/local-dir/data/more-records.txt
A asterix-app/data/local-dir/data/records.txt
A
asterix-app/src/test/resources/runtimets/queries/external/query1/query1.1.ddl.aql
A
asterix-app/src/test/resources/runtimets/queries/external/query1/query1.2.update.aql
A
asterix-app/src/test/resources/runtimets/queries/external/query1/query1.3.query.aql
A asterix-app/src/test/resources/runtimets/results/external/query1/query1.1.adm
M asterix-app/src/test/resources/runtimets/testsuite.xml
M
asterix-common/src/main/java/org/apache/asterix/common/parse/ITupleForwardPolicy.java
A
asterix-external-data/src/main/java/org/apache/asterix/external/adapter/factory/LocalFileSystemFeedAdapterFactory.java
M
asterix-external-data/src/main/java/org/apache/asterix/external/adapter/factory/NCFileSystemAdapterFactory.java
M
asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/ClientBasedFeedAdapter.java
M
asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/FeedClient.java
A
asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/FileSystemWatcher.java
A
asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/LocalFileSystemFeedAdapter.java
A
asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/LocalFilesystemFeedClient.java
M
asterix-external-data/src/main/java/org/apache/asterix/external/dataset/adapter/NCFileSystemAdapter.java
A
asterix-external-data/src/main/java/org/apache/asterix/external/indexing/input/NCLocalFilesystemDirecoryInputStream.java
M
asterix-metadata/src/main/java/org/apache/asterix/metadata/bootstrap/MetadataBootstrap.java
M
asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/AbstractTupleParser.java
M
asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/AsterixTupleParserFactory.java
M
asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/CounterTimerTupleForwardPolicy.java
M
asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/FrameFullTupleForwardPolicy.java
M
asterix-runtime/src/main/java/org/apache/asterix/runtime/operators/file/RateControlledTupleForwardPolicy.java
26 files changed, 1,097 insertions(+), 115 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb
refs/changes/76/376/2
--
To view, visit https://asterix-gerrit.ics.uci.edu/376
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I707756e3b4c9ffca4b55ec9817a08e5c16333010
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: abdullah alamoudi <[email protected]>
Gerrit-Reviewer: Ian Maxon <[email protected]>
Gerrit-Reviewer: Jenkins <[email protected]>
Gerrit-Reviewer: Murtadha Hubail <[email protected]>
Gerrit-Reviewer: Steven Jacobs <[email protected]>
