Mark Payne wrote
> Dale,
>
> I think an image of the flow would be useful. Or better yet, if you can, a
> template of the flow, so
> that we can see all of the configuration being used.
>
> When you said you "get stuck at around 20 MB and then NiFi moves to a
> crawl" I'm not clear on
> what you are saying exactly. After you process 20 MB of the 189 MB CSV
> file? After you ingest
> 20 MB worth of files via the second FetchFile?
>
> Also, which directory has 85,000 files? The first directory being polled
> via ListFile, or the directory
> that you are picking up from via the second FetchFile?
>
> Thanks
> -Mark
Attached below you should find the template the flow in question. I had to
remove or alter some information, but that does not impact the workflow.
We have to ingest a CSV (called a DAT file) with variable delimiters and
qualifiers (so it's not really a CSV). The CSV has headers and many lines.
Each line corresponds to one text document. Each line also contains metadata
about and a URI to that document.
There are several folders which contain text files that are described by the
big CSV file. Two FetchFiles later in the flow will read attempt to find the
text documents corresponding to the URIs.
Here's a description of the directory structure:
*/dat* contains the gigantic CSV
*/directory1* contains thousands of text documents that are described by the
CSV
*/directory2* contains additional documents described by the CSV
*/directory3*... and so on
Here are the steps to my flow:
1) The first List-Fetch File you will find in the first Process Group named
"Find and Read DATFile". The DAT file reads the CSV that contains hundreds
of thousands of lines.
2) The Split DATFile Process Group chunks the CSV file into individual
FlowFiles.
3) In the Clean/Extract Metadata Process Group, we have to use regular
expressions via ExtractText to write the metadata to FlowFile attributes, to
then use AttributesToJSON and then store those JSON documents to Solr. The
Processors in this group use regular expressions to clean and validate the
later generated JSON document.
4) The Read Extracted Text Process Group contains the second FetchFile that
reads in files according to the URIs listed in the CSV. This is where the
read/write speed dips ("NiFi moves to a crawl") once 20-30 MB of text files
have been read through the second FetchFile.
5) The Store in Solr Process Group batches up JSON documents and stores them
to SolrCloud.
Document_Ingestion_Redacted.xml
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/n9916/Document_Ingestion_Redacted.xml>
--
View this message in context:
http://apache-nifi-developer-list.39713.n7.nabble.com/FetchFile-Cannot-Allocate-Enough-Memory-tp9720p9916.html
Sent from the Apache NiFi Developer List mailing list archive at Nabble.com.
