1) And, file permissions may not necessarily allow a write-lock on a
file. The NiFi user might only be allowed read permissions to a given file.
2) I really like this concept, +1 to the idea. In this way, the "List"
operation is following the unix design philosophy of doing exactly one
thing o
On 1) there are very few guarantees across os. Some support locking but
many apps dont use it. File io is wild wild west of idioms.
On 2) you certainly can tackle it that way. This gets into the more art
than science part of designing and composing processors. Key is to always
keep the oper
Matt,
That is a good tip - I will be sure to do that when I get to the
implementation testing - right now I am still at the research phase, trying
to determine what combination of technologies are likely to deliver what we
need.
Ah - very good point - I will be sure to build some kind of algorithm
1. Is there any reason it wouldn't work to try to open the files for write
and only begin to handle it when it is writable? It seems like a file
source would typically open for write, write everything, and then close.
Cases where something re-opens and appends would obviously not work in that
case
Cool - https://issues.apache.org/jira/browse/NIFI-797
On Wed, Jul 29, 2015 at 11:08 AM, Joe Witt wrote:
> Certainly. Please do file a JIRA for it of course and if you run into
> anything needing help just fire away.
>
> Thanks!
>
> Joe
>
> On Wed, Jul 29, 2015 at 11:06 AM, Ryan Hendrickson
> w
Certainly. Please do file a JIRA for it of course and if you run into
anything needing help just fire away.
Thanks!
Joe
On Wed, Jul 29, 2015 at 11:06 AM, Ryan Hendrickson
wrote:
> Hi all,
>If agreed upon, I don't mind putting the fix in, but it won't be til
> Sunday or next week..
>
>T
Hi all,
If agreed upon, I don't mind putting the fix in, but it won't be til
Sunday or next week..
The PutS3Object reports to provenance that it uses an HTTP url, when the
default for the S3 is actually HTTPS. This makes it pretty misleading as
to what is happening.
This is verified HT
Awesome! Thanks Joe.
*Jeremiah Adams*
Senior Software Developer
Pearson
2154 East Commons Ave.
Suite 400
Centennial, CO 80122
Always Learning
Learn more at www.pearson.com
On Tue, Jul 28, 2015 at 4:52 PM, Joe Witt wrote:
> You'll want to uncomment both of these lines actually:
>
> #java.arg
Thanks Dan. Turned it into a Jira [1]. Let us know if you're
interested in submitting a patch for that. Happy to help you get over
any setup/env hurdles you run into.
https://issues.apache.org/jira/browse/NIFI-796
On Wed, Jul 29, 2015 at 10:21 AM, Dan S wrote:
> On lines 81-83 for the descrip
Some additional feature requests for sake of consideration...
For some file systems (I can think of one), the last modified date may not
be dependable or possibly not high enough precision. Additional strategies
could be considered for determining whether a file has been previously
processed. Fo
Turning noatime on kicks last mod out the window. It is for sure the
case when dealing with file IO that there really are no rules. As
Adam notes it is about giving options/strategies.
Surprisingly hard to do this well. But good MVP options exist to get
something out and get more feedback on tr
On lines 81-83 for the description of RATE_CONTROL_ATTRIBUTE_NAME
it states
"The name of an attribute whose values build toward the rate limit if Rate
Control Criteria is set to 'attribute value'. "+ "The value of the
attribute referenced by this property must be a positive integer, or the
FlowFi
Mark,
Yes, what your seeing is a bug. I've created a ticket [1] which will be
included in our next release. Check out the ticket if you want the
technical details of the issue.
Matt
[1] https://issues.apache.org/jira/browse/NIFI-795
On Wed, Jul 29, 2015 at 9:44 AM, Dooley, Mark
wrote:
> Hi Ni
Hi NiFi Devs,
I've been using the NiFi Rest API and noticed that the response codes from the
Rest API are different between single-node and clustered mode.
If I'm running a single-node and drag in a new ProcessGroup the response code
is 201.
If I'm running in clustered mode (1 NCM + 2 Nodes) an
Joe S,
I agree, i think the design of List/Fetch HDFS is extremely applicable to this.
The way it saves state is by
using a DistributedMapCacheServer. The intent is to run the List processor on
primary node only, and it
will store its state there so that if the primary node is changed, any other
JoeS
Sounds great. I'd ignore my provenance comment as that was really
more about how something external could keep tabs on progress, etc..
Mark Payne designed/built the List/Fetch HDFS one so I'll defer to him
for the good bits. But the logic to follow for saving state you'll
want is probably t
Joe,
I'm interested in working on List/FetchFile. It seems like starting with
[NIFI-631|https://issues.apache.org/jira/browse/NIFI-631] makes sense.
I'll look at List/FetchHDFS, but is there any further detail on how this
functionality should differ from GetFile? As for keeping state,
provenanc
17 matches
Mail list logo
