The change was made to mitigate CVE-2018-17195 [1], which allowed a malicious
actor in a specific scenario to upload a template without authorization. This
could result in RCE. Denes’ suggestion about rewriting the Origin header in
your proxy should work.
[1] https://nifi.apache.org/security.h
Boris,
We also use them for data cleanup. A common pattern I established on my
team is to script out a service with ScriptedLookupService and use it to
either regenerate a missing field from other fields or rewrite a field with
bad data.
On Fri, Feb 22, 2019 at 2:38 PM Boris Tyukin wrote:
> awe
awesome, thanks, guys! I will try both options but lookup makes a lot of
sense and probably will be easier to support and understand.
We are planning to get NiFi 1.9 soon too, really excited with all the new
features especially load balancing connections and Hive 1.1 processor.
Which is funny beca
@Boris
Mark's approach will work for a lot of scenarios. I've used it extensively
with different clients.
On Fri, Feb 22, 2019 at 1:10 PM Mark Payne wrote:
> This is certainly a better route to go than my previous suggestion :) Have
> one flow that grabs one of the datasets and stores it somewh
Thanks Bryan -
I checked this out in the UI and, sure enough, Controller Service B doesn’t
show up as a referencing component when enabling controller service B. So the
UI is acting exactly as I would expect given what I am seeing via the API. For
some reason the references API is not returnin
This is certainly a better route to go than my previous suggestion :) Have one
flow that grabs one of the datasets and stores it somewhere.
In a CSV or XML file, even. Then, have a second flow that pulls the other
dataset and uses LookupRecord to perform
the enrichment. The CSVLookupService and X
Boris,
I would echo the cautions from Bryan & Joe. However, you could perceivably
achieve this by extracting out some id
into an attribute that would associate the two FlowFiles together (for example
'dataset.id'). Use MergeRecord or MergeContent
to merge the data together using that as a correl
I should add you can use NiFi to update the reference dataset in a
database/backing store in one flow. And have another flow that handles the
live stream/lookup,etc. MarkPayne/Others: I think there are blogs that
describe this pattern. Anyone have links?
On Fri, Feb 22, 2019 at 12:27 PM Joe Wit
Boris,
Great. So have a process to load the periodic dataset into a lookup
service. COuld be backed by a simple file, a database, Hive, whatever.
Then have the live flow run against that.
This reminds me - we should make a Kudu based lookup service i think. I'll
chat with some of our new Kudu
Thanks Joe and Bryan. In this case I don't need to do it in real-time,
probably once a day only.
I am thinking to trigger both pulls by generateflow processor, then merge
datasets somehow since flowfile id will be the same for both sets. And then
need to join somehow.
Would like to use nifi still
Right I agree with Bryan so let me expand a bit.
There are some key primitives that stream processing systems address as it
relates to joining two live streams that those systems are designed to
solve well. NiFi offers nothing special/unique in that space.
Now, as Bryan pointed out a really comm
Hi Boris,
Joining across two different data streams is not really something NiFi
is aiming to solve.
Generally I think we'd say that you'd use one of the stream processing
systems like Flink, Spark, Storm, etc.
Another possible option might be to pull the data and land it in a
common location li
Tim,
Hard to say what is going on with the references end-point, I would
have to play around with it, but I believe calling
"controller-services//references" should return other services
that reference the service with the given id. It is used in the UI
when you go to enable/disable a service, and
Hi guys,
I pull two datasets from two different databases on schedule and need to
join both on some ID and then publish combined dataset to Kafka.
What is the best way to do this? Puzzled how I would synchronize two data
pulls so data is joined for exact flowfiles I need, i.e. if there are
errors
Hi Elemir,
As a workaround you can try to overwrite the Origin header in the request
to the value which is expected by NiFi, in your case it should be
https://localhost. (i.e. add proxy_set_header Origin https://localhost; to
your nginx config).
I hope this helps,
Denes
On Fri, Feb 22, 2019 at 1
Andy, thank you very much. This is a section of my code excerpted and hand
jammed in here by me to give the question context. I apologize: I
introduced those typos when hand jamming. They are not in the code, but
thanks for pointing them out here. I do have the correct imports.
Unfortunately I hav
Hi Elemir,
I was able to reproduce your issue with a simple nginx-NiFi setup, both
running on localhost.
My guess is that the cause is that POST is missing from allowed methods
list from the /process-groups/*/templates/upload path [1].
The commit which introduced this change explicitly states that
Hey Maximilian,
It seems like format operator expects your timestamp to be a 13 digits integer.
I have tried with 1550648697000, and the output seems correct.
Also watch out, it seem that your format is incorrect,
format("-MM-dd'T'HH:mm:dd' [...]")
Thanks,
Fabian HUTIN
__
Thanks for the suggestion, for the record, this is exactly what I did and it
worked well.
At first I had a setup where I truncate the table before inserting it, but
first it makes the table for a (short) while completely empty which is not
desirable in all cases, also the flow is more clear wh
Hello!
I have a JSON file like so:
{"PROJECT_KEY":"XYZ ","LAST_METRIC_DATE":"1550648697"}
I put this into EvaluateJsonPath and create the following attributes:
Attribute:lastMetricDate
JSON path: $.LAST_METRIC_DATE
Which works, the file gets an attribute l
20 matches
Mail list logo
