[jira] [Comment Edited] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16211006#comment-16211006 ] marco polo edited comment on MINIFICPP-39 at 10/19/17 1:09 PM: --- [~calebj] I'll try to pull down your PR. I've built master on U14 a few days ago, so I will try with that PR. We may be linking to the wrong object in the test if it's attempting the static lib. was (Author: phrocker): [~calebj] I'll try to pull down your PR. I've built master on U14 a few days ago, so I will try with that PR. We may be linking to the wrong object in the test. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16209461#comment-16209461 ] Andrew Christianson edited comment on MINIFICPP-39 at 10/18/17 2:44 PM: If your cloud9 environment is based on ubuntu, you should be able to apt-get install rocksdb & the development headers (check the nifi dev mailing list for a recent post by [~phrocker] on this.) I looked at the code, and see your function to verify the contents of the file in the unit test. Nice work! I can't think of anything else off-hand that would preclude a PR where the development community can formally review everything. Lucky for you, another developer recently brought in libarchive (MINIFICPP-72), and I think they worked out the somewhat complex LICENSING documentation. Watch out for some minor conflicts since that PR isn't merged yet (both that branch and yours will bring in libarchive). Things should be compatible, but I imagine there will be a small amount of conflicting/redundant code in the CMakeLists. was (Author: achristianson): If your cloud9 environment is based on ubuntu, you should be able to apt-get install rocksdb & the development headers (check the nifi dev mailing list for a recent post by [~phrocker] on this. I looked at the code, and see your function to verify the contents of the file in the unit test. Nice work! I can't think of anything else off-hand that would preclude a PR where the development community can formally review everything. Lucky for you, another developer recently brought in libarchive (MINIFICPP-72), and I think they worked out the somewhat complex LICENSING documentation. Watch out for some minor conflicts since that PR isn't merged yet (both that branch and yours will bring in libarchive). Things should be compatible, but I imagine there will be a small amount of conflicting/redundant code in the CMakeLists. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (MINIFICPP-39) Create FocusArchive processor
[
https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16202138#comment-16202138
]
Andrew Christianson edited comment on MINIFICPP-39 at 10/12/17 3:51 PM:
[~calebj] nice job on the rebase and setting up tests. A couple of notes:
1) I noticed this bit seems to be unlimplemented in UnfocusArchive.onTrigger:
{code}
// Set new/updated lens stack to attribute; height(stack') = height(stack) - 1
// if (!flowFile->updateAttribute("lens.archive.stack", stackStr))
// {
//flowFile->addAttribute("lens.archive.stack", stackStr);
// }
{code}
The lens stack is already "popped" with this line:
lensStack.resize(lensStack.size() - 1);, so I think we just need to serialize
the json after that and set it back into the lens.archive.stack attribute.
2) The unit tests do cover onTrigger of both Focus & Unfocus, which is good,
but it would be nice to have some additional verification of the actual output
file. I.e. verify that it is a valid tar/what have you, and that the content is
valid. Along similar lines, it would be nice to verify that the flow file
content is correct after FocusArchive.
Looking good over all. I think this is close to ready for a PR.
was (Author: achristianson):
[~calebj] nice job on the rebase and setting up tests. A couple of notes:
1) I noticed this bit seems to be unlimplemented in UnfocusArchive.onTrigger:
{{
// Set new/updated lens stack to attribute; height(stack') = height(stack) - 1
// if (!flowFile->updateAttribute("lens.archive.stack", stackStr))
// {
//flowFile->addAttribute("lens.archive.stack", stackStr);
// }
}}
The lens stack is already "popped" with this line:
lensStack.resize(lensStack.size() - 1);, so I think we just need to serialize
the json after that and set it back into the lens.archive.stack attribute.
2) The unit tests do cover onTrigger of both Focus & Unfocus, which is good,
but it would be nice to have some additional verification of the actual output
file. I.e. verify that it is a valid tar/what have you, and that the content is
valid. Along similar lines, it would be nice to verify that the flow file
content is correct after FocusArchive.
Looking good over all. I think this is close to ready for a PR.
> Create FocusArchive processor
> -
>
> Key: MINIFICPP-39
> URL: https://issues.apache.org/jira/browse/MINIFICPP-39
> Project: NiFi MiNiFi C++
> Issue Type: Task
>Reporter: Andrew Christianson
>Assignee: Andrew Christianson
>Priority: Minor
>
> Create an FocusArchive processor which implements a lens over an archive
> (tar, etc.). A concise, though informal, definition of a lens is as follows:
> "Essentially, they represent the act of “peering into” or “focusing in on”
> some particular piece/path of a complex data object such that you can more
> precisely target particular operations without losing the context or
> structure of the overall data you’re working with."
> https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq
> Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on"
> an entry in the archive, perform processing *in-context* of that entry, then
> re-focus on the overall archive. This allows for transformation or other
> processing of an entry in the archive without losing the overall context of
> the archive.
> Initial format support is tar, due to its simplicity and ubiquity.
> Attributes:
> - Path (the path in the archive to focus; "/" to re-focus the overall archive)
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Comment Edited] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200729#comment-16200729 ] Caleb Johnson edited comment on MINIFICPP-39 at 10/11/17 6:39 PM: -- [~achristianson], Can you give [this branch|https://github.com/NiFiLocal/nifi-minifi-cpp/tree/MINIFI-244-rebase2] a look? If it looks fine, the next step for me is to switch everything that uses rapidjson over to jsoncpp. But for now, it's rebased to the latest master and I _think_ it'll build. was (Author: calebj): [~achristianson], Can you give [this branch|https://github.com/NiFiLocal/nifi-minifi-cpp/tree/MINIFI-244-rebase2] a look? If it looks fine, the next step for me is to switch everything that uses rapidjson over to jsoncpp. But for now, it's rebased to the latest master and I _think _ it'll build. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (MINIFICPP-39) Create FocusArchive processor
[ https://issues.apache.org/jira/browse/MINIFICPP-39?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16200729#comment-16200729 ] Caleb Johnson edited comment on MINIFICPP-39 at 10/11/17 6:38 PM: -- [~achristianson], Can you give [this branch|https://github.com/NiFiLocal/nifi-minifi-cpp/tree/MINIFI-244-rebase2] a look? If it looks fine, the next step for me is to switch everything that uses rapidjson over to jsoncpp. But for now, it's rebased to the latest master and I _think _ it'll build. was (Author: calebj): [~achristianson], Can you give [this branch|https://github.com/NiFiLocal/nifi-minifi-cpp/tree/MINIFI-244-rebase2] a look? If it looks fine, the next step for me is to switch everything that uses rapidjson over to jsoncpp. But for now, it's rebased to the latest master and I _think _it'll build. > Create FocusArchive processor > - > > Key: MINIFICPP-39 > URL: https://issues.apache.org/jira/browse/MINIFICPP-39 > Project: NiFi MiNiFi C++ > Issue Type: Task >Reporter: Andrew Christianson >Assignee: Andrew Christianson >Priority: Minor > > Create an FocusArchive processor which implements a lens over an archive > (tar, etc.). A concise, though informal, definition of a lens is as follows: > "Essentially, they represent the act of “peering into” or “focusing in on” > some particular piece/path of a complex data object such that you can more > precisely target particular operations without losing the context or > structure of the overall data you’re working with." > https://medium.com/@dtipson/functional-lenses-d1aba9e52254#.hdgsvbraq > Why an FocusArchive in MiNiFi? Simply put, it will enable us to "focus in on" > an entry in the archive, perform processing *in-context* of that entry, then > re-focus on the overall archive. This allows for transformation or other > processing of an entry in the archive without losing the overall context of > the archive. > Initial format support is tar, due to its simplicity and ubiquity. > Attributes: > - Path (the path in the archive to focus; "/" to re-focus the overall archive) -- This message was sent by Atlassian JIRA (v6.4.14#64029)
