[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16213159#comment-16213159 ] ASF GitHub Bot commented on MINIFICPP-72: - Github user minifirocks closed the pull request at: https://github.com/apache/nifi-minifi-cpp/pull/146 > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16208625#comment-16208625 ] ASF GitHub Bot commented on MINIFICPP-72: - Github user minifirocks commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/146 @apiri @phrocker rebased. > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[
https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206170#comment-16206170
]
ASF GitHub Bot commented on MINIFICPP-72:
-
Github user minifirocks commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/146#discussion_r144901568
--- Diff: libminifi/include/processors/MergeContent.h ---
@@ -67,15 +70,19 @@ class BinaryConcatenationMerge : public MergeBin {
~ReadCallback() {
}
int64_t process(std::shared_ptr stream) {
- uint8_t buffer[buffer_size_];
--- End diff --
@apiri fixed for incremental read
> Add tar and compression support for MergeContent
>
>
> Key: MINIFICPP-72
> URL: https://issues.apache.org/jira/browse/MINIFICPP-72
> Project: NiFi MiNiFi C++
> Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: bqiu
> Fix For: 1.0.0
>
>
> Add tar and compression support for MergeContent
> will use the https://www.libarchive.org
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206167#comment-16206167 ] ASF GitHub Bot commented on MINIFICPP-72: - Github user minifirocks commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/146#discussion_r144901439 --- Diff: LICENSE --- @@ -564,4 +564,68 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE --- End diff -- @apiri fix the license > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[
https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16206160#comment-16206160
]
ASF GitHub Bot commented on MINIFICPP-72:
-
Github user minifirocks commented on a diff in the pull request:
https://github.com/apache/nifi-minifi-cpp/pull/146#discussion_r144900901
--- Diff: libminifi/include/processors/MergeContent.h ---
@@ -125,6 +127,127 @@ class BinaryConcatenationMerge : public MergeBin {
};
+// Archive Class
+class ArchiveMerge {
+public:
+ // Nest Callback Class for read stream
+ class ReadCallback: public InputStreamCallback {
+ public:
+ReadCallback(uint64_t size, struct archive *arch, struct archive_entry
*entry) :
+buffer_size_(size), arch_(arch), entry_(entry) {
+}
+~ReadCallback() {
+}
+int64_t process(std::shared_ptr stream) {
+ uint8_t buffer[buffer_size_];
+ int64_t ret = 0;
+ uint64_t read_size;
+ ret = stream->read(buffer, buffer_size_);
+ if (!stream)
+read_size = stream->getSize();
+ else
+read_size = buffer_size_;
+ ret = archive_write_header(arch_, entry_);
+ ret += archive_write_data(arch_, buffer, read_size);
+ return ret;
+}
+uint64_t buffer_size_;
+struct archive *arch_;
+struct archive_entry *entry_;
+ };
+ // Nest Callback Class for write stream
+ class WriteCallback: public OutputStreamCallback {
+ public:
+WriteCallback(std::string merge_type,
std::deque> &flows, core::ProcessSession
*session) :
+merge_type_(merge_type), flows_(flows), session_(session) {
+ size_ = 0;
+ stream_ = nullptr;
+}
+~WriteCallback() {
+}
+
+std::string merge_type_;
+std::deque> &flows_;
+core::ProcessSession *session_;
+std::shared_ptr stream_;
+int64_t size_;
+
+static la_ssize_t archive_write(struct archive *arch, void *context,
const void *buff, size_t size) {
+ WriteCallback *callback = (WriteCallback *) context;
+ la_ssize_t ret =
callback->stream_->write(reinterpret_cast(const_cast(buff)),
size);
+ if (ret > 0)
+callback->size_ += (int64_t) ret;
+ return ret;
+}
+
+int64_t process(std::shared_ptr stream) {
+ int64_t ret = 0;
+ struct archive *arch;
+
+ arch = archive_write_new();
+ if (merge_type_ == MERGE_FORMAT_TAR_VALUE) {
+archive_write_set_format_pax_restricted(arch); // tar format
+ }
+ if (merge_type_ == MERGE_FORMAT_ZIP_VALUE) {
+archive_write_set_format_zip(arch); // zip format
+ }
+ archive_write_set_bytes_per_block(arch, 0);
+ archive_write_add_filter_none(arch);
+ this->stream_ = stream;
+ archive_write_open(arch, this, NULL, archive_write, NULL);
+
+ for (auto flow : flows_) {
+struct archive_entry *entry = archive_entry_new();
+std::string fileName;
+flow->getAttribute(FlowAttributeKey(FILENAME), fileName);
+archive_entry_set_pathname(entry, fileName.c_str());
+archive_entry_set_size(entry, flow->getSize());
+archive_entry_set_mode(entry, S_IFREG | 0755);
+if (merge_type_ == MERGE_FORMAT_TAR_VALUE) {
+ std::string perm;
+ int permInt;
+ if (flow->getAttribute(BinFiles::TAR_PERMISSIONS_ATTRIBUTE,
perm)) {
+try {
+ permInt = std::stoi(perm);
+ archive_entry_set_perm(entry, (mode_t) permInt);
+} catch (...) {
--- End diff --
@apiri @phrocker fixed.
> Add tar and compression support for MergeContent
>
>
> Key: MINIFICPP-72
> URL: https://issues.apache.org/jira/browse/MINIFICPP-72
> Project: NiFi MiNiFi C++
> Issue Type: New Feature
>Affects Versions: 1.0.0
>Reporter: bqiu
> Fix For: 1.0.0
>
>
> Add tar and compression support for MergeContent
> will use the https://www.libarchive.org
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205883#comment-16205883 ] ASF GitHub Bot commented on MINIFICPP-72: - Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/146#discussion_r144844826 --- Diff: LICENSE --- @@ -564,4 +564,64 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +The libarchive distribution as a whole is Copyright by Tim Kientzle and others --- End diff -- This should be converted to a "This project includes ... which is available under ..." phrasing in lieu of taking the associated words from the contributing license. > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16205882#comment-16205882 ] ASF GitHub Bot commented on MINIFICPP-72: - Github user apiri commented on a diff in the pull request: https://github.com/apache/nifi-minifi-cpp/pull/146#discussion_r144844644 --- Diff: LICENSE --- @@ -564,4 +564,64 @@ ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE. +The libarchive distribution as a whole is Copyright by Tim Kientzle and others +and is subject to the copyright notice as below + +Copyright (c) 2003-2009 +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions +are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer + in this position and unchanged. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE AUTHOR(S) ``AS IS'' AND ANY EXPRESS OR +IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES +OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. +IN NO EVENT SHALL THE AUTHOR(S) BE LIABLE FOR ANY DIRECT, INDIRECT, +INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT +NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, +DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY +THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF +THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. (END LICENSE TEXT) + +Each individual file in this distribution should have a clear --- End diff -- We need to call out the specific items and not just include the boilerplate message by the source LICENSE. In the case of those exclusions we must note that "the project includes which is available under " and the associated copyright. As a reference, scope out the other items in this LICENSE to see how they are handled. We should not have the caveats listed by the source but interpret them and include the appropriate clauses for those items which we do include. > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16203763#comment-16203763 ] bqiu commented on MINIFICPP-72: --- Aldrin, Currently NIFI support merge format tar and zip. I already committed merge content for minifi. This jira is to add merge fomat tar and zip for minifI so that merge content processor for minifi has feature parity with nifi. I will add compress content processor for minifi in different jira > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16202873#comment-16202873 ] ASF GitHub Bot commented on MINIFICPP-72: - Github user minifirocks commented on the issue: https://github.com/apache/nifi-minifi-cpp/pull/146 @apiri remove test and doc from lib archieve, update LICENSE to add each author. > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172364#comment-16172364 ] bqiu commented on MINIFICPP-72: --- Hey, Aldrin the NiFi merge content processor support merge format like Binary Concatenation/TAR/ZIP. The MergeContent Processor that i did for MiNiFI only support Binary Concatenation for the merge BIN files. We need to add TAR and ZIP format. The compress content processor is different than merge content, if we get the common libarchive in, we can scope the work for compress content processor also. > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (MINIFICPP-72) Add tar and compression support for MergeContent
[ https://issues.apache.org/jira/browse/MINIFICPP-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16172078#comment-16172078 ] Aldrin Piri commented on MINIFICPP-72: -- Hey [~bqiu] Looks like this could also be involved with/more appropriate for a CompressContent processor. Could you outline what you propose covering with these enhancements to merge content? > Add tar and compression support for MergeContent > > > Key: MINIFICPP-72 > URL: https://issues.apache.org/jira/browse/MINIFICPP-72 > Project: NiFi MiNiFi C++ > Issue Type: New Feature >Affects Versions: 1.0.0 >Reporter: bqiu > Fix For: 1.0.0 > > > Add tar and compression support for MergeContent > will use the https://www.libarchive.org -- This message was sent by Atlassian JIRA (v6.4.14#64029)
