NiFi Versioned Process Group Status Icons

[Peter Wicks (pwicks)]

Why does NiFi show status icons for Versioned Process Group's on servers that 
are not configured to connect to a NiFi Registry?

Thanks,
  Peter

Re: [VOTE] Release Apache NiFi 1.6.0

[Andrew Lim]

+1 (non-binding)

-Ran full clean install on OS X (10.11.6)
-Tested integration with Secure NiFi Registry (1.5.0)
-Tested fine grained restricted component policies.  Found two minor issues:
Components are marked with the "restricted" red shield icon, but are 
not tagged as “restricted" [1]
PutParquet processor should require "write filesystem” permission [2]
-Reviewed documentation

Drew

[1] https://issues.apache.org/jira/browse/NIFI-5008
[2] https://issues.apache.org/jira/browse/NIFI-5009

> On Mar 22, 2018, at 12:11 PM, Joe Witt  wrote:
> 
> Hello,
> 
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
> 
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1122
> 
> The Git tag is nifi-1.6.0-RC1
> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=49a71f4740c9fac38958961f78dd3cde874b0e45
> 
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
> SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720ac891
> SHA512: 
> 6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f75519e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
> 
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
> 
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
> 
> 135 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422
> 
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0
> 
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test.  The please vote:
> 
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...

Re: [VOTE] Release Apache NiFi 1.6.0

[Aldrin Piri]

+1, binding


comments:

* the listed commit is the new one for 1.7.0 development, its parent
99bc762a181892aa9ac50b0c6c81e8159b052137 matches the source (
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=99bc762a181892aa9ac50b0c6c81e8159b052137
)
* signature and hashes look good
* build, tests, and contrib were all good on Debian 9, CentOS 7, and OS X
10.12
* ran some simple flows and functionality looked good
* did a cursory inspection of generated L files and didn't see anything
egregious.  We should make sure to update our copyright to 2018 for our
next release where applicable [1]

A lot of great additions for this release.  Great work by all the
contributors!

--aldrin

[1] https://issues.apache.org/jira/browse/NIFI-5006


On Thu, Mar 22, 2018 at 3:25 PM, o...@apache.org  wrote:

> +1
>
> Signed correctly
> Checksums correct
> Builds with checks
> Runs with simple flow
> Versions correct in about and documentation
> Documentation links work
>
> nice job.
>
> ottO/ottobackwards
>
>
> On March 22, 2018 at 12:11:35, Joe Witt (joew...@apache.org) wrote:
>
> Hello,
>
> I am pleased to be calling this vote for the source release of Apache
> NiFi nifi-1.6.0.
>
> The source zip, including signatures, digests, etc. can be found at:
> https://repository.apache.org/content/repositories/orgapachenifi-1122
>
> The Git tag is nifi-1.6.0-RC1
> The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
> https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;
> h=49a71f4740c9fac38958961f78dd3cde874b0e45
>
> Checksums of nifi-1.6.0-source-release.zip:
> SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
> SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720ac891
> SHA512: 6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f755
> 19e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/joewitt.asc
>
> KEYS file available here:
> https://dist.apache.org/repos/dist/release/nifi/KEYS
>
> 135 issues were closed/resolved for this release:
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?proje
> ctId=12316020=12342422
>
> Release note highlights can be found here:
> https://cwiki.apache.org/confluence/display/NIFI/Release+
> Notes#ReleaseNotes-Version1.6.0
>
> The vote will be open for 72 hours.
> Please download the release candidate and evaluate the necessary items
> including checking hashes, signatures, build
> from source, and test. The please vote:
>
> [ ] +1 Release this package as nifi-1.6.0
> [ ] +0 no opinion
> [ ] -1 Do not release this package because...
>
>

Re: [EXT] Re: Add interrupt option for stopped processors with active threads

[Joe Witt]

Karthik

I believe there are ways to determine underlying threads being
executed tied to particular linux processes in some cases.  However,
NiFi doesn't pin a processor to a specific/single thread.  A single
processor could use every thread in the thread pool during its
lifetime.  What would be cool to track if it can be done in an open
source friendly and opt-in for a given environment model is to show
CPU and memory usage for a single processor over time.

Thanks
Joe

On Thu, Mar 22, 2018 at 2:50 PM, Karthik Kothareddy (karthikk) [CONT -
Type 2]  wrote:
>
> Also on the same note, is there a way to get the underlying PID (unix) for a 
> Processor? I know we can get the PID for the NiFi itself and get the child 
> processes for that. But is there a way we can correlate them back to actual 
> NiFi Processors ?
>
>
>
> -Original Message-
> From: Matt Gilman [mailto:matt.c.gil...@gmail.com]
> Sent: Thursday, March 22, 2018 11:46 AM
> To: dev@nifi.apache.org
> Subject: [EXT] Re: Add interrupt option for stopped processors with active 
> threads
>
> There is a PR available for the backend work [1]. It is actively being 
> reviewed. Following that, there is additional work to make the front-end 
> changes [2]. It doesn't look like it's going to make it into 1.6.0.
> However, it should be in scope for 1.7.0 assuming sufficient review traction 
> for both efforts.
>
> Matt
>
> [1] https://github.com/apache/nifi/pull/2555
> [2] https://issues.apache.org/jira/browse/NIFI-1295
>
> On Thu, Mar 22, 2018 at 1:37 PM, Karthik Kothareddy (karthikk) [CONT - Type 
> 2]  wrote:
>
>> Hello,
>>
>> Is there an ETA on the below Improvement? Or is it in the scope for
>> any future releases?
>>
>> https://issues.apache.org/jira/browse/NIFI-78
>>
>> Thanks
>> Karthik
>>

RE: [EXT] Re: Add interrupt option for stopped processors with active threads

[Karthik Kothareddy (karthikk) [CONT - Type 2]]

Also on the same note, is there a way to get the underlying PID (unix) for a 
Processor? I know we can get the PID for the NiFi itself and get the child 
processes for that. But is there a way we can correlate them back to actual 
NiFi Processors ? 



-Original Message-
From: Matt Gilman [mailto:matt.c.gil...@gmail.com] 
Sent: Thursday, March 22, 2018 11:46 AM
To: dev@nifi.apache.org
Subject: [EXT] Re: Add interrupt option for stopped processors with active 
threads

There is a PR available for the backend work [1]. It is actively being 
reviewed. Following that, there is additional work to make the front-end 
changes [2]. It doesn't look like it's going to make it into 1.6.0.
However, it should be in scope for 1.7.0 assuming sufficient review traction 
for both efforts.

Matt

[1] https://github.com/apache/nifi/pull/2555
[2] https://issues.apache.org/jira/browse/NIFI-1295

On Thu, Mar 22, 2018 at 1:37 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] 
 wrote:

> Hello,
>
> Is there an ETA on the below Improvement? Or is it in the scope for 
> any future releases?
>
> https://issues.apache.org/jira/browse/NIFI-78
>
> Thanks
> Karthik
>

Apache NiFi 1.6.0 RC1 Release Helper Guide

[Joe Witt]

Hello Apache NiFi community,

Please find the associated guidance to help those interested in
validating/verifying the release so they can vote.

# Download latest KEYS file:
https://dist.apache.org/repos/dist/dev/nifi/KEYS

# Import keys file:
gpg --import KEYS

# [optional] Clear out local maven artifact repository

# Pull down nifi-1.6.0 source release artifacts for review:

wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.6.0/nifi-1.6.0-source-release.zip
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.6.0/nifi-1.6.0-source-release.zip.asc
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.6.0/nifi-1.6.0-source-release.zip.sha1
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.6.0/nifi-1.6.0-source-release.zip.sha256
wget 
https://dist.apache.org/repos/dist/dev/nifi/nifi-1.6.0/nifi-1.6.0-source-release.zip.sha512

# Verify the signature
gpg --verify nifi-1.6.0-source-release.zip.asc

# Verify the hashes (sha1, sha256, sha512) match the source and what
was provided in the vote email thread
shasum -a 1 nifi-1.6.0-source-release.zip
shasum -a 256 nifi-1.6.0-source-release.zip
shasum -a 512 nifi-1.6.0-source-release.zip

# Unzip nifi-1.6.0-source-release.zip

# Verify the build works including release audit tool (RAT) checks
cd nifi-1.6.0
mvn clean install -Pcontrib-check,include-grpc

# Verify the contents contain a good README, NOTICE, and LICENSE.

# Verify the git commit ID is correct

# Verify the RC was branched off the correct git commit ID

# Look at the resulting convenience binary as found in nifi-assembly/target

# Make sure the README, NOTICE, and LICENSE are present and correct

# Run the resulting convenience binary and make sure it works as expected

# Send a response to the vote thread indicating a +1, 0, -1 based on
your findings.

Thank you for your time and effort to validate the release!

RE: [EXT] Re: Add interrupt option for stopped processors with active threads

[Karthik Kothareddy (karthikk) [CONT - Type 2]]

Thanks Matt, will keep an eye on it.

-Original Message-
From: Matt Gilman [mailto:matt.c.gil...@gmail.com] 
Sent: Thursday, March 22, 2018 11:46 AM
To: dev@nifi.apache.org
Subject: [EXT] Re: Add interrupt option for stopped processors with active 
threads

There is a PR available for the backend work [1]. It is actively being 
reviewed. Following that, there is additional work to make the front-end 
changes [2]. It doesn't look like it's going to make it into 1.6.0.
However, it should be in scope for 1.7.0 assuming sufficient review traction 
for both efforts.

Matt

[1] https://github.com/apache/nifi/pull/2555
[2] https://issues.apache.org/jira/browse/NIFI-1295

On Thu, Mar 22, 2018 at 1:37 PM, Karthik Kothareddy (karthikk) [CONT - Type 2] 
 wrote:

> Hello,
>
> Is there an ETA on the below Improvement? Or is it in the scope for 
> any future releases?
>
> https://issues.apache.org/jira/browse/NIFI-78
>
> Thanks
> Karthik
>

Re: Add interrupt option for stopped processors with active threads

[Matt Gilman]

There is a PR available for the backend work [1]. It is actively being
reviewed. Following that, there is additional work to make the front-end
changes [2]. It doesn't look like it's going to make it into 1.6.0.
However, it should be in scope for 1.7.0 assuming sufficient review
traction for both efforts.

Matt

[1] https://github.com/apache/nifi/pull/2555
[2] https://issues.apache.org/jira/browse/NIFI-1295

On Thu, Mar 22, 2018 at 1:37 PM, Karthik Kothareddy (karthikk) [CONT - Type
2]  wrote:

> Hello,
>
> Is there an ETA on the below Improvement? Or is it in the scope for any
> future releases?
>
> https://issues.apache.org/jira/browse/NIFI-78
>
> Thanks
> Karthik
>

Add interrupt option for stopped processors with active threads

[Karthik Kothareddy (karthikk) [CONT - Type 2]]

Hello,

Is there an ETA on the below Improvement? Or is it in the scope for any future 
releases?

https://issues.apache.org/jira/browse/NIFI-78

Thanks
Karthik

[VOTE] Release Apache NiFi 1.6.0

[Joe Witt]

Hello,

I am pleased to be calling this vote for the source release of Apache
NiFi nifi-1.6.0.

The source zip, including signatures, digests, etc. can be found at:
https://repository.apache.org/content/repositories/orgapachenifi-1122

The Git tag is nifi-1.6.0-RC1
The Git commit ID is 49a71f4740c9fac38958961f78dd3cde874b0e45
https://git-wip-us.apache.org/repos/asf?p=nifi.git;a=commit;h=49a71f4740c9fac38958961f78dd3cde874b0e45

Checksums of nifi-1.6.0-source-release.zip:
SHA1: 4fb82f386a0aa83614cc01449e540527443811ce
SHA256: 4df6638ec87a5bee12e7978abc64137ee5da5fc8c399e34cf34ca1c3720ac891
SHA512: 
6fa536f9618c6c153c04df5db59913eaf3dd54ae2389368129ac6237f75519e1eee7ba0ca70145a95de01517a1c3ea1f36975895d9d72bb04abb56b7934e013a

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/joewitt.asc

KEYS file available here:
https://dist.apache.org/repos/dist/release/nifi/KEYS

135 issues were closed/resolved for this release:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316020=12342422

Release note highlights can be found here:
https://cwiki.apache.org/confluence/display/NIFI/Release+Notes#ReleaseNotes-Version1.6.0

The vote will be open for 72 hours.
Please download the release candidate and evaluate the necessary items
including checking hashes, signatures, build
from source, and test.  The please vote:

[ ] +1 Release this package as nifi-1.6.0
[ ] +0 no opinion
[ ] -1 Do not release this package because...

Re: [ANNOUNCE] New Apache NiFi Committer Mike Thomsen

[Joe Skora]

Congrats Mike!

On Wed, Mar 21, 2018 at 1:52 PM, Joe Percivall 
wrote:

> Thank you for your prior and continued contributions Mike, and congrats!
>
> On Wed, Mar 21, 2018 at 12:59 PM, Marco Gaido 
> wrote:
>
> > Congrats Mike! Well deserved!
> >
> > On Wed, 21 Mar 2018, 16:53 Sivaprasanna, 
> > wrote:
> >
> > > Congratulations, Mike :)
> > >
> > > On Wed, 21 Mar 2018 at 8:57 PM, Jeff  wrote:
> > >
> > > > Congrats Mike!
> > > >
> > > > On Wed, Mar 21, 2018 at 11:20 AM Scott Aslan 
> > > > wrote:
> > > >
> > > > > Congrats!
> > > > >
> > > > > On Wed, Mar 21, 2018 at 10:15 AM, Pierre Villard <
> > > > > pierre.villard...@gmail.com> wrote:
> > > > >
> > > > > > Congrats Mike, well deserved!
> > > > > >
> > > > > > 2018-03-21 14:55 GMT+01:00 Mike Thomsen  >:
> > > > > >
> > > > > > > Thanks everyone!
> > > > > > >
> > > > > > > On Wed, Mar 21, 2018 at 9:55 AM, Joe Witt 
> > > > wrote:
> > > > > > >
> > > > > > > > Mike
> > > > > > > >
> > > > > > > > Thanks for all the great contributions and reviews and
> > > discussions
> > > > > and
> > > > > > > > congratulations on the well deserved commit bit!
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > > Joe
> > > > > > > >
> > > > > > > > On Wed, Mar 21, 2018 at 1:53 PM, Kevin Doran <
> > kdo...@apache.org>
> > > > > > wrote:
> > > > > > > > > Congrats, Mike!
> > > > > > > > >
> > > > > > > > > On 3/21/18, 09:42, "Tony Kurc"  wrote:
> > > > > > > > >
> > > > > > > > > On behalf of the Apache NiFI PMC, I am very pleased to
> > > > announce
> > > > > > > that
> > > > > > > > Mike
> > > > > > > > > has accepted the PMC's invitation to become a committer
> > on
> > > > the
> > > > > > > > Apache NiFi
> > > > > > > > > project. We greatly appreciate all of Mike's hard work
> > and
> > > > > > generous
> > > > > > > > > contributions to the project. We look forward to his
> > > > continued
> > > > > > > > involvement
> > > > > > > > > in the project.
> > > > > > > > >
> > > > > > > > > Mike has been contributing to the project for quite a
> > > while,
> > > > > > > > contributing
> > > > > > > > > features such as  enhancements to MongoDB and HBase
> > > > processors
> > > > > as
> > > > > > > > well as
> > > > > > > > > things like improvements to Expression Language and
> > > > > > LookupService.
> > > > > > > > I'm sure
> > > > > > > > > many of you have interacted with Mike on the mailing
> > lists
> > > > > where
> > > > > > he
> > > > > > > > has
> > > > > > > > > been giving great input on both the project and
> > community.
> > > We
> > > > > > also
> > > > > > > > > appreciate his work reviewing and providing feedback on
> > new
> > > > > > > > contributions.
> > > > > > > > >
> > > > > > > > > Welcome and congratulations!
> > > > > > > > > Tony
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
>
>
> --
> *Joe Percivall*
> linkedin.com/in/Percivall
> e: jperciv...@apache.com
>

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Jorge Machado]

That would be probably the fastest way. 

Jorge Machado
www.jmachado.me





> On 22 Mar 2018, at 08:38, Brett Ryan  wrote:
> 
> Hmmm, now I’m doubting myself. It’s possible we sqoop to hdfs then sqoop
> out, will have to look, sorry if I am wrong.
> 
> On Thu, 22 Mar 2018 at 18:27, Jorge Machado  wrote:
> 
>> Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can
>> you past the command that you use ? That would be new to me and I would be
>> interested.
>> 
>> Jorge Machado
>> 
>> 
>> 
>> 
>> 
>>> On 22 Mar 2018, at 08:22, Brett Ryan  wrote:
>>> 
>>> Sure it does, I’m using it for postgres and MariaDB (which is
>> essentially MySQL).
>>> 
 On 22 Mar 2018, at 18:18, Jorge Machado  wrote:
 
 Sqoop does not import into a mySql database. Just into Hive if you tell
>> him to do so.
 You could use Nifi but if you have a lot of data may be you should try
>> Spark.  which reads and writes in Parallel.
 Using Nifi would work to but you have the overhead of pumping the data
>> over “insert” unless you copy the files into the server and on the server
>> then use some import bulk…..
 
 Jorge Machado
 
 
 
 
 
> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
> 
> Could Sqoop [1] be an option?
> 
> [1]: http://sqoop.apache.org/
> 
>> On 22 Mar 2018, at 16:33, Sivaprasanna 
>> wrote:
>> 
>> I had a chance to attempt a question raised on stackoverflow regarding
>> moving data from SQL Server to MySQL using NiFi. The user is using
>> GenerateTableFetch to read data from SQL Server and then try to use
>> LOAD
>> DATA command in ExecuteSQL but this involves writing the read SQL
>> Server
>> data to filesystem and then load it, which is a performance hit, I
>> suggested the user to try PutDatabaseRecord but I have never tried the
>> approach myself and going by the docs, I think it won't show any
>> performance benefit than LOAD DATA because the former reads from file
>> and
>> inserts at a high speed while the latter reads content and parses it
>> according to the configured Record Reader and insert the rows as a
>> single
>> batch. Confused, I wanted to get the community's opinion/thoughts on
>> this.
>> Please attempt the questions, if you have better suggestions.
>> 
>> Links:
>> 
>> -
>> 
>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>> -
>> 
>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>> 
>> Thanks,
>> 
>> Sivaprasanna
 
>> 
>> 

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Brett Ryan]

Hmmm, now I’m doubting myself. It’s possible we sqoop to hdfs then sqoop
out, will have to look, sorry if I am wrong.

On Thu, 22 Mar 2018 at 18:27, Jorge Machado  wrote:

> Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can
> you past the command that you use ? That would be new to me and I would be
> interested.
>
> Jorge Machado
>
>
>
>
>
> > On 22 Mar 2018, at 08:22, Brett Ryan  wrote:
> >
> > Sure it does, I’m using it for postgres and MariaDB (which is
> essentially MySQL).
> >
> >> On 22 Mar 2018, at 18:18, Jorge Machado  wrote:
> >>
> >> Sqoop does not import into a mySql database. Just into Hive if you tell
> him to do so.
> >> You could use Nifi but if you have a lot of data may be you should try
> Spark.  which reads and writes in Parallel.
> >> Using Nifi would work to but you have the overhead of pumping the data
> over “insert” unless you copy the files into the server and on the server
> then use some import bulk…..
> >>
> >> Jorge Machado
> >>
> >>
> >>
> >>
> >>
> >>> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
> >>>
> >>> Could Sqoop [1] be an option?
> >>>
> >>> [1]: http://sqoop.apache.org/
> >>>
>  On 22 Mar 2018, at 16:33, Sivaprasanna 
> wrote:
> 
>  I had a chance to attempt a question raised on stackoverflow regarding
>  moving data from SQL Server to MySQL using NiFi. The user is using
>  GenerateTableFetch to read data from SQL Server and then try to use
> LOAD
>  DATA command in ExecuteSQL but this involves writing the read SQL
> Server
>  data to filesystem and then load it, which is a performance hit, I
>  suggested the user to try PutDatabaseRecord but I have never tried the
>  approach myself and going by the docs, I think it won't show any
>  performance benefit than LOAD DATA because the former reads from file
> and
>  inserts at a high speed while the latter reads content and parses it
>  according to the configured Record Reader and insert the rows as a
> single
>  batch. Confused, I wanted to get the community's opinion/thoughts on
> this.
>  Please attempt the questions, if you have better suggestions.
> 
>  Links:
> 
>  -
> 
> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>  -
> 
> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
> 
>  Thanks,
> 
>  Sivaprasanna
> >>
>
>

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Brett Ryan]

Sure it does, I’m using it for postgres and MariaDB (which is essentially 
MySQL).

> On 22 Mar 2018, at 18:18, Jorge Machado  wrote:
> 
> Sqoop does not import into a mySql database. Just into Hive if you tell him 
> to do so. 
> You could use Nifi but if you have a lot of data may be you should try Spark. 
>  which reads and writes in Parallel. 
> Using Nifi would work to but you have the overhead of pumping the data over 
> “insert” unless you copy the files into the server and on the server then use 
> some import bulk…..
> 
> Jorge Machado
> 
> 
> 
> 
> 
>> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
>> 
>> Could Sqoop [1] be an option?
>> 
>> [1]: http://sqoop.apache.org/
>> 
>>> On 22 Mar 2018, at 16:33, Sivaprasanna  wrote:
>>> 
>>> I had a chance to attempt a question raised on stackoverflow regarding
>>> moving data from SQL Server to MySQL using NiFi. The user is using
>>> GenerateTableFetch to read data from SQL Server and then try to use LOAD
>>> DATA command in ExecuteSQL but this involves writing the read SQL Server
>>> data to filesystem and then load it, which is a performance hit, I
>>> suggested the user to try PutDatabaseRecord but I have never tried the
>>> approach myself and going by the docs, I think it won't show any
>>> performance benefit than LOAD DATA because the former reads from file and
>>> inserts at a high speed while the latter reads content and parses it
>>> according to the configured Record Reader and insert the rows as a single
>>> batch. Confused, I wanted to get the community's opinion/thoughts on this.
>>> Please attempt the questions, if you have better suggestions.
>>> 
>>> Links:
>>> 
>>> -
>>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>>> -
>>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>>> 
>>> Thanks,
>>> 
>>> Sivaprasanna
> 

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Jorge Machado]

Sqoop does not import into a mySql database. Just into Hive if you tell him to 
do so. 
You could use Nifi but if you have a lot of data may be you should try Spark.  
which reads and writes in Parallel. 
Using Nifi would work to but you have the overhead of pumping the data over 
“insert” unless you copy the files into the server and on the server then use 
some import bulk…..

Jorge Machado





> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
> 
> Could Sqoop [1] be an option?
> 
>  [1]: http://sqoop.apache.org/
> 
>> On 22 Mar 2018, at 16:33, Sivaprasanna  wrote:
>> 
>> I had a chance to attempt a question raised on stackoverflow regarding
>> moving data from SQL Server to MySQL using NiFi. The user is using
>> GenerateTableFetch to read data from SQL Server and then try to use LOAD
>> DATA command in ExecuteSQL but this involves writing the read SQL Server
>> data to filesystem and then load it, which is a performance hit, I
>> suggested the user to try PutDatabaseRecord but I have never tried the
>> approach myself and going by the docs, I think it won't show any
>> performance benefit than LOAD DATA because the former reads from file and
>> inserts at a high speed while the latter reads content and parses it
>> according to the configured Record Reader and insert the rows as a single
>> batch. Confused, I wanted to get the community's opinion/thoughts on this.
>> Please attempt the questions, if you have better suggestions.
>> 
>> Links:
>> 
>>  -
>>  
>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>>  -
>>  
>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>> 
>> Thanks,
>> 
>> Sivaprasanna

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Brett Ryan]

Could Sqoop [1] be an option?

  [1]: http://sqoop.apache.org/

> On 22 Mar 2018, at 16:33, Sivaprasanna  wrote:
> 
> I had a chance to attempt a question raised on stackoverflow regarding
> moving data from SQL Server to MySQL using NiFi. The user is using
> GenerateTableFetch to read data from SQL Server and then try to use LOAD
> DATA command in ExecuteSQL but this involves writing the read SQL Server
> data to filesystem and then load it, which is a performance hit, I
> suggested the user to try PutDatabaseRecord but I have never tried the
> approach myself and going by the docs, I think it won't show any
> performance benefit than LOAD DATA because the former reads from file and
> inserts at a high speed while the latter reads content and parses it
> according to the configured Record Reader and insert the rows as a single
> batch. Confused, I wanted to get the community's opinion/thoughts on this.
> Please attempt the questions, if you have better suggestions.
> 
> Links:
> 
>   -
>   
> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>   -
>   
> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
> 
> Thanks,
> 
> Sivaprasanna