Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Jorge Machado]

That would be probably the fastest way. 

Jorge Machado
www.jmachado.me





> On 22 Mar 2018, at 08:38, Brett Ryan  wrote:
> 
> Hmmm, now I’m doubting myself. It’s possible we sqoop to hdfs then sqoop
> out, will have to look, sorry if I am wrong.
> 
> On Thu, 22 Mar 2018 at 18:27, Jorge Machado  wrote:
> 
>> Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can
>> you past the command that you use ? That would be new to me and I would be
>> interested.
>> 
>> Jorge Machado
>> 
>> 
>> 
>> 
>> 
>>> On 22 Mar 2018, at 08:22, Brett Ryan  wrote:
>>> 
>>> Sure it does, I’m using it for postgres and MariaDB (which is
>> essentially MySQL).
>>> 
 On 22 Mar 2018, at 18:18, Jorge Machado  wrote:
 
 Sqoop does not import into a mySql database. Just into Hive if you tell
>> him to do so.
 You could use Nifi but if you have a lot of data may be you should try
>> Spark.  which reads and writes in Parallel.
 Using Nifi would work to but you have the overhead of pumping the data
>> over “insert” unless you copy the files into the server and on the server
>> then use some import bulk…..
 
 Jorge Machado
 
 
 
 
 
> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
> 
> Could Sqoop [1] be an option?
> 
> [1]: http://sqoop.apache.org/
> 
>> On 22 Mar 2018, at 16:33, Sivaprasanna 
>> wrote:
>> 
>> I had a chance to attempt a question raised on stackoverflow regarding
>> moving data from SQL Server to MySQL using NiFi. The user is using
>> GenerateTableFetch to read data from SQL Server and then try to use
>> LOAD
>> DATA command in ExecuteSQL but this involves writing the read SQL
>> Server
>> data to filesystem and then load it, which is a performance hit, I
>> suggested the user to try PutDatabaseRecord but I have never tried the
>> approach myself and going by the docs, I think it won't show any
>> performance benefit than LOAD DATA because the former reads from file
>> and
>> inserts at a high speed while the latter reads content and parses it
>> according to the configured Record Reader and insert the rows as a
>> single
>> batch. Confused, I wanted to get the community's opinion/thoughts on
>> this.
>> Please attempt the questions, if you have better suggestions.
>> 
>> Links:
>> 
>> -
>> 
>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>> -
>> 
>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>> 
>> Thanks,
>> 
>> Sivaprasanna
 
>> 
>> 

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Brett Ryan]

Hmmm, now I’m doubting myself. It’s possible we sqoop to hdfs then sqoop
out, will have to look, sorry if I am wrong.

On Thu, 22 Mar 2018 at 18:27, Jorge Machado  wrote:

> Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can
> you past the command that you use ? That would be new to me and I would be
> interested.
>
> Jorge Machado
>
>
>
>
>
> > On 22 Mar 2018, at 08:22, Brett Ryan  wrote:
> >
> > Sure it does, I’m using it for postgres and MariaDB (which is
> essentially MySQL).
> >
> >> On 22 Mar 2018, at 18:18, Jorge Machado  wrote:
> >>
> >> Sqoop does not import into a mySql database. Just into Hive if you tell
> him to do so.
> >> You could use Nifi but if you have a lot of data may be you should try
> Spark.  which reads and writes in Parallel.
> >> Using Nifi would work to but you have the overhead of pumping the data
> over “insert” unless you copy the files into the server and on the server
> then use some import bulk…..
> >>
> >> Jorge Machado
> >>
> >>
> >>
> >>
> >>
> >>> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
> >>>
> >>> Could Sqoop [1] be an option?
> >>>
> >>> [1]: http://sqoop.apache.org/
> >>>
>  On 22 Mar 2018, at 16:33, Sivaprasanna 
> wrote:
> 
>  I had a chance to attempt a question raised on stackoverflow regarding
>  moving data from SQL Server to MySQL using NiFi. The user is using
>  GenerateTableFetch to read data from SQL Server and then try to use
> LOAD
>  DATA command in ExecuteSQL but this involves writing the read SQL
> Server
>  data to filesystem and then load it, which is a performance hit, I
>  suggested the user to try PutDatabaseRecord but I have never tried the
>  approach myself and going by the docs, I think it won't show any
>  performance benefit than LOAD DATA because the former reads from file
> and
>  inserts at a high speed while the latter reads content and parses it
>  according to the configured Record Reader and insert the rows as a
> single
>  batch. Confused, I wanted to get the community's opinion/thoughts on
> this.
>  Please attempt the questions, if you have better suggestions.
> 
>  Links:
> 
>  -
> 
> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>  -
> 
> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
> 
>  Thanks,
> 
>  Sivaprasanna
> >>
>
>

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Brett Ryan]

Sure it does, I’m using it for postgres and MariaDB (which is essentially 
MySQL).

> On 22 Mar 2018, at 18:18, Jorge Machado  wrote:
> 
> Sqoop does not import into a mySql database. Just into Hive if you tell him 
> to do so. 
> You could use Nifi but if you have a lot of data may be you should try Spark. 
>  which reads and writes in Parallel. 
> Using Nifi would work to but you have the overhead of pumping the data over 
> “insert” unless you copy the files into the server and on the server then use 
> some import bulk…..
> 
> Jorge Machado
> 
> 
> 
> 
> 
>> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
>> 
>> Could Sqoop [1] be an option?
>> 
>> [1]: http://sqoop.apache.org/
>> 
>>> On 22 Mar 2018, at 16:33, Sivaprasanna  wrote:
>>> 
>>> I had a chance to attempt a question raised on stackoverflow regarding
>>> moving data from SQL Server to MySQL using NiFi. The user is using
>>> GenerateTableFetch to read data from SQL Server and then try to use LOAD
>>> DATA command in ExecuteSQL but this involves writing the read SQL Server
>>> data to filesystem and then load it, which is a performance hit, I
>>> suggested the user to try PutDatabaseRecord but I have never tried the
>>> approach myself and going by the docs, I think it won't show any
>>> performance benefit than LOAD DATA because the former reads from file and
>>> inserts at a high speed while the latter reads content and parses it
>>> according to the configured Record Reader and insert the rows as a single
>>> batch. Confused, I wanted to get the community's opinion/thoughts on this.
>>> Please attempt the questions, if you have better suggestions.
>>> 
>>> Links:
>>> 
>>> -
>>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>>> -
>>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>>> 
>>> Thanks,
>>> 
>>> Sivaprasanna
> 

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Jorge Machado]

Sqoop does not import into a mySql database. Just into Hive if you tell him to 
do so. 
You could use Nifi but if you have a lot of data may be you should try Spark.  
which reads and writes in Parallel. 
Using Nifi would work to but you have the overhead of pumping the data over 
“insert” unless you copy the files into the server and on the server then use 
some import bulk…..

Jorge Machado





> On 22 Mar 2018, at 08:13, Brett Ryan  wrote:
> 
> Could Sqoop [1] be an option?
> 
>  [1]: http://sqoop.apache.org/
> 
>> On 22 Mar 2018, at 16:33, Sivaprasanna  wrote:
>> 
>> I had a chance to attempt a question raised on stackoverflow regarding
>> moving data from SQL Server to MySQL using NiFi. The user is using
>> GenerateTableFetch to read data from SQL Server and then try to use LOAD
>> DATA command in ExecuteSQL but this involves writing the read SQL Server
>> data to filesystem and then load it, which is a performance hit, I
>> suggested the user to try PutDatabaseRecord but I have never tried the
>> approach myself and going by the docs, I think it won't show any
>> performance benefit than LOAD DATA because the former reads from file and
>> inserts at a high speed while the latter reads content and parses it
>> according to the configured Record Reader and insert the rows as a single
>> batch. Confused, I wanted to get the community's opinion/thoughts on this.
>> Please attempt the questions, if you have better suggestions.
>> 
>> Links:
>> 
>>  -
>>  
>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>>  -
>>  
>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>> 
>> Thanks,
>> 
>> Sivaprasanna

Re: Stackoverflow question: Moving data from one RDB to another through NiFi

[Brett Ryan]

Could Sqoop [1] be an option?

  [1]: http://sqoop.apache.org/

> On 22 Mar 2018, at 16:33, Sivaprasanna  wrote:
> 
> I had a chance to attempt a question raised on stackoverflow regarding
> moving data from SQL Server to MySQL using NiFi. The user is using
> GenerateTableFetch to read data from SQL Server and then try to use LOAD
> DATA command in ExecuteSQL but this involves writing the read SQL Server
> data to filesystem and then load it, which is a performance hit, I
> suggested the user to try PutDatabaseRecord but I have never tried the
> approach myself and going by the docs, I think it won't show any
> performance benefit than LOAD DATA because the former reads from file and
> inserts at a high speed while the latter reads content and parses it
> according to the configured Record Reader and insert the rows as a single
> batch. Confused, I wanted to get the community's opinion/thoughts on this.
> Please attempt the questions, if you have better suggestions.
> 
> Links:
> 
>   -
>   
> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>   -
>   
> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
> 
> Thanks,
> 
> Sivaprasanna

Stackoverflow question: Moving data from one RDB to another through NiFi

[Sivaprasanna]

I had a chance to attempt a question raised on stackoverflow regarding
moving data from SQL Server to MySQL using NiFi. The user is using
GenerateTableFetch to read data from SQL Server and then try to use LOAD
DATA command in ExecuteSQL but this involves writing the read SQL Server
data to filesystem and then load it, which is a performance hit, I
suggested the user to try PutDatabaseRecord but I have never tried the
approach myself and going by the docs, I think it won't show any
performance benefit than LOAD DATA because the former reads from file and
inserts at a high speed while the latter reads content and parses it
according to the configured Record Reader and insert the rows as a single
batch. Confused, I wanted to get the community's opinion/thoughts on this.
Please attempt the questions, if you have better suggestions.

Links:

   -
   
https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
   -
   
https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500

Thanks,

Sivaprasanna