Hi Matt, Thanks for your reply.
I will get updates regularly but I want to load the updated data once in a week. Staging table may solve this issue, but I'm looking for how row updated time should include in the query. Thanks On Tue, Apr 11, 2017 at 2:59 PM Matt Deaver <[email protected]> wrote: > Do you have updates coming in on your data flow? If so, you will need a > staging table and a merge process into your Teradata tables. > > If you do not have updated rows aka your Teradata tables are append-only > you can process data and insert (bulk load) into Teradata. > > I don't have experience doing this directly in Spark, though, but > according to this post > https://community.hortonworks.com/questions/63826/hi-is-there-any-connector-for-teradata-to-sparkwe.html > you will need to use a JDBC driver to connect. > > On Tue, Apr 11, 2017 at 1:23 PM, Vamsi Makkena <[email protected]> > wrote: > > I am reading the data from Oracle tables and Flat files (new excel file > every week) and write it to Teradata weekly using Pyspark. > > In the initial run it will load the all the data to Teradata. But in the > later runs I just want to read the new records from Oracle and Flatfiles > and want to append it to teradata tables. > > How can I do this using Pyspark, without touching the oracle and teradata > tables? > > Please post the sample code if possible. > > Thanks > > > > > -- > Regards, > > Matt > Data Engineer > https://www.linkedin.com/in/mdeaver > http://mattdeav.pythonanywhere.com/ >
