Sure you can use createData() for a complex use case, but if all you're doing is setting a label then SQL bulk inserts will also work fine :)
cheers, Justin -- Justin Carter http://www.madfellas.com/blog http://twitter.com/justincarter On Tue, Jan 14, 2014 at 10:41 AM, Jeff Coughlin <[email protected]>wrote: > Justin, > > For bulk inserts we typically insert directly into the DB too for > performance reasons. As Jeff says, the two main things to be aware of are > setting the label and setting the friendly URL (assuming that content has a > friendly URL). If you have any other calculated/derived properties then you > could run the beforeSave and setDate as a post import process, but it > depends on how complex your custom types are as to what business logic you > need to run. > > > You're making it more complicated than it needs to be. There are more > efficient (and cleaner) ways to do it. I have scripts that import > thousands (and sometimes millions) of records. You can do it all with > createData() as long as you properly batch it. See my answer to Scott > below for more details. > > --- > Scott, > > So you do bursts of imports, on a scheduled task or something? > > > Whether it's a scheduled task or not is irrelevant (unless you plan to use > onComplete in CF10's schedule - which is just awesome). > > Whatever process you're using now to import the data, when your cfm file > is triggered you import, say, the first 100 items - in batches of 100 for > this example (*see note below). Whatever the last item was, set that > record's unique identifier to a temp variable (or whatever value you want > to use to identify the last record). Then call the same import file at the > bottom using cflocation and give it the variable for the unique ID. Using > that ID, tell your script to start from the next record. Each time the > file finishes (and before it calls itself again) the process is flushed > from memory (which is the whole point of batching). > > *note: The batch size depends on how much RAM is available to the JVM. > Some clients I have can do batches 50k and higher batches, but it really > doesn't matter if you use low numbers. Even it you set it to a low number, > it will still import very quickly using this process. The only difference > is that you might have to run a query each time at the top of the file (I > suggest running a manual query and limiting it to return only 100 records - > or whatever your batch size is). > > There are a bunch of other things I'd do of course, like set a tracking > variable that allows me to cancel a long-running import with a click from > within the webtop, setting <cfsetting/> at the top of the file with a > decent length requestTimeout, using a condition around the cflocation to > know if the import process is complete and to stop (don't want a non-ending > loop), etc. But you get the idea. > > I have one client that actually does import millions of records at a time > using this process and it is quite fast and efficient (well, it takes many > hours, but still faster than other solutions I've done in the past). And > there is no need to run all your methods manually and then update each > record a second time using setData(). *Note:* For that particular client I > find it faster to import the data with no solr indexing. Once complete I > index the data in batches using the same batch process. > > Let me know how it goes for you. > > -- > Jeff Coughlin > > On Jan 13, 2014, at 6:19 PM, Justin Carter <[email protected]> > wrote: > > For bulk inserts we typically insert directly into the DB too for > performance reasons. As Jeff says, the two main things to be aware of are > setting the label and setting the friendly URL (assuming that content has a > friendly URL). If you have any other calculated/derived properties then you > could run the beforeSave and setDate as a post import process, but it > depends on how complex your custom types are as to what business logic you > need to run. > > Glad you got it sorted though :) > > cheers, > Justin > > -- > Justin Carter > http://www.madfellas.com/blog > http://twitter.com/justincarter > > > On Tue, Jan 14, 2014 at 10:02 AM, Scott Mebberson < > [email protected]> wrote: > >> Hey guys, >> >> Yeah. We don't always import using createData or setData. When doing it >> on 1000s of rows (daily) it's WAYYYYY to slow. >> >> On one instance, we wrote a bunch of import scripts based largely in bulk >> inserts within MYSQL (CSV straight to MYSQL) and then filled in the gaps >> that the framework usually does via createData and setData. >> >> Lots of testing to get that right though! >> >> My current instance is slightly different though. But it's all working >> sweet now, thanks. >> >> cheers, >> Scott. >> >> On 14/01/2014, at 9:28 AM, Jeff Coughlin <[email protected]> wrote: >> >> Yeah, the label property will only get set by the framework if setData() >> is called >> >> >> or when using createData() - which is what you'd normally use when >> manually importing data into FarCry. >> >> ...Unless you are setting the label in beforeSave() in which case >> manually calling createData() does not call beforeSave(). This is a known >> bug that's just been around for years (can't find the bug number). We just >> always make sure to call beforeSave() manually before using createData(). >> >> And always make sure to call application.fc.factory.farFU.setSystemFU() >> to create the friendly url as well after using createData(), as >> createData() doesn't call that either if also using displaySystemFU.cfm >> (bug number FC-2268). So just to be safe, I always call the aforementioned >> method for FU's after calling createData(). >> >> Good luck. >> >> -- >> Jeff Coughlin >> >> On Jan 13, 2014, at 5:36 PM, Justin Carter <[email protected]> >> wrote: >> >> Yeah, the label property will only get set by the framework if setData() >> is called, so if it was a direct SQL insert then it will bypass the >> framework :) >> >> cheers, >> Justin >> >> -- >> Justin Carter >> http://www.madfellas.com/blog >> http://twitter.com/justincarter >> >> >> On Tue, Jan 14, 2014 at 9:31 AM, Scott Mebberson < >> [email protected]> wrote: >> >>> Hey Jeff, >>> >>> Of course. I'll give that a shot, and verify it that it doesn't happen >>> for new instances. I'm working with data imported by someone else, so >>> chances are they didn't script label field up when inserting the data. >>> >>> I think everyone can ignore this post! Sorry. >>> >>> cheers, >>> Scott. >>> >>> On 14/01/2014, at 8:57 AM, Jeff Coughlin <[email protected]> wrote: >>> >>> I haven't seen this, but I'm still on FC 6.3 at the moment (except for >>> some testing I'm doing in 7). Until a fix is found, one thing you can do >>> is force it to save the title to the label by creating a custom >>> beforeSave() method inside the custom type. If anything it will at least >>> give you a temporary workaround. But if this is a bug in FC7, then it's a >>> pretty serious one (IMO). >>> >>> Example: >>> >>> <cffunction name="beforeSave" access="public" output="false" >>> returntype="struct"> >>> <cfargument name="stProperties" required="true" type="struct" /> >>> <cfargument name="stFields" required="true" type="struct" /> >>> <cfargument name="stFormPost" required="false" type="struct" /> >>> >>> <cfset arguments.stProperties.label = arguments.stProperties.title /> >>> >>> <cfreturn super.beforeSave(argumentCollection=arguments) /> >>> </cffunction> >>> >>> If you feel comfortable sharing your CFC, go ahead and post it (maybe >>> we'll see something you're missing - sometimes it just takes a second set >>> of eyes to see something we've overlooked). If not, you are welcome to >>> send it to just me and I'll be happy to see if something jumps out. >>> >>> -- >>> Jeff Coughlin >>> >>> On Jan 13, 2014, at 5:12 PM, Scott Mebberson <[email protected]> >>> wrote: >>> >>> Has anyone run into a problem in which labels for a custom object type >>> aren't working on all occasions. Works for some instances of the object >>> type, but not others. The type does have a title property, which always >>> contains the real value. >>> >>> I've tried the following: >>> >>> - updateApp (a few dozen times, and after each of the following) >>> - added bLabel to the title cfproperty tag >>> - added displayLabel.cfm to the type folder in webskins and am >>> outputting stObj.title >>> >>> But still, no label. The label doesn't exist in the stObj that is passed >>> around to the various webskins. >>> >>> Any ideas? >>> >>> I'm running FarCry 7 (I did git pull on both core and farcrycms this >>> morning so they're fresh!). >>> >>> Thanks, >>> Scott. >>> >>> > -- > You received this message cos you are subscribed to "farcry-dev" Google > group. > To post, email: [email protected] > To unsubscribe, email: [email protected] > For more options: http://groups.google.com/group/farcry-dev > -------------------------------- > Follow us on Twitter: http://twitter.com/farcry > --- > You received this message because you are subscribed to the Google Groups > "farcry-dev" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message cos you are subscribed to "farcry-dev" Google group. To post, email: [email protected] To unsubscribe, email: [email protected] For more options: http://groups.google.com/group/farcry-dev -------------------------------- Follow us on Twitter: http://twitter.com/farcry --- You received this message because you are subscribed to the Google Groups "farcry-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
