But wouldn't your process be more complex? What about refObjects? Your extra setData() calls? Friendly URLs? Caching (object broker)?
We are talking about importing into FarCry, right? (maybe I missed part of the discussion :) -- Jeff Coughlin On Jan 13, 2014, at 7:09 PM, Justin Carter <[email protected]> wrote: > Sure you can use createData() for a complex use case, but if all you're doing > is setting a label then SQL bulk inserts will also work fine :) > > cheers, > Justin > > -- > Justin Carter > http://www.madfellas.com/blog > http://twitter.com/justincarter > > > On Tue, Jan 14, 2014 at 10:41 AM, Jeff Coughlin <[email protected]> wrote: > Justin, > >> For bulk inserts we typically insert directly into the DB too for >> performance reasons. As Jeff says, the two main things to be aware of are >> setting the label and setting the friendly URL (assuming that content has a >> friendly URL). If you have any other calculated/derived properties then you >> could run the beforeSave and setDate as a post import process, but it >> depends on how complex your custom types are as to what business logic you >> need to run. > > > You're making it more complicated than it needs to be. There are more > efficient (and cleaner) ways to do it. I have scripts that import thousands > (and sometimes millions) of records. You can do it all with createData() as > long as you properly batch it. See my answer to Scott below for more details. > > --- > Scott, > >> So you do bursts of imports, on a scheduled task or something? > > > Whether it's a scheduled task or not is irrelevant (unless you plan to use > onComplete in CF10's schedule - which is just awesome). > > Whatever process you're using now to import the data, when your cfm file is > triggered you import, say, the first 100 items - in batches of 100 for this > example (*see note below). Whatever the last item was, set that record's > unique identifier to a temp variable (or whatever value you want to use to > identify the last record). Then call the same import file at the bottom > using cflocation and give it the variable for the unique ID. Using that ID, > tell your script to start from the next record. Each time the file finishes > (and before it calls itself again) the process is flushed from memory (which > is the whole point of batching). > > *note: The batch size depends on how much RAM is available to the JVM. Some > clients I have can do batches 50k and higher batches, but it really doesn't > matter if you use low numbers. Even it you set it to a low number, it will > still import very quickly using this process. The only difference is that > you might have to run a query each time at the top of the file (I suggest > running a manual query and limiting it to return only 100 records - or > whatever your batch size is). > > There are a bunch of other things I'd do of course, like set a tracking > variable that allows me to cancel a long-running import with a click from > within the webtop, setting <cfsetting/> at the top of the file with a decent > length requestTimeout, using a condition around the cflocation to know if the > import process is complete and to stop (don't want a non-ending loop), etc. > But you get the idea. > > I have one client that actually does import millions of records at a time > using this process and it is quite fast and efficient (well, it takes many > hours, but still faster than other solutions I've done in the past). And > there is no need to run all your methods manually and then update each record > a second time using setData(). *Note:* For that particular client I find it > faster to import the data with no solr indexing. Once complete I index the > data in batches using the same batch process. > > Let me know how it goes for you. > > -- > Jeff Coughlin > > On Jan 13, 2014, at 6:19 PM, Justin Carter <[email protected]> wrote: > >> For bulk inserts we typically insert directly into the DB too for >> performance reasons. As Jeff says, the two main things to be aware of are >> setting the label and setting the friendly URL (assuming that content has a >> friendly URL). If you have any other calculated/derived properties then you >> could run the beforeSave and setDate as a post import process, but it >> depends on how complex your custom types are as to what business logic you >> need to run. >> >> Glad you got it sorted though :) >> >> cheers, >> Justin >> >> -- >> Justin Carter >> http://www.madfellas.com/blog >> http://twitter.com/justincarter >> >> >> On Tue, Jan 14, 2014 at 10:02 AM, Scott Mebberson <[email protected]> >> wrote: >> Hey guys, >> >> Yeah. We don't always import using createData or setData. When doing it on >> 1000s of rows (daily) it's WAYYYYY to slow. >> >> On one instance, we wrote a bunch of import scripts based largely in bulk >> inserts within MYSQL (CSV straight to MYSQL) and then filled in the gaps >> that the framework usually does via createData and setData. >> >> Lots of testing to get that right though! >> >> My current instance is slightly different though. But it's all working sweet >> now, thanks. >> >> cheers, >> Scott. >> >> On 14/01/2014, at 9:28 AM, Jeff Coughlin <[email protected]> wrote: >> >>>> Yeah, the label property will only get set by the framework if setData() >>>> is called >>> >>> or when using createData() - which is what you'd normally use when manually >>> importing data into FarCry. >>> >>> ...Unless you are setting the label in beforeSave() in which case manually >>> calling createData() does not call beforeSave(). This is a known bug >>> that's just been around for years (can't find the bug number). We just >>> always make sure to call beforeSave() manually before using createData(). >>> >>> And always make sure to call application.fc.factory.farFU.setSystemFU() to >>> create the friendly url as well after using createData(), as createData() >>> doesn't call that either if also using displaySystemFU.cfm (bug number >>> FC-2268). So just to be safe, I always call the aforementioned method for >>> FU's after calling createData(). >>> >>> Good luck. >>> >>> -- >>> Jeff Coughlin >>> >>> On Jan 13, 2014, at 5:36 PM, Justin Carter <[email protected]> >>> wrote: >>> >>>> Yeah, the label property will only get set by the framework if setData() >>>> is called, so if it was a direct SQL insert then it will bypass the >>>> framework :) >>>> >>>> cheers, >>>> Justin >>>> >>>> -- >>>> Justin Carter >>>> http://www.madfellas.com/blog >>>> http://twitter.com/justincarter >>>> >>>> >>>> On Tue, Jan 14, 2014 at 9:31 AM, Scott Mebberson >>>> <[email protected]> wrote: >>>> Hey Jeff, >>>> >>>> Of course. I'll give that a shot, and verify it that it doesn't happen for >>>> new instances. I'm working with data imported by someone else, so chances >>>> are they didn't script label field up when inserting the data. >>>> >>>> I think everyone can ignore this post! Sorry. >>>> >>>> cheers, >>>> Scott. >>>> >>>> On 14/01/2014, at 8:57 AM, Jeff Coughlin <[email protected]> wrote: >>>> >>>>> I haven't seen this, but I'm still on FC 6.3 at the moment (except for >>>>> some testing I'm doing in 7). Until a fix is found, one thing you can do >>>>> is force it to save the title to the label by creating a custom >>>>> beforeSave() method inside the custom type. If anything it will at least >>>>> give you a temporary workaround. But if this is a bug in FC7, then it's >>>>> a pretty serious one (IMO). >>>>> >>>>> Example: >>>>> >>>>> <cffunction name="beforeSave" access="public" output="false" >>>>> returntype="struct"> >>>>> <cfargument name="stProperties" required="true" type="struct" /> >>>>> <cfargument name="stFields" required="true" type="struct" /> >>>>> <cfargument name="stFormPost" required="false" type="struct" /> >>>>> >>>>> <cfset arguments.stProperties.label = arguments.stProperties.title /> >>>>> >>>>> <cfreturn super.beforeSave(argumentCollection=arguments) /> >>>>> </cffunction> >>>>> >>>>> If you feel comfortable sharing your CFC, go ahead and post it (maybe >>>>> we'll see something you're missing - sometimes it just takes a second set >>>>> of eyes to see something we've overlooked). If not, you are welcome to >>>>> send it to just me and I'll be happy to see if something jumps out. >>>>> >>>>> -- >>>>> Jeff Coughlin >>>>> >>>>> On Jan 13, 2014, at 5:12 PM, Scott Mebberson <[email protected]> >>>>> wrote: >>>>> >>>>>> Has anyone run into a problem in which labels for a custom object type >>>>>> aren't working on all occasions. Works for some instances of the object >>>>>> type, but not others. The type does have a title property, which always >>>>>> contains the real value. >>>>>> >>>>>> I've tried the following: >>>>>> updateApp (a few dozen times, and after each of the following) >>>>>> added bLabel to the title cfproperty tag >>>>>> added displayLabel.cfm to the type folder in webskins and am outputting >>>>>> stObj.title >>>>>> But still, no label. The label doesn't exist in the stObj that is passed >>>>>> around to the various webskins. >>>>>> >>>>>> Any ideas? >>>>>> >>>>>> I'm running FarCry 7 (I did git pull on both core and farcrycms this >>>>>> morning so they're fresh!). >>>>>> >>>>>> Thanks, >>>>>> Scott. -- You received this message cos you are subscribed to "farcry-dev" Google group. To post, email: [email protected] To unsubscribe, email: [email protected] For more options: http://groups.google.com/group/farcry-dev -------------------------------- Follow us on Twitter: http://twitter.com/farcry --- You received this message because you are subscribed to the Google Groups "farcry-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
