Whoops, scratch the caching part (not sure what I was thinking there). -- Jeff Coughlin
On Jan 13, 2014, at 7:16 PM, Jeff Coughlin <[email protected]> wrote: > But wouldn't your process be more complex? What about refObjects? Your > extra setData() calls? Friendly URLs? Caching (object broker)? > > We are talking about importing into FarCry, right? (maybe I missed part of > the discussion :) > > -- > Jeff Coughlin > > On Jan 13, 2014, at 7:09 PM, Justin Carter <[email protected]> wrote: > >> Sure you can use createData() for a complex use case, but if all you're >> doing is setting a label then SQL bulk inserts will also work fine :) >> >> cheers, >> Justin >> >> -- >> Justin Carter >> http://www.madfellas.com/blog >> http://twitter.com/justincarter >> >> >> On Tue, Jan 14, 2014 at 10:41 AM, Jeff Coughlin <[email protected]> >> wrote: >> Justin, >> >>> For bulk inserts we typically insert directly into the DB too for >>> performance reasons. As Jeff says, the two main things to be aware of are >>> setting the label and setting the friendly URL (assuming that content has a >>> friendly URL). If you have any other calculated/derived properties then you >>> could run the beforeSave and setDate as a post import process, but it >>> depends on how complex your custom types are as to what business logic you >>> need to run. >> >> >> You're making it more complicated than it needs to be. There are more >> efficient (and cleaner) ways to do it. I have scripts that import thousands >> (and sometimes millions) of records. You can do it all with createData() as >> long as you properly batch it. See my answer to Scott below for more >> details. >> >> --- >> Scott, >> >>> So you do bursts of imports, on a scheduled task or something? >> >> >> Whether it's a scheduled task or not is irrelevant (unless you plan to use >> onComplete in CF10's schedule - which is just awesome). >> >> Whatever process you're using now to import the data, when your cfm file is >> triggered you import, say, the first 100 items - in batches of 100 for this >> example (*see note below). Whatever the last item was, set that record's >> unique identifier to a temp variable (or whatever value you want to use to >> identify the last record). Then call the same import file at the bottom >> using cflocation and give it the variable for the unique ID. Using that ID, >> tell your script to start from the next record. Each time the file finishes >> (and before it calls itself again) the process is flushed from memory (which >> is the whole point of batching). >> >> *note: The batch size depends on how much RAM is available to the JVM. Some >> clients I have can do batches 50k and higher batches, but it really doesn't >> matter if you use low numbers. Even it you set it to a low number, it will >> still import very quickly using this process. The only difference is that >> you might have to run a query each time at the top of the file (I suggest >> running a manual query and limiting it to return only 100 records - or >> whatever your batch size is). >> >> There are a bunch of other things I'd do of course, like set a tracking >> variable that allows me to cancel a long-running import with a click from >> within the webtop, setting <cfsetting/> at the top of the file with a decent >> length requestTimeout, using a condition around the cflocation to know if >> the import process is complete and to stop (don't want a non-ending loop), >> etc. But you get the idea. >> >> I have one client that actually does import millions of records at a time >> using this process and it is quite fast and efficient (well, it takes many >> hours, but still faster than other solutions I've done in the past). And >> there is no need to run all your methods manually and then update each >> record a second time using setData(). *Note:* For that particular client I >> find it faster to import the data with no solr indexing. Once complete I >> index the data in batches using the same batch process. >> >> Let me know how it goes for you. >> >> -- >> Jeff Coughlin >> >> On Jan 13, 2014, at 6:19 PM, Justin Carter <[email protected]> wrote: >> >>> For bulk inserts we typically insert directly into the DB too for >>> performance reasons. As Jeff says, the two main things to be aware of are >>> setting the label and setting the friendly URL (assuming that content has a >>> friendly URL). If you have any other calculated/derived properties then you >>> could run the beforeSave and setDate as a post import process, but it >>> depends on how complex your custom types are as to what business logic you >>> need to run. >>> >>> Glad you got it sorted though :) >>> >>> cheers, >>> Justin >>> >>> -- >>> Justin Carter >>> http://www.madfellas.com/blog >>> http://twitter.com/justincarter >>> >>> >>> On Tue, Jan 14, 2014 at 10:02 AM, Scott Mebberson >>> <[email protected]> wrote: >>> Hey guys, >>> >>> Yeah. We don't always import using createData or setData. When doing it on >>> 1000s of rows (daily) it's WAYYYYY to slow. >>> >>> On one instance, we wrote a bunch of import scripts based largely in bulk >>> inserts within MYSQL (CSV straight to MYSQL) and then filled in the gaps >>> that the framework usually does via createData and setData. >>> >>> Lots of testing to get that right though! >>> >>> My current instance is slightly different though. But it's all working >>> sweet now, thanks. >>> >>> cheers, >>> Scott. >>> >>> On 14/01/2014, at 9:28 AM, Jeff Coughlin <[email protected]> wrote: >>> >>>>> Yeah, the label property will only get set by the framework if setData() >>>>> is called >>>> >>>> or when using createData() - which is what you'd normally use when >>>> manually importing data into FarCry. >>>> >>>> ...Unless you are setting the label in beforeSave() in which case manually >>>> calling createData() does not call beforeSave(). This is a known bug >>>> that's just been around for years (can't find the bug number). We just >>>> always make sure to call beforeSave() manually before using createData(). >>>> >>>> And always make sure to call application.fc.factory.farFU.setSystemFU() to >>>> create the friendly url as well after using createData(), as createData() >>>> doesn't call that either if also using displaySystemFU.cfm (bug number >>>> FC-2268). So just to be safe, I always call the aforementioned method for >>>> FU's after calling createData(). >>>> >>>> Good luck. >>>> >>>> -- >>>> Jeff Coughlin >>>> >>>> On Jan 13, 2014, at 5:36 PM, Justin Carter <[email protected]> >>>> wrote: >>>> >>>>> Yeah, the label property will only get set by the framework if setData() >>>>> is called, so if it was a direct SQL insert then it will bypass the >>>>> framework :) >>>>> >>>>> cheers, >>>>> Justin >>>>> >>>>> -- >>>>> Justin Carter >>>>> http://www.madfellas.com/blog >>>>> http://twitter.com/justincarter >>>>> >>>>> >>>>> On Tue, Jan 14, 2014 at 9:31 AM, Scott Mebberson >>>>> <[email protected]> wrote: >>>>> Hey Jeff, >>>>> >>>>> Of course. I'll give that a shot, and verify it that it doesn't happen >>>>> for new instances. I'm working with data imported by someone else, so >>>>> chances are they didn't script label field up when inserting the data. >>>>> >>>>> I think everyone can ignore this post! Sorry. >>>>> >>>>> cheers, >>>>> Scott. >>>>> >>>>> On 14/01/2014, at 8:57 AM, Jeff Coughlin <[email protected]> wrote: >>>>> >>>>>> I haven't seen this, but I'm still on FC 6.3 at the moment (except for >>>>>> some testing I'm doing in 7). Until a fix is found, one thing you can >>>>>> do is force it to save the title to the label by creating a custom >>>>>> beforeSave() method inside the custom type. If anything it will at >>>>>> least give you a temporary workaround. But if this is a bug in FC7, >>>>>> then it's a pretty serious one (IMO). >>>>>> >>>>>> Example: >>>>>> >>>>>> <cffunction name="beforeSave" access="public" output="false" >>>>>> returntype="struct"> >>>>>> <cfargument name="stProperties" required="true" type="struct" /> >>>>>> <cfargument name="stFields" required="true" type="struct" /> >>>>>> <cfargument name="stFormPost" required="false" type="struct" /> >>>>>> >>>>>> <cfset arguments.stProperties.label = arguments.stProperties.title /> >>>>>> >>>>>> <cfreturn super.beforeSave(argumentCollection=arguments) /> >>>>>> </cffunction> >>>>>> >>>>>> If you feel comfortable sharing your CFC, go ahead and post it (maybe >>>>>> we'll see something you're missing - sometimes it just takes a second >>>>>> set of eyes to see something we've overlooked). If not, you are welcome >>>>>> to send it to just me and I'll be happy to see if something jumps out. >>>>>> >>>>>> -- >>>>>> Jeff Coughlin >>>>>> >>>>>> On Jan 13, 2014, at 5:12 PM, Scott Mebberson <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Has anyone run into a problem in which labels for a custom object type >>>>>>> aren't working on all occasions. Works for some instances of the object >>>>>>> type, but not others. The type does have a title property, which always >>>>>>> contains the real value. >>>>>>> >>>>>>> I've tried the following: >>>>>>> updateApp (a few dozen times, and after each of the following) >>>>>>> added bLabel to the title cfproperty tag >>>>>>> added displayLabel.cfm to the type folder in webskins and am outputting >>>>>>> stObj.title >>>>>>> But still, no label. The label doesn't exist in the stObj that is >>>>>>> passed around to the various webskins. >>>>>>> >>>>>>> Any ideas? >>>>>>> >>>>>>> I'm running FarCry 7 (I did git pull on both core and farcrycms this >>>>>>> morning so they're fresh!). >>>>>>> >>>>>>> Thanks, >>>>>>> Scott. > -- You received this message cos you are subscribed to "farcry-dev" Google group. To post, email: [email protected] To unsubscribe, email: [email protected] For more options: http://groups.google.com/group/farcry-dev -------------------------------- Follow us on Twitter: http://twitter.com/farcry --- You received this message because you are subscribed to the Google Groups "farcry-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
