Hey Jeff, It's just different scenarios require different methods.
Just to preface, we're now getting into a general discussion about imports, not my particular scenario which is now solved. In our scenario, we were updating thousands of rows, and only creating a few tens of rows. Once the update was complete, our client wanted to then peruse those rows and decide weather or not to publish them, edit them, etc. In this scenario, an import over hours is not acceptable. An import within minutes was required. Our first version of the import script timed-out (due to Java memory issues, which your bursting would have resolved, but not been acceptable within the greater solution). Our second version worked but took a couple of hours and the ran CPU up enormously. Our last version (dozens of versions later) which didn't use the framework at all, took only a couple of minutes for 40k+ rows. Not bad at all! So yeah, there is never a one method that suits all. Unfortunately! cheers, Scott. On 14/01/2014, at 10:47 AM, Jeff Coughlin <[email protected]> wrote: > Whoops, scratch the caching part (not sure what I was thinking there). > > -- > Jeff Coughlin > > On Jan 13, 2014, at 7:16 PM, Jeff Coughlin <[email protected]> wrote: > >> But wouldn't your process be more complex? What about refObjects? Your >> extra setData() calls? Friendly URLs? Caching (object broker)? >> >> We are talking about importing into FarCry, right? (maybe I missed part of >> the discussion :) >> >> -- >> Jeff Coughlin >> >> On Jan 13, 2014, at 7:09 PM, Justin Carter <[email protected]> wrote: >> >>> Sure you can use createData() for a complex use case, but if all you're >>> doing is setting a label then SQL bulk inserts will also work fine :) >>> >>> cheers, >>> Justin >>> >>> -- >>> Justin Carter >>> http://www.madfellas.com/blog >>> http://twitter.com/justincarter >>> >>> >>> On Tue, Jan 14, 2014 at 10:41 AM, Jeff Coughlin <[email protected]> >>> wrote: >>> Justin, >>> >>>> For bulk inserts we typically insert directly into the DB too for >>>> performance reasons. As Jeff says, the two main things to be aware of are >>>> setting the label and setting the friendly URL (assuming that content has >>>> a friendly URL). If you have any other calculated/derived properties then >>>> you could run the beforeSave and setDate as a post import process, but it >>>> depends on how complex your custom types are as to what business logic you >>>> need to run. >>> >>> >>> You're making it more complicated than it needs to be. There are more >>> efficient (and cleaner) ways to do it. I have scripts that import >>> thousands (and sometimes millions) of records. You can do it all with >>> createData() as long as you properly batch it. See my answer to Scott >>> below for more details. >>> >>> --- >>> Scott, >>> >>>> So you do bursts of imports, on a scheduled task or something? >>> >>> >>> Whether it's a scheduled task or not is irrelevant (unless you plan to use >>> onComplete in CF10's schedule - which is just awesome). >>> >>> Whatever process you're using now to import the data, when your cfm file is >>> triggered you import, say, the first 100 items - in batches of 100 for this >>> example (*see note below). Whatever the last item was, set that record's >>> unique identifier to a temp variable (or whatever value you want to use to >>> identify the last record). Then call the same import file at the bottom >>> using cflocation and give it the variable for the unique ID. Using that >>> ID, tell your script to start from the next record. Each time the file >>> finishes (and before it calls itself again) the process is flushed from >>> memory (which is the whole point of batching). >>> >>> *note: The batch size depends on how much RAM is available to the JVM. Some >>> clients I have can do batches 50k and higher batches, but it really doesn't >>> matter if you use low numbers. Even it you set it to a low number, it will >>> still import very quickly using this process. The only difference is that >>> you might have to run a query each time at the top of the file (I suggest >>> running a manual query and limiting it to return only 100 records - or >>> whatever your batch size is). >>> >>> There are a bunch of other things I'd do of course, like set a tracking >>> variable that allows me to cancel a long-running import with a click from >>> within the webtop, setting <cfsetting/> at the top of the file with a >>> decent length requestTimeout, using a condition around the cflocation to >>> know if the import process is complete and to stop (don't want a non-ending >>> loop), etc. But you get the idea. >>> >>> I have one client that actually does import millions of records at a time >>> using this process and it is quite fast and efficient (well, it takes many >>> hours, but still faster than other solutions I've done in the past). And >>> there is no need to run all your methods manually and then update each >>> record a second time using setData(). *Note:* For that particular client I >>> find it faster to import the data with no solr indexing. Once complete I >>> index the data in batches using the same batch process. >>> >>> Let me know how it goes for you. >>> >>> -- >>> Jeff Coughlin >>> >>> On Jan 13, 2014, at 6:19 PM, Justin Carter <[email protected]> >>> wrote: >>> >>>> For bulk inserts we typically insert directly into the DB too for >>>> performance reasons. As Jeff says, the two main things to be aware of are >>>> setting the label and setting the friendly URL (assuming that content has >>>> a friendly URL). If you have any other calculated/derived properties then >>>> you could run the beforeSave and setDate as a post import process, but it >>>> depends on how complex your custom types are as to what business logic you >>>> need to run. >>>> >>>> Glad you got it sorted though :) >>>> >>>> cheers, >>>> Justin >>>> >>>> -- >>>> Justin Carter >>>> http://www.madfellas.com/blog >>>> http://twitter.com/justincarter >>>> >>>> >>>> On Tue, Jan 14, 2014 at 10:02 AM, Scott Mebberson >>>> <[email protected]> wrote: >>>> Hey guys, >>>> >>>> Yeah. We don't always import using createData or setData. When doing it on >>>> 1000s of rows (daily) it's WAYYYYY to slow. >>>> >>>> On one instance, we wrote a bunch of import scripts based largely in bulk >>>> inserts within MYSQL (CSV straight to MYSQL) and then filled in the gaps >>>> that the framework usually does via createData and setData. >>>> >>>> Lots of testing to get that right though! >>>> >>>> My current instance is slightly different though. But it's all working >>>> sweet now, thanks. >>>> >>>> cheers, >>>> Scott. >>>> >>>> On 14/01/2014, at 9:28 AM, Jeff Coughlin <[email protected]> wrote: >>>> >>>>>> Yeah, the label property will only get set by the framework if setData() >>>>>> is called >>>>> >>>>> or when using createData() - which is what you'd normally use when >>>>> manually importing data into FarCry. >>>>> >>>>> ...Unless you are setting the label in beforeSave() in which case >>>>> manually calling createData() does not call beforeSave(). This is a >>>>> known bug that's just been around for years (can't find the bug number). >>>>> We just always make sure to call beforeSave() manually before using >>>>> createData(). >>>>> >>>>> And always make sure to call application.fc.factory.farFU.setSystemFU() >>>>> to create the friendly url as well after using createData(), as >>>>> createData() doesn't call that either if also using displaySystemFU.cfm >>>>> (bug number FC-2268). So just to be safe, I always call the >>>>> aforementioned method for FU's after calling createData(). >>>>> >>>>> Good luck. >>>>> >>>>> -- >>>>> Jeff Coughlin >>>>> >>>>> On Jan 13, 2014, at 5:36 PM, Justin Carter <[email protected]> >>>>> wrote: >>>>> >>>>>> Yeah, the label property will only get set by the framework if setData() >>>>>> is called, so if it was a direct SQL insert then it will bypass the >>>>>> framework :) >>>>>> >>>>>> cheers, >>>>>> Justin >>>>>> >>>>>> -- >>>>>> Justin Carter >>>>>> http://www.madfellas.com/blog >>>>>> http://twitter.com/justincarter >>>>>> >>>>>> >>>>>> On Tue, Jan 14, 2014 at 9:31 AM, Scott Mebberson >>>>>> <[email protected]> wrote: >>>>>> Hey Jeff, >>>>>> >>>>>> Of course. I'll give that a shot, and verify it that it doesn't happen >>>>>> for new instances. I'm working with data imported by someone else, so >>>>>> chances are they didn't script label field up when inserting the data. >>>>>> >>>>>> I think everyone can ignore this post! Sorry. >>>>>> >>>>>> cheers, >>>>>> Scott. >>>>>> >>>>>> On 14/01/2014, at 8:57 AM, Jeff Coughlin <[email protected]> wrote: >>>>>> >>>>>>> I haven't seen this, but I'm still on FC 6.3 at the moment (except for >>>>>>> some testing I'm doing in 7). Until a fix is found, one thing you can >>>>>>> do is force it to save the title to the label by creating a custom >>>>>>> beforeSave() method inside the custom type. If anything it will at >>>>>>> least give you a temporary workaround. But if this is a bug in FC7, >>>>>>> then it's a pretty serious one (IMO). >>>>>>> >>>>>>> Example: >>>>>>> >>>>>>> <cffunction name="beforeSave" access="public" output="false" >>>>>>> returntype="struct"> >>>>>>> <cfargument name="stProperties" required="true" type="struct" /> >>>>>>> <cfargument name="stFields" required="true" type="struct" /> >>>>>>> <cfargument name="stFormPost" required="false" type="struct" /> >>>>>>> >>>>>>> <cfset arguments.stProperties.label = >>>>>>> arguments.stProperties.title /> >>>>>>> >>>>>>> <cfreturn super.beforeSave(argumentCollection=arguments) /> >>>>>>> </cffunction> >>>>>>> >>>>>>> If you feel comfortable sharing your CFC, go ahead and post it (maybe >>>>>>> we'll see something you're missing - sometimes it just takes a second >>>>>>> set of eyes to see something we've overlooked). If not, you are >>>>>>> welcome to send it to just me and I'll be happy to see if something >>>>>>> jumps out. >>>>>>> >>>>>>> -- >>>>>>> Jeff Coughlin >>>>>>> >>>>>>> On Jan 13, 2014, at 5:12 PM, Scott Mebberson <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Has anyone run into a problem in which labels for a custom object type >>>>>>>> aren't working on all occasions. Works for some instances of the >>>>>>>> object type, but not others. The type does have a title property, >>>>>>>> which always contains the real value. >>>>>>>> >>>>>>>> I've tried the following: >>>>>>>> updateApp (a few dozen times, and after each of the following) >>>>>>>> added bLabel to the title cfproperty tag >>>>>>>> added displayLabel.cfm to the type folder in webskins and am >>>>>>>> outputting stObj.title >>>>>>>> But still, no label. The label doesn't exist in the stObj that is >>>>>>>> passed around to the various webskins. >>>>>>>> >>>>>>>> Any ideas? >>>>>>>> >>>>>>>> I'm running FarCry 7 (I did git pull on both core and farcrycms this >>>>>>>> morning so they're fresh!). >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Scott. >> > > > -- > You received this message cos you are subscribed to "farcry-dev" Google group. > To post, email: [email protected] > To unsubscribe, email: [email protected] > For more options: http://groups.google.com/group/farcry-dev > -------------------------------- > Follow us on Twitter: http://twitter.com/farcry > --- > You received this message because you are subscribed to a topic in the Google > Groups "farcry-dev" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/farcry-dev/yrrRHhE0H-4/unsubscribe. > To unsubscribe from this group and all of its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. -- You received this message cos you are subscribed to "farcry-dev" Google group. To post, email: [email protected] To unsubscribe, email: [email protected] For more options: http://groups.google.com/group/farcry-dev -------------------------------- Follow us on Twitter: http://twitter.com/farcry --- You received this message because you are subscribed to the Google Groups "farcry-dev" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
