Re: Core Data Reverse Engineering KickStarter Project
Just curious. Which Core Data document format are you trying to reverse engineer: XML, binary, SQLite, or all of them? Richard Somers On Jun 25, 2013, at 8:31 PM, Michael Crawford mdcrawf...@gmail.com wrote: Take a guess at the document format. Write an importer that reads that format. Put assertions everywhere. If your first guess is wrong, you'll trip an assertion. Lather. Rinse. Repeat. ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
At 8:52 PM -0500 6/22/13, Charles Srstka wrote: 2. Since the format isn't published, there's no reason that Apple might make changes to it in the future, I think you'll find that the reason Apple doesn't publish some specifications is specifically so they CAN change them w/o breaking 3rd party applications. A counter-example would be QuickTime which is (was) a bit too well documented to make the transition to 64-bit. - The safest thing to would probably to be to implement a Core Data workalike with a documented database schema and possibly an importer for real code data files. HTH, -Steve ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On Jun 25, 2013, at 8:44 AM, Steve Sisak wrote: The safest thing to would probably to be to implement a Core Data workalike with a documented database schema and possibly an importer for real code data files. This is what I've been thinking--with the importer asserted to a crazy extent, so that you get notified of anything that it doesn't completely understand. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On Jun 25, 2013, at 9:44 AM, Steve Sisak sgs-li...@codewell.com wrote: At 8:52 PM -0500 6/22/13, Charles Srstka wrote: 2. Since the format isn't published, there's no reason that Apple might make changes to it in the future, I think you'll find that the reason Apple doesn't publish some specifications is specifically so they CAN change them w/o breaking 3rd party applications. Yeah, I meant to say that there wasn't any reason that Apple might *not* make changes to it in the future, which to me is a problem. If Apple were to change the Core Data format, then FooWriter 1.0 running on 10.10, FooWriter 1.0 running on 10.9, FooWriter 2.0 running on 10.10, and FooWriter 2.0 running on 10.9 would be completely different document formats. A user could find himself incapable of opening a FooWriter 2.0 document even though he has FooWriter 2.0, because he's using a different version of the OS. Another user might try to send a file to another user for collaboration and find that he can't export it in a way that the other user will be able to open, since the other user is running an earlier version of OS X. Can you imagine the kind of confusion this would cause to the userbase? I guarantee you'd be hearing about it extremely frequently from your customers. Charles ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
Using coredata with a documented format should probably be accomplished with an NSIncrementalStore subclass rather than trying to reverse engineer the existing undocumented format. Noah Desch On Jun 25, 2013, at 11:30 AM, Scott Ribe scott_r...@elevated-dev.com wrote: On Jun 25, 2013, at 8:44 AM, Steve Sisak wrote: The safest thing to would probably to be to implement a Core Data workalike with a documented database schema and possibly an importer for real code data files. This is what I've been thinking--with the importer asserted to a crazy extent, so that you get notified of anything that it doesn't completely understand. -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/deschnl%40me.com This email sent to desc...@me.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On 25 Jun 2013, at 7:44 AM, Steve Sisak wrote: The safest thing to would probably to be to implement a Core Data workalike with a documented database schema and possibly an importer for real code data files. FWIW, Omni's open-source data objects framework was our response to our difficulties with CD. It doesn't do everything CoreData does, and it does some things CoreData doesn't, but if you're considering taking this route, you might be interested in looking at it. https://github.com/omnigroup/OmniGroup/tree/master/Frameworks/OmniDataObjects ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On Jun 25, 2013, at 10:47 AM, Wim Lewis w...@omnigroup.com wrote: FWIW, Omni's open-source data objects framework was our response to our difficulties with CD. It doesn't do everything CoreData does, and it does some things CoreData doesn't, but if you're considering taking this route, you might be interested in looking at it. Likewise, Couchbase Lite[1] and its predecessor TouchDB[2] are other alternative persistent storage libraries (which I happen to work on!) Their priorities are different than CoreData — sync is #1, with with object-mapping being less advanced. If your app needs to be able to sync data with a server or with other clients, I think it’s a no-brainer to choose over CoreData, given the limitations and poor quality of the latter’s iCloud integration. But they’re good as general purpose data stores as well, since they’re open-source, open-format, cross-platform, and give you API access to the underlying document-based NoSQL database. —Jens [1] https://github.com/couchbase/couchbase-lite-ios [2] http://touchdb.org ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On 6/25/13, Scott Ribe scott_r...@elevated-dev.com wrote: This is what I've been thinking--with the importer asserted to a crazy extent, so that you get notified of anything that it doesn't completely understand. Using assertions in an importer is one of my more-effective reverse engineering techniques. Take a guess at the document format. Write an importer that reads that format. Put assertions everywhere. If your first guess is wrong, you'll trip an assertion. Lather. Rinse. Repeat. The other thing I do is run the importer under Valgrind. That will even flag out-of-bound reads by so much as one byte. Valgrind includes a machine code interpreter, that validates every memory access, before passing system and library calls off to native code. -- Michael David Crawford mdcrawf...@gmail.com ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
Scott, How do you do it? Honestly I want to know. The best I've ever been able to come up with is that if someone always writes the same kind of code, say repeatedly writing eCommerce sites for different clients, then they can base an estimate for a new project on experience with past project. But that's not the way I've ever worked. I always write completely new things, that bear no relation at all to past projects, so I don't have any meaningful basis for estimates. My colleagues tell me I should know how to estimate. I can see how it would be helpful but I've never been able to figure out how to actually do it. Mike On Sun, Jun 23, 2013 at 5:49 AM, Scott Ribe scott_r...@elevated-dev.com wrote: On Jun 22, 2013, at 7:38 PM, Michael Crawford wrote: If you claim you know how to estimate software development time and cost, I don't believe you. I do, and I do *very* well at it. But I certainly cannot estimate reverse-engineering an undocumented format; it would be madness for anyone to claim otherwise ;-) -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice -- Michael David Crawford mdcrawford at gmail dot com Custom Software Development for the iPhone and Mac OS X http://www.dulcineatech.com/custom-software-development/ ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On 2013 Jun 24, at 06:10, Michael Crawford wrote: Scott, How do you do it? Honestly I want to know. The best I've ever been able to come up with is that if someone always writes the same kind of code, say repeatedly writing eCommerce sites for different clients, then they can base an estimate for a new project on experience with past project. But that's not the way I've ever worked. I always write completely new things, that bear no relation at all to past projects, so I don't have any meaningful basis for estimates. My colleagues tell me I should know how to estimate. I can see how it would be helpful but I've never been able to figure out how to actually do it. I concur in both personal experience and wondering how some manage to estimate (with more or less windage and the occasional planned miracle). The big sticking point is that, when you're always doing something new, there are a number of points in the project when you have to do research, and repeated rounds of experimentation... only you never know how many cycles of experimentation will be required to arrive at a satisfactory solution. It's not as though we haven't read a book or three on the subject of estimation (and the sections in the texts), but they seem to have a blind-spot. Or do they simply avoid doing significantly new things, and instead restrict what they do to the well-known? ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On 23 Jun 2013, at 11:38, Michael Crawford mdcrawf...@gmail.com wrote: To me, it's not that you'd have to write all the code from scratch that makes Core Data concerning, it's the fact that the format is undocumented. If Apple published a complete specification for the format, I'd be willing to use Core Data, but as it is, the prospect of having one's own file format as a black box is a huge turn-off to me. Thanks for tip, I was unaware of that. Sure I can see how it's nice to use the Core Data API, but one would have to be completely out of one's tree to save end-user documents in a completely undocumented format, even if it were cross-platform. I don't think you would have to be completely out of your tree. The precedent is encapsulation - you should not care about internal layouts, only what the API exposes it. Seems that is exactly what CD does. And I think CD does give some extra discipline in modelling things correctly and making it easy to persist things, so I think it's doing a lot for you, notwithstanding my concerns that it might be more ER than relational. Most DBMSs also do not publish their internal file formats because these can change and if the abstraction is good - completely irrelevant. Thus application programmers should be encouraged to be loose coupled on the internals of a system, only using it in the published ways. However, I can see your point that it is a concern that if Apple discontinues CD that suddenly we have a whole lot of unreadable data and really anything stored persistently should be cross-platform whether stored on disk, or transmitted across a network. But perhaps to do that, just a small utility program that uses to CD API to get the data will suffice. But it is a little like the Carl Sagan records sent out on Voyager I (I think) for aliens to enjoy - now how are they going to do that without a record player? It will be like trying to decipher the Dravidian script of Mohinjo-Daro (we got lucky with Hieroglyphics with the Rosetta Stone). Vint Cerf recently stated that we might lose a lot of data because we lose the ability to interpret it (not such a new thought - one of the threads of Godel, escher, Bach). I lost what would have otherwise been a really lucrative, really fun consulting gig in which I would have been able to use my Physics degree. I'm afraid that I made it clear to the hiring manager that I considered him an idiot for even asking me whether I knew Core Data. I didn't come right out and say so, but I sure did make it clear that I considered him ill-advised. Just now I'm about to register a KickStarter project that would compensate me for completely reverse-engineering the Core Data formats. However I need to set a fixed price goal, as well as a finite delivery schedule. One only gets money at all from kickstarter if both goals are met. That kind of project estimation, in my own experience, is an as-yet unsolved problem, not just for me but for every software engineer. If you claim you know how to estimate software development time and cost, I don't believe you. Don't even _think_ about suggesting that such reverse-engineering is illegal. Reverse-engineer is specifically protected under the law. If you have some proprietary method you want legal protection for, that's what patents are for. Trade secrets are used either when the invention is either not novel, or not unobvious and so not legally patentable, or when one hopes to maintain the secret for longer than the patent's twenty-year term, such as the secret recipe for Coca-Cola. There are only a very few, very trusted people who know what all the ingredients are. Had they patented it, the recipe would have been placed into the public domain a hundred years ago. I know this very well. I've reverse-engineed a whole bunch of different kinds of things. That resulted in a Senior Engineer position at Apple itself in the mid-90s, where in my role as a Debug Meister for the Traditional OS Integration team, I reverse-engineered a great many third-party applications that were found by QA to stimulate crashes in new builds of the Mac OS System. I worked on 7.5.2 and 7.5.3. For example, I once supplied Microsoft with the exact byte offset in the Word 6 binary, at which they started a one-shot timer, then after it fired, reset it, so it would tick continuously. Unfortunately, the Classic Mac OS didn't have any really clear concept of a process - some other Apple engineer once described the System as just a bunch of subroutines, so under a very heavy paging load, that timer would fire well after Word's executable code had been overwritten with some other data. My very first consulting gig when I hung out my shingle in April 1998, was to reverse engineer the Movie Magic Scheduling database format, so that Graphical Planet's handheld devices could interoperate with Movie Magic. Graphical Planet did
Re: Core Data Reverse Engineering KickStarter Project
On Jun 22, 2013, at 7:38 PM, Michael Crawford wrote: If you claim you know how to estimate software development time and cost, I don't believe you. I do, and I do *very* well at it. But I certainly cannot estimate reverse-engineering an undocumented format; it would be madness for anyone to claim otherwise ;-) -- Scott Ribe scott_r...@elevated-dev.com http://www.elevated-dev.com/ (303) 722-0567 voice ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On Jun 22, 2013, at 8:38 PM, Michael Crawford mdcrawf...@gmail.com wrote: Just now I'm about to register a KickStarter project that would compensate me for completely reverse-engineering the Core Data formats. I believe that some of the work on this has been done already; Cocotron and GNUStep both seem to have at least a partial implementation of Core Data. As I stated before, this still makes me hesitant to use it, since even with a reverse engineered version of the framework, you still have the same two problems: 1. You don't know if the reverse engineered framework is 100% complete and covers every single corner case that might arise in a Core Data file, and 2. Since the format isn't published, there's no reason that Apple might make changes to it in the future, including but not limited to features that are technically in the private, internal specification which haven't been implemented yet, but which a reader would still need to be able to deal with. Charles ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
Oopsy-Doodle. I'll look into what Cocotron and GNUStep have done before actually launching my KickStarter Project. Just about their very first requirement is to state one's funding goal and time deadline. I just took a wild guess and specified thirty days, and twenty thousand dollars. That might sound like a really high hourly rate for just one month's work, but I would have to work almost 24/7 to pull it off. Mike On Sat, Jun 22, 2013 at 6:52 PM, Charles Srstka cocoa...@charlessoft.com wrote: On Jun 22, 2013, at 8:38 PM, Michael Crawford mdcrawf...@gmail.com wrote: Just now I'm about to register a KickStarter project that would compensate me for completely reverse-engineering the Core Data formats. I believe that some of the work on this has been done already; Cocotron and GNUStep both seem to have at least a partial implementation of Core Data. As I stated before, this still makes me hesitant to use it, since even with a reverse engineered version of the framework, you still have the same two problems: 1. You don't know if the reverse engineered framework is 100% complete and covers every single corner case that might arise in a Core Data file, and 2. Since the format isn't published, there's no reason that Apple might make changes to it in the future, including but not limited to features that are technically in the private, internal specification which haven't been implemented yet, but which a reader would still need to be able to deal with. Charles -- Michael David Crawford mdcrawford at gmail dot com Custom Software Development for the iPhone and Mac OS X http://www.dulcineatech.com/custom-software-development/ ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
OK this is really creepy. KickStarter has the reasonable requirement that one verify one's identity. So I entered my first and last name, my real birthday - which I never post online _anywhere_, so as to avoid identity theft, and my present home address, which no one at all knows about because I'm presently couch-surfing. In less than ten seconds it asked which of four street addresses in lived at, in Owl's Head, Maine. While I did live in Owl's Head back in the day, there were very, very few people who knew my street address. KickStarter also knew the make and model of the car I owned at the time, as well as the exact name of a business partnership that I co-owned in the late eighties. I can see how they might come up with my street address and car from a credit report, but I don't have the first clue how they identified my business partnership. We only did one very small consulting gig, and received just one client paycheck. We did have a bank account, but that was over twenty years ago. I wouldn't have expected such records to be kept so long, and to be available so quickly. Mike On Sat, Jun 22, 2013 at 7:53 PM, Michael Crawford mdcrawf...@gmail.com wrote: Oopsy-Doodle. I'll look into what Cocotron and GNUStep have done before actually launching my KickStarter Project. Just about their very first requirement is to state one's funding goal and time deadline. I just took a wild guess and specified thirty days, and twenty thousand dollars. That might sound like a really high hourly rate for just one month's work, but I would have to work almost 24/7 to pull it off. Mike On Sat, Jun 22, 2013 at 6:52 PM, Charles Srstka cocoa...@charlessoft.com wrote: On Jun 22, 2013, at 8:38 PM, Michael Crawford mdcrawf...@gmail.com wrote: Just now I'm about to register a KickStarter project that would compensate me for completely reverse-engineering the Core Data formats. I believe that some of the work on this has been done already; Cocotron and GNUStep both seem to have at least a partial implementation of Core Data. As I stated before, this still makes me hesitant to use it, since even with a reverse engineered version of the framework, you still have the same two problems: 1. You don't know if the reverse engineered framework is 100% complete and covers every single corner case that might arise in a Core Data file, and 2. Since the format isn't published, there's no reason that Apple might make changes to it in the future, including but not limited to features that are technically in the private, internal specification which haven't been implemented yet, but which a reader would still need to be able to deal with. Charles -- Michael David Crawford mdcrawford at gmail dot com Custom Software Development for the iPhone and Mac OS X http://www.dulcineatech.com/custom-software-development/ -- Michael David Crawford mdcrawford at gmail dot com Custom Software Development for the iPhone and Mac OS X http://www.dulcineatech.com/custom-software-development/ ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: Core Data Reverse Engineering KickStarter Project
On Jun 22, 2013, at 7:58 PM, Michael Crawford mdcrawf...@gmail.com wrote: In less than ten seconds it asked which of four street addresses in lived at, in Owl's Head, Maine. While I did live in Owl's Head back in the day, there were very, very few people who knew my street address. KickStarter also knew the make and model of the car I owned at the time, as well as the exact name of a business partnership that I co-owned in the late eighties. You have encountered the Lexis-Nexis identity checker. This is OT for the list, so let's leave it at that. --Kyle Sluder ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
