Re: asynchronous nsurlconnection in nsoperation
Andreas,
thank you for your answer! What is the well? I believe the streaming work
so we unarchive the this on the fly then write to url to avoid unnecessary
actions with copy
*
*
On Thu, Mar 22, 2012 at 4:10 PM, Andreas Grosam agro...@onlinehome.dewrote:
On Mar 21, 2012, at 4:59 PM, Ariel Feinerman wrote:
Hi,
I wish to insert an asynchronous NSURLConnection into non-concurrent
NSOperation
the reason is to allow necessarily unarchive actions on the streaming
date
Is this for iOS or Mac OS?
This for iOS
The date is very large their amount is 400 Mb so the asynchronous is
necessarily
one cannot be in memory at once
- (void) connection: (NSURLConnection *) connection didReceiveData:
(NSData*) data {
// Append the new data to receivedData.
[_receivedData appendData: data];
if ([_receivedData length] = MAX_CHUNK) {
// unarchive
// write to file
// [receivedData setLength: 0];
}
}
Is there a correct way to do ?
Yes.
And your described approach works fine - if your connection delegates run
on a secondary thread. Not sure, if you are actually required to use
NSOperation, but basically, when you start your connection on a secondary
thread (in order to keep the main thread responsive), it should work fine.
(See the code below, how one can accomplish to start a connection on a
secondary thread.)
Though, your approach is not optimal and it also **requires** that you can
actually process (unarchive) **partial** data.
If you cannot process partial data, you have to search for a more
elaborated solution to this problem:
There are several approaches, from simple to more complex. And, there are
approaches which look promising but won't work!
What you need to use in any case is an **asynchronous** (NSURL)connection.
Don't use a synchronously scheduled NSURLConnection with that amount of
data!
1) The simplest one that works is to immediately write the received data
to a temporary file (appending partial data to a file works fine). Then
when finished downloading, process the temporary file, possibly leveraging
mmap, NSStream, GCD or NSOperation. For this approach, you don't need
anything special in the NSURLConnection. Starting it from the main thread
is sufficient. Writing to the temp file is usually fast enough to keep the
main thread responsive.
When you are processing the data, you can schedule the task on a secondary
thread.
This approach is simple to implement, but is suboptimal performance wise -
especially on devices with more than one CPU.
2) An approach that combines high performance with low memory foot-print
is one that is a bit more elaborated. This would also require to schedule
the connection on a secondary thread. (example on request)
3) A few approaches that WONT work and will likely eventually crash are
the following:
3.a)
- (void) connection:(NSURLConnection*)connection didReceiveData:(NSData*)
data
{
dispatch_async(queue, ^{processData:data;}];
}
where processData: is supposed to be able to handle partial data.
This will likely crash due to memory running out: If downloading is fast,
and processing is slow, GCD queues a lot of buffers - until memory runs out.
3.b)
- (void) connection:(NSURLConnection*)connection didReceiveData:(NSData*)
data
{
[_receivedData appendData: data];
}
And when finished, processing _receivedData.
This will crash due to memory running out.
Regards
Andreas
==
You can start a connection in a secondary thread, as follows:
Anywhere, for instance a ViewController handling the Start Button:
// start the NSURLConnection in a secondary thread:
[NSThread detachNewThreadSelector:@selector
(startConnectionInSecondaryThread)
toTarget:self withObject:nil];
And -startConnectionInSecondaryThread is implemented:
static NSString* kDownloadConnectionRunLoopMode =
@MyViewControllerDownloadConnectionRunMode;
Won't this thrash the work?
- (void) startConnectionInSecondaryThread
{
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
[self startConnection];
runLoopDone_ = NO;
// Enable the run loop:
// first, add a dummy source in order to prevent the Run loop from
exiting
// immediately when the connection closes :
[[NSRunLoop currentRunLoop] addPort:[NSMachPort port]
forMode:kDownloadConnectionRunLoopMode];
do {
BOOL processedSource = [[NSRunLoop currentRunLoop]
runMode:kDownloadConnectionRunLoopMode beforeDate:[NSDate distantFuture]];
} while (!runLoopDone_);
[pool release];
}
And finally, -startConnection, which is shown in more detail, to show some
things you need to care about:
- (void) startConnection
{
// Note: startConnection can be performed on secondary threads, thus we
need
// to schedule UIKit methods onto the main thread.
NSString* urlString = @http://exmample.com;;
Re: asynchronous nsurlconnection in nsoperation
At 6:59 PM +0300 3/21/12, Ariel Feinerman wrote: I wish to insert an asynchronous NSURLConnection into non-concurrent NSOperation the reason is to allow necessarily unarchive actions on the streaming date so Look at (and steal code from) the LinkedImageFetcher sample: http://developer.apple.com/library/mac/#samplecode/LinkedImageFetcher/ I'm pretty sure if you steal the Core Code directory, and subclass QHTTPOperation, that ought to do pretty much what you want. HTH, -Steve ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
Re: asynchronous nsurlconnection in nsoperation
On Mar 23, 2012, at 11:25 AM, Steve Sisak wrote:
At 6:59 PM +0300 3/21/12, Ariel Feinerman wrote:
I wish to insert an asynchronous NSURLConnection into non-concurrent
NSOperation the reason is to allow necessarily unarchive actions on the
streaming date so
Look at (and steal code from) the LinkedImageFetcher sample:
http://developer.apple.com/library/mac/#samplecode/LinkedImageFetcher/
I'm pretty sure if you steal the Core Code directory, and subclass
QHTTPOperation, that ought to do pretty much what you want.
HTH,
-Steve
The LinkedImageFetcher sample is quite a good example to look at when you are
required to use NSOperation to wrap a download. For a sample, it is quite
complex though.
So, basically, QHTTPOperation is a good but rather complex example which shows
how to run a NSURLConnection on a secondary thread, and also how to wrap it
into a NSOperation which can be canceled etc.
Still, the sample is not a good example to show how to *process* arbitrary data
*during* the download process. The sample shows two ways to handle data:
- write the received data into an NSOutputStream, and
- store the content into an accumulator - which is a mutable NSData.
The latter disqualifies since the OP's data is about several MBytes large, too
large for a mobile device.
And when the NSOutputStream is an ordinary file stream, we gain nothing than
writing into a file directly, which can be accomplished much easier.
If you subclass the NSOutputStream in order to achieve a fancier processing,
you will nevertheless run into issues which are completely orthogonal to using
NSOperation.
Furthermore, using NSStreams is often unduly cumbersome and often not that easy
as it sounds. You need a bound stream pair, and there have been various issues
with this.
Still, any process that handles the data via NSStreams must be able to handle
such data in **chunks**. If this is not possible, the solution with NSStream
becomes even more complex - if feasible at all. See also,
https://devforums.apple.com/message/608213#608213
If the data *can* be processed *partially*, then the simplest approach is just
doing this in
- (void) connection:(NSURLConnection*)connection didReceiveData:(NSData*) data
{
[self processChunk:data];
}
where processChunk: must perform synchronously - that is, it shall block until
after that chunk of data has been processed completely and has relinquished
ownership from data.
If it would perform asynchronously, many NSData buffers may possibly queued -
and this may consume too much memory.
Andreas
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: asynchronous nsurlconnection in nsoperation
Is this for iOS or Mac OS?
This for iOS
// dispatch_async(kBgQueue, ^{ NSData* data = [NSData
dataWithContentsOfURL: myUrl]; // THIS IS WHAT YOU WANT Ariel **
The date is very large their amount is 400 Mb so the asynchronous is
necessarily
one cannot be in memory at once
On Wed, Mar 21, 2012 at 10:36 PM, Sebastian Celis
li...@sebastiancelis.comwrote:
On Wed, Mar 21, 2012 at 10:59 AM, Ariel Feinerman arielfap...@gmail.com
wrote:
I wish to insert an asynchronous NSURLConnection into non-concurrent
NSOperation
Hi Ariel,
MBRequest does this. It is done by forcing the operation's runloop to
continue running while the network connection is in progress. I would
recommend looking though the source code. Specifically, take a look at
MBURLConnectionOperation.
https://github.com/mobiata/MBRequest
https://github.com/mobiata/MBRequest/blob/master/Classes/MBURLConnectionOperation.m
- Sebastian
--
best regards
Ariel
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: asynchronous nsurlconnection in nsoperation
On Mar 21, 2012, at 4:59 PM, Ariel Feinerman wrote:
Hi,
I wish to insert an asynchronous NSURLConnection into non-concurrent
NSOperation
the reason is to allow necessarily unarchive actions on the streaming date
Is this for iOS or Mac OS?
This for iOS
The date is very large their amount is 400 Mb so the asynchronous is
necessarily
one cannot be in memory at once
- (void) connection: (NSURLConnection *) connection didReceiveData:
(NSData*) data {
// Append the new data to receivedData.
[_receivedData appendData: data];
if ([_receivedData length] = MAX_CHUNK) {
// unarchive
// write to file
// [receivedData setLength: 0];
}
}
Is there a correct way to do ?
Yes.
And your described approach works fine - if your connection delegates run on a
secondary thread. Not sure, if you are actually required to use NSOperation,
but basically, when you start your connection on a secondary thread (in order
to keep the main thread responsive), it should work fine. (See the code below,
how one can accomplish to start a connection on a secondary thread.)
Though, your approach is not optimal and it also **requires** that you can
actually process (unarchive) **partial** data.
If you cannot process partial data, you have to search for a more elaborated
solution to this problem:
There are several approaches, from simple to more complex. And, there are
approaches which look promising but won't work!
What you need to use in any case is an **asynchronous** (NSURL)connection.
Don't use a synchronously scheduled NSURLConnection with that amount of data!
1) The simplest one that works is to immediately write the received data to a
temporary file (appending partial data to a file works fine). Then when
finished downloading, process the temporary file, possibly leveraging mmap,
NSStream, GCD or NSOperation. For this approach, you don't need anything
special in the NSURLConnection. Starting it from the main thread is sufficient.
Writing to the temp file is usually fast enough to keep the main thread
responsive.
When you are processing the data, you can schedule the task on a secondary
thread.
This approach is simple to implement, but is suboptimal performance wise -
especially on devices with more than one CPU.
2) An approach that combines high performance with low memory foot-print is one
that is a bit more elaborated. This would also require to schedule the
connection on a secondary thread. (example on request)
3) A few approaches that WONT work and will likely eventually crash are the
following:
3.a)
- (void) connection:(NSURLConnection*)connection didReceiveData:(NSData*) data
{
dispatch_async(queue, ^{processData:data;}];
}
where processData: is supposed to be able to handle partial data.
This will likely crash due to memory running out: If downloading is fast, and
processing is slow, GCD queues a lot of buffers - until memory runs out.
3.b)
- (void) connection:(NSURLConnection*)connection didReceiveData:(NSData*) data
{
[_receivedData appendData: data];
}
And when finished, processing _receivedData.
This will crash due to memory running out.
Regards
Andreas
==
You can start a connection in a secondary thread, as follows:
Anywhere, for instance a ViewController handling the Start Button:
// start the NSURLConnection in a secondary thread:
[NSThread
detachNewThreadSelector:@selector(startConnectionInSecondaryThread)
toTarget:self withObject:nil];
And -startConnectionInSecondaryThread is implemented:
static NSString* kDownloadConnectionRunLoopMode =
@MyViewControllerDownloadConnectionRunMode;
- (void) startConnectionInSecondaryThread
{
NSAutoreleasePool* pool = [[NSAutoreleasePool alloc] init];
[self startConnection];
runLoopDone_ = NO;
// Enable the run loop:
// first, add a dummy source in order to prevent the Run loop from exiting
// immediately when the connection closes :
[[NSRunLoop currentRunLoop] addPort:[NSMachPort port]
forMode:kDownloadConnectionRunLoopMode];
do {
BOOL processedSource = [[NSRunLoop currentRunLoop]
runMode:kDownloadConnectionRunLoopMode beforeDate:[NSDate distantFuture]];
} while (!runLoopDone_);
[pool release];
}
And finally, -startConnection, which is shown in more detail, to show some
things you need to care about:
- (void) startConnection
{
// Note: startConnection can be performed on secondary threads, thus we need
// to schedule UIKit methods onto the main thread.
NSString* urlString = @http://exmample.com;;
NSURL* url = [NSURL URLWithString:urlString];
// Possibly configure/clear URL cache
NSTimeInterval request_timeout = 60.0;
NSMutableURLRequest* request = [NSMutableURLRequest requestWithURL:url
cachePolicy:0 timeoutInterval:request_timeout];
[request setValue:@gzip,deflate
asynchronous nsurlconnection in nsoperation
Hi,
I wish to insert an asynchronous NSURLConnection into non-concurrent
NSOperation
the reason is to allow necessarily unarchive actions on the streaming date
so
- (void) connection: (NSURLConnection *) connection didReceiveData:
(NSData*) data {
// Append the new data to receivedData.
[_receivedData appendData: data];
if ([_receivedData length] = MAX_CHUNK) {
// unarchive
// write to file
// [receivedData setLength: 0];
}
}
Is there a correct way to do ?
--
best regards
Ariel
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: asynchronous nsurlconnection in nsoperation
Is this for iOS or Mac OS?
I've used Grand Central Dispatch and [NSURLRequest requestWithURL:[NSURL
URLWithString:urlString]];
The second example is synchronous.
There is also:
NSData *response_NSData = [NSURLConnection
sendSynchronousRequest:my_NSURLRequest returningResponse:my_NSURLResponse
error:my_NSError];
and
sendAsynchronousRequest:queue:completionHandler:
Loads the data for a URL request and executes a handler block on an operation
queue when the request completes or fails.
+ (void)sendAsynchronousRequest:(NSURLRequest *)request queue:(NSOperationQueue
*)queue completionHandler:(void (^)(NSURLResponse*, NSData*, NSError*))handler
Look up the above in the NSURLConnection Class Reference
And also this, but disregard the older stuff.
http://www.raywenderlich.com/4295/multithreading-and-grand-central-dispatch-on-ios-for-beginners-tutorial
Here is my GDC stuff.
- (void)viewDidLoad {
NSLog(@In viewDidLoad);
[super viewDidLoad];
//NSString * const kRootURLString = @http://10.6.2.137/;;
NSURL *wordsURL = [NSURL
URLWithString:@http://localhost/~zav/Offices.txt?;];
NSURL *myUrl = [NSURL URLWithString:URLstring];
// dispatch_async(kBgQueue, ^{ NSData* data = [NSData
dataWithContentsOfURL: myUrl]; // THIS IS WHAT YOU WANT Ariel **
dispatch_sync(kBgQueue, ^{ NSData* data = [NSData
dataWithContentsOfURL: myUrl]; // doing a sync dispatch here - we want to wait
for it to finish
[self performSelectorOnMainThread:@selector(fetchedData:)
withObject:data waitUntilDone:YES];
});
// Uncomment the following line to display an Edit button in the navigation
bar for this view controller.
// self.navigationItem.rightBarButtonItem = self.editButtonItem;
}
...
- (void)fetchedData:(NSData *)responseData {
NSLog(@In fetchedData);
NSLog(@ );
// log URL data
NSString* newStr = [[NSString alloc] initWithData:responseData
encoding:NSUTF8StringEncoding];
NSLog(@%@, newStr);
}
Practice on a few simple cases until you find a method that you like. Note
that I haven't checked my code for any leaks. This is old play code from 2
weeks ago.
On Mar 21, 2012, at 11:59 AM, Ariel Feinerman wrote:
Hi,
I wish to insert an asynchronous NSURLConnection into non-concurrent
NSOperation
the reason is to allow necessarily unarchive actions on the streaming date
so
- (void) connection: (NSURLConnection *) connection didReceiveData:
(NSData*) data {
// Append the new data to receivedData.
[_receivedData appendData: data];
if ([_receivedData length] = MAX_CHUNK) {
// unarchive
// write to file
// [receivedData setLength: 0];
}
}
Is there a correct way to do ?
--
best regards
Ariel
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/zav%40mac.com
This email sent to z...@mac.com
- Alex Zavatone
___
Cocoa-dev mailing list (Cocoa-dev@lists.apple.com)
Please do not post admin requests or moderator comments to the list.
Contact the moderators at cocoa-dev-admins(at)lists.apple.com
Help/Unsubscribe/Update your Subscription:
https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com
This email sent to arch...@mail-archive.com
Re: asynchronous nsurlconnection in nsoperation
On Wed, Mar 21, 2012 at 10:59 AM, Ariel Feinerman arielfap...@gmail.com wrote: I wish to insert an asynchronous NSURLConnection into non-concurrent NSOperation Hi Ariel, MBRequest does this. It is done by forcing the operation's runloop to continue running while the network connection is in progress. I would recommend looking though the source code. Specifically, take a look at MBURLConnectionOperation. https://github.com/mobiata/MBRequest https://github.com/mobiata/MBRequest/blob/master/Classes/MBURLConnectionOperation.m - Sebastian ___ Cocoa-dev mailing list (Cocoa-dev@lists.apple.com) Please do not post admin requests or moderator comments to the list. Contact the moderators at cocoa-dev-admins(at)lists.apple.com Help/Unsubscribe/Update your Subscription: https://lists.apple.com/mailman/options/cocoa-dev/archive%40mail-archive.com This email sent to arch...@mail-archive.com
