Since you are using a direct solver almost for sure a bug in your parallel function or parallel Jacobian.
Try -snes_mf_operator try -snes_fd try -snes_type test as three different approaches to see what is going on. Barry > On Jan 13, 2016, at 2:51 PM, David Knezevic <david.kneze...@akselos.com> > wrote: > > Oops! I pasted the wrong text for the serial case. The correct text is below: > > Serial case: > NL step 0, |residual|_2 = 4.714515e-02 > Line search: gnorm after quadratic fit 7.862867755130e-02 > Line search: Cubically determined step, current gnorm > 4.663945044088e-02 lambda=1.4276549223307832e-02 > NL step 1, |residual|_2 = 4.663945e-02 > Line search: gnorm after quadratic fit 6.977268532963e-02 > Line search: Cubically determined step, current gnorm > 4.594912791877e-02 lambda=2.3644826349821228e-02 > NL step 2, |residual|_2 = 4.594913e-02 > Line search: gnorm after quadratic fit 5.502067915588e-02 > Line search: Cubically determined step, current gnorm > 4.494531287593e-02 lambda=4.1260496881982515e-02 > NL step 3, |residual|_2 = 4.494531e-02 > Line search: gnorm after quadratic fit 5.415371014813e-02 > Line search: Cubically determined step, current gnorm > 4.392165909219e-02 lambda=3.6375617606865668e-02 > NL step 4, |residual|_2 = 4.392166e-02 > Line search: gnorm after quadratic fit 4.631663907262e-02 > Line search: Cubically determined step, current gnorm > 4.246200768767e-02 lambda=5.0000000000000003e-02 > NL step 5, |residual|_2 = 4.246201e-02 > Line search: gnorm after quadratic fit 4.222105256158e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 6, |residual|_2 = 4.222105e-02 > Line search: gnorm after quadratic fit 4.026081168915e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 7, |residual|_2 = 4.026081e-02 > Line search: gnorm after quadratic fit 3.776439443011e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 8, |residual|_2 = 3.776439e-02 > Line search: gnorm after quadratic fit 3.659796213553e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 9, |residual|_2 = 3.659796e-02 > Line search: gnorm after quadratic fit 3.423207563496e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 10, |residual|_2 = 3.423208e-02 > Line search: gnorm after quadratic fit 3.116928356075e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 11, |residual|_2 = 3.116928e-02 > Line search: gnorm after quadratic fit 2.874310673331e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 12, |residual|_2 = 2.874311e-02 > Line search: gnorm after quadratic fit 2.587826447631e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 13, |residual|_2 = 2.587826e-02 > Line search: gnorm after quadratic fit 2.344160918669e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 14, |residual|_2 = 2.344161e-02 > Line search: gnorm after quadratic fit 2.187719801063e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 15, |residual|_2 = 2.187720e-02 > Line search: gnorm after quadratic fit 1.983089025936e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 16, |residual|_2 = 1.983089e-02 > Line search: gnorm after quadratic fit 1.791227696650e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 17, |residual|_2 = 1.791228e-02 > Line search: gnorm after quadratic fit 1.613250592206e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 18, |residual|_2 = 1.613251e-02 > Line search: gnorm after quadratic fit 1.455841890804e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 19, |residual|_2 = 1.455842e-02 > Line search: gnorm after quadratic fit 1.321849665170e-02 > Line search: Quadratically determined step, > lambda=1.0574900347563776e-01 > NL step 20, |residual|_2 = 1.321850e-02 > Line search: gnorm after quadratic fit 9.209642717528e-03 > Line search: Quadratically determined step, > lambda=3.0589679103560180e-01 > NL step 21, |residual|_2 = 9.209643e-03 > Line search: gnorm after quadratic fit 7.590944125425e-03 > Line search: Quadratically determined step, > lambda=2.0920307644146574e-01 > NL step 22, |residual|_2 = 7.590944e-03 > Line search: gnorm after quadratic fit 4.373921456388e-03 > Line search: Quadratically determined step, > lambda=4.2379743756255861e-01 > NL step 23, |residual|_2 = 4.373921e-03 > Line search: gnorm after quadratic fit 3.681355014898e-03 > Line search: Quadratically determined step, > lambda=1.9626628361883081e-01 > NL step 24, |residual|_2 = 3.681355e-03 > Line search: gnorm after quadratic fit 2.594785108727e-03 > Line search: Quadratically determined step, > lambda=3.8057573229158653e-01 > NL step 25, |residual|_2 = 2.594785e-03 > Line search: gnorm after quadratic fit 1.803191839408e-03 > Line search: Quadratically determined step, > lambda=4.3574150080610474e-01 > NL step 26, |residual|_2 = 1.803192e-03 > Line search: Using full step: fnorm 1.803191839408e-03 gnorm > 9.015954497317e-04 > NL step 27, |residual|_2 = 9.015954e-04 > Line search: Using full step: fnorm 9.015954497317e-04 gnorm > 1.390181456520e-13 > NL step 28, |residual|_2 = 1.390181e-13 > Number of nonlinear iterations: 28 > > > On Wed, Jan 13, 2016 at 3:48 PM, David Knezevic <david.kneze...@akselos.com> > wrote: > I'm using NEWTONLS (with mumps for the linear solves) to do a nonlinear PDE > solve. It converges well when I use 1 core. When I use 2 or more cores, the > line search stagnates. I've pasted the output of -snes_linesearch_monitor > below in these two cases. > > I was wondering if this implies that I must have a bug in parallel, or if > perhaps the NEWTONLS solver can behave slightly differently in parallel? > > Thanks, > David > > --------------------------------------------------------------------------------------- > > > > Parallel case: > NL step 0, |residual|_2 = 4.714515e-02 > Line search: gnorm after quadratic fit 7.862867755323e-02 > Line search: Cubically determined step, current gnorm > 4.663945043239e-02 lambda=1.4276549921126183e-02 > NL step 1, |residual|_2 = 4.663945e-02 > Line search: gnorm after quadratic fit 6.977268575068e-02 > Line search: Cubically determined step, current gnorm > 4.594912794004e-02 lambda=2.3644825912085998e-02 > NL step 2, |residual|_2 = 4.594913e-02 > Line search: gnorm after quadratic fit 5.502067932478e-02 > Line search: Cubically determined step, current gnorm > 4.494531294405e-02 lambda=4.1260497615261321e-02 > NL step 3, |residual|_2 = 4.494531e-02 > Line search: gnorm after quadratic fit 5.415371063247e-02 > Line search: Cubically determined step, current gnorm > 4.392165925471e-02 lambda=3.6375618871780056e-02 > NL step 4, |residual|_2 = 4.392166e-02 > Line search: gnorm after quadratic fit 4.631663976615e-02 > Line search: Cubically determined step, current gnorm > 4.246200798775e-02 lambda=5.0000000000000003e-02 > NL step 5, |residual|_2 = 4.246201e-02 > Line search: gnorm after quadratic fit 4.222105321728e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 6, |residual|_2 = 4.222105e-02 > Line search: gnorm after quadratic fit 4.026081251872e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 7, |residual|_2 = 4.026081e-02 > Line search: gnorm after quadratic fit 3.776439532346e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 8, |residual|_2 = 3.776440e-02 > Line search: gnorm after quadratic fit 3.659796311121e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 9, |residual|_2 = 3.659796e-02 > Line search: gnorm after quadratic fit 3.423207664901e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 10, |residual|_2 = 3.423208e-02 > Line search: gnorm after quadratic fit 3.116928452225e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 11, |residual|_2 = 3.116928e-02 > Line search: gnorm after quadratic fit 2.874310955274e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 12, |residual|_2 = 2.874311e-02 > Line search: gnorm after quadratic fit 2.587826662305e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 13, |residual|_2 = 2.587827e-02 > Line search: gnorm after quadratic fit 2.344161073075e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 14, |residual|_2 = 2.344161e-02 > Line search: gnorm after quadratic fit 2.187719889554e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 15, |residual|_2 = 2.187720e-02 > Line search: gnorm after quadratic fit 1.983089075086e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 16, |residual|_2 = 1.983089e-02 > Line search: gnorm after quadratic fit 1.791227711151e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 17, |residual|_2 = 1.791228e-02 > Line search: gnorm after quadratic fit 1.613250573900e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 18, |residual|_2 = 1.613251e-02 > Line search: gnorm after quadratic fit 1.455841843183e-02 > Line search: Quadratically determined step, > lambda=1.0000000000000001e-01 > NL step 19, |residual|_2 = 1.455842e-02 > Line search: gnorm after quadratic fit 1.321849780208e-02 > Line search: Quadratically determined step, > lambda=1.0574876450981290e-01 > NL step 20, |residual|_2 = 1.321850e-02 > Line search: gnorm after quadratic fit 9.209641609489e-03 > Line search: Quadratically determined step, > lambda=3.0589684959139674e-01 > NL step 21, |residual|_2 = 9.209642e-03 > Line search: gnorm after quadratic fit 7.590942028574e-03 > Line search: Quadratically determined step, > lambda=2.0920305898507460e-01 > NL step 22, |residual|_2 = 7.590942e-03 > Line search: gnorm after quadratic fit 4.373918927227e-03 > Line search: Quadratically determined step, > lambda=4.2379743128074154e-01 > NL step 23, |residual|_2 = 4.373919e-03 > Line search: gnorm after quadratic fit 3.681351665911e-03 > Line search: Quadratically determined step, > lambda=1.9626618428089049e-01 > NL step 24, |residual|_2 = 3.681352e-03 > Line search: gnorm after quadratic fit 2.594782418891e-03 > Line search: Quadratically determined step, > lambda=3.8057533372167579e-01 > NL step 25, |residual|_2 = 2.594782e-03 > Line search: gnorm after quadratic fit 1.803188279452e-03 > Line search: Quadratically determined step, > lambda=4.3574109448916826e-01 > NL step 26, |residual|_2 = 1.803188e-03 > Line search: Using full step: fnorm 1.803188279452e-03 gnorm > 9.015947319176e-04 > NL step 27, |residual|_2 = 9.015947e-04 > Line search: Using full step: fnorm 9.015947319176e-04 gnorm > 7.088879385731e-08 > NL step 28, |residual|_2 = 7.088879e-08 > Line search: gnorm after quadratic fit 7.088878906502e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088878957116e-08 lambda=2.1132490715284968e-01 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385683e-08 lambda=9.2196195824189087e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385711e-08 lambda=4.0004532931495446e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385722e-08 lambda=1.7374764617622523e-02 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385726e-08 lambda=7.5449542135114234e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=3.2764749100364717e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.4228361655588414e-03 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=6.1787884492365153e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=2.6831916265377548e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.1651988987471248e-04 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=5.0599757911789984e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=2.1973377296845284e-05 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=9.5421268734746417e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=4.1437501409853001e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.7994589108402447e-06 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=7.8143041004756041e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=3.3934283359762142e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.4736252548330828e-07 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=6.3993436038104693e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=2.7789696481734489e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.2067913185456743e-08 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=5.2405944320925838e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=2.2757729177880525e-09 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=9.8827383810151057e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=4.2916635989551390e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.8636915940199893e-10 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=8.0932400164504977e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=3.5145586412497970e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.5262271250668997e-11 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=6.6277717206096633e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=2.8781665100197773e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=1.2498684035299616e-12 > Line search: Cubic step no good, shrinking lambda, current gnorm > 7.088879385731e-08 lambda=5.4276603549660526e-13 > Line search: unable to find good step length! After 33 tries > Line search: fnorm=7.0888793857309783e-08, > gnorm=7.0888793857309783e-08, ynorm=2.4650076775058285e-08, > minlambda=9.9999999999999998e-13, lambda=5.4276603549660526e-13, initial > slope=-5.0252210945441613e-15 > >