Re: [R] Warning messages while parallel computing
Perhaps just try
source("xx.R")
invisible(gc()) # do warnings appear after this?
It does seem likely that the sourced file causes the problem, since
parallel::createCluster labels the connections ":" and that label would show up in the warning.
On Thu, Mar 4, 2021 at 11:28 AM Henrik Bengtsson
wrote:
>
> Test with:
>
> clusterCall(cl, function() { suppressWarnings(source("xx.R")) })
>
> If the warnings disappear, then the warnings are produced on the
> workers from source():ing the file.
>
> /Henrik
>
> On Thu, Mar 4, 2021 at 10:20 AM Bill Dunlap wrote:
> >
> > The warnings come from the garbage collector, which may be called from
> > almost anywhere. It is possible that the file that is sourced causes
> > the problem, but if you don't call parallel::stopCluster before
> > removing the cluster object you will get those warnings.
> >
> > > cl <- parallel::makeCluster(3, type="PSOCK")
> > > invisible(gc())
> > > rm(cl)
> > > invisible(gc())
> > Warning messages:
> > 1: In .Internal(gc(verbose, reset, full)) :
> > closing unused connection 6 (<-Bill-T490:11216)
> > 2: In .Internal(gc(verbose, reset, full)) :
> > closing unused connection 5 (<-Bill-T490:11216)
> > 3: In .Internal(gc(verbose, reset, full)) :
> > closing unused connection 4 (<-Bill-T490:11216)
> > >
> > >
> > > cl <- parallel::makeCluster(3, type="PSOCK")
> > > invisible(gc())
> > > parallel::stopCluster(cl)
> > > invisible(gc())
> > > rm(cl)
> > > invisible(gc())
> > >
> >
> > The fact that he got 8 warnings when playing a cluster of size 8 makes
> > me suspect that omitting stopCluster is the problem.
> >
> > -Bill
> >
> > On Thu, Mar 4, 2021 at 10:12 AM Henrik Bengtsson
> > wrote:
> > >
> > > I don't think 'parallel' is to blame in this case. Those warnings:
> > >
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 19
> > >
> > > come from base::source()
> > > [https://github.com/wch/r-source/blob/9caddc1eaad1f480283f1e98af34a328699d1869/src/library/base/R/source.R#L166-L244].
> > >
> > > Unless there's a bug in source() that leaves connections open, which
> > > is unlikely, I think there's something in the 'xx.R' script that opens
> > > a connection but doesn't close it. Possibly multiple times. A good
> > > check is to see if the same warnings are produced when calling
> > > source("xx.R") sequentially in a for() loop or an lapply() call.
> > >
> > > Hope this helps,
> > >
> > > Henrik
> > >
> > > On Thu, Mar 4, 2021 at 9:58 AM Bill Dunlap
> > > wrote:
> > > >
> > > > To avoid the warnings from gc(), call parallel::stopCluster(cl) before
> > > > removing or overwriting cl.
> > > >
> > > > -Bill
> > > >
> > > > On Thu, Mar 4, 2021 at 1:52 AM Shah Alam
> > > > wrote:
> > > > >
> > > > > Hello everyone,
> > > > >
> > > > > I am using the "parallel" R package for parallel computation.
> > > > >
> > > > > Code:
> > > > >
> > > > ># set number of cores
> > > > > cl <- makeCluster(8, type = "PSOCK") # Mac/Linux need to set as
> > > > > "FORK"
> > > > >
> > > > > # pass functions and objects to the cluster environment and set
> > > > > seed
> > > > > # all the items exported need to stay in the global
> > > > > environment!!
> > > > > clusterCall(cl, function() { source("xx.R" )})
> > > > > clusterExport(cl, list("a", "b", "c", "d",
> > > > > "5"))
> > > > > clusterSetRNGStream(cl, 1)
> > > > >
> > > > > While parallel processing, I receive the following warning signs. Do
> > > > > I
> > > > > need to ignore these signs or do they potentially slow the whole
> > > > > process?
> > > > >
> > > > > * Warning signs:*
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 19
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 18
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 17
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 16
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 15
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 14
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 13
> > > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > > closing unused connection 12
> > > > >
> > > > > Best regards,
> > > > > Shah Alam
> > > > >
> > > > > [[alternative HTML version deleted]]
> > > > >
> > > > > __
> > > > > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > > PLEASE do read the posting guide
> > > > > http://www.R-project.org/posting-guide.html
> > > > > and provide commented, minimal, self-contained, reproducible code.
> > > >
> > > > __
> > > >
Re: [R] Warning messages while parallel computing
Test with:
clusterCall(cl, function() { suppressWarnings(source("xx.R")) })
If the warnings disappear, then the warnings are produced on the
workers from source():ing the file.
/Henrik
On Thu, Mar 4, 2021 at 10:20 AM Bill Dunlap wrote:
>
> The warnings come from the garbage collector, which may be called from
> almost anywhere. It is possible that the file that is sourced causes
> the problem, but if you don't call parallel::stopCluster before
> removing the cluster object you will get those warnings.
>
> > cl <- parallel::makeCluster(3, type="PSOCK")
> > invisible(gc())
> > rm(cl)
> > invisible(gc())
> Warning messages:
> 1: In .Internal(gc(verbose, reset, full)) :
> closing unused connection 6 (<-Bill-T490:11216)
> 2: In .Internal(gc(verbose, reset, full)) :
> closing unused connection 5 (<-Bill-T490:11216)
> 3: In .Internal(gc(verbose, reset, full)) :
> closing unused connection 4 (<-Bill-T490:11216)
> >
> >
> > cl <- parallel::makeCluster(3, type="PSOCK")
> > invisible(gc())
> > parallel::stopCluster(cl)
> > invisible(gc())
> > rm(cl)
> > invisible(gc())
> >
>
> The fact that he got 8 warnings when playing a cluster of size 8 makes
> me suspect that omitting stopCluster is the problem.
>
> -Bill
>
> On Thu, Mar 4, 2021 at 10:12 AM Henrik Bengtsson
> wrote:
> >
> > I don't think 'parallel' is to blame in this case. Those warnings:
> >
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 19
> >
> > come from base::source()
> > [https://github.com/wch/r-source/blob/9caddc1eaad1f480283f1e98af34a328699d1869/src/library/base/R/source.R#L166-L244].
> >
> > Unless there's a bug in source() that leaves connections open, which
> > is unlikely, I think there's something in the 'xx.R' script that opens
> > a connection but doesn't close it. Possibly multiple times. A good
> > check is to see if the same warnings are produced when calling
> > source("xx.R") sequentially in a for() loop or an lapply() call.
> >
> > Hope this helps,
> >
> > Henrik
> >
> > On Thu, Mar 4, 2021 at 9:58 AM Bill Dunlap wrote:
> > >
> > > To avoid the warnings from gc(), call parallel::stopCluster(cl) before
> > > removing or overwriting cl.
> > >
> > > -Bill
> > >
> > > On Thu, Mar 4, 2021 at 1:52 AM Shah Alam wrote:
> > > >
> > > > Hello everyone,
> > > >
> > > > I am using the "parallel" R package for parallel computation.
> > > >
> > > > Code:
> > > >
> > > ># set number of cores
> > > > cl <- makeCluster(8, type = "PSOCK") # Mac/Linux need to set as
> > > > "FORK"
> > > >
> > > > # pass functions and objects to the cluster environment and set
> > > > seed
> > > > # all the items exported need to stay in the global environment!!
> > > > clusterCall(cl, function() { source("xx.R" )})
> > > > clusterExport(cl, list("a", "b", "c", "d",
> > > > "5"))
> > > > clusterSetRNGStream(cl, 1)
> > > >
> > > > While parallel processing, I receive the following warning signs. Do I
> > > > need to ignore these signs or do they potentially slow the whole
> > > > process?
> > > >
> > > > * Warning signs:*
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 19
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 18
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 17
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 16
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 15
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 14
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 13
> > > > Warning in for (i in seq_len(Ne + echo)) { :
> > > > closing unused connection 12
> > > >
> > > > Best regards,
> > > > Shah Alam
> > > >
> > > > [[alternative HTML version deleted]]
> > > >
> > > > __
> > > > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > > PLEASE do read the posting guide
> > > > http://www.R-project.org/posting-guide.html
> > > > and provide commented, minimal, self-contained, reproducible code.
> > >
> > > __
> > > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide
> > > http://www.R-project.org/posting-guide.html
> > > and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Warning messages while parallel computing
The warnings come from the garbage collector, which may be called from
almost anywhere. It is possible that the file that is sourced causes
the problem, but if you don't call parallel::stopCluster before
removing the cluster object you will get those warnings.
> cl <- parallel::makeCluster(3, type="PSOCK")
> invisible(gc())
> rm(cl)
> invisible(gc())
Warning messages:
1: In .Internal(gc(verbose, reset, full)) :
closing unused connection 6 (<-Bill-T490:11216)
2: In .Internal(gc(verbose, reset, full)) :
closing unused connection 5 (<-Bill-T490:11216)
3: In .Internal(gc(verbose, reset, full)) :
closing unused connection 4 (<-Bill-T490:11216)
>
>
> cl <- parallel::makeCluster(3, type="PSOCK")
> invisible(gc())
> parallel::stopCluster(cl)
> invisible(gc())
> rm(cl)
> invisible(gc())
>
The fact that he got 8 warnings when playing a cluster of size 8 makes
me suspect that omitting stopCluster is the problem.
-Bill
On Thu, Mar 4, 2021 at 10:12 AM Henrik Bengtsson
wrote:
>
> I don't think 'parallel' is to blame in this case. Those warnings:
>
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 19
>
> come from base::source()
> [https://github.com/wch/r-source/blob/9caddc1eaad1f480283f1e98af34a328699d1869/src/library/base/R/source.R#L166-L244].
>
> Unless there's a bug in source() that leaves connections open, which
> is unlikely, I think there's something in the 'xx.R' script that opens
> a connection but doesn't close it. Possibly multiple times. A good
> check is to see if the same warnings are produced when calling
> source("xx.R") sequentially in a for() loop or an lapply() call.
>
> Hope this helps,
>
> Henrik
>
> On Thu, Mar 4, 2021 at 9:58 AM Bill Dunlap wrote:
> >
> > To avoid the warnings from gc(), call parallel::stopCluster(cl) before
> > removing or overwriting cl.
> >
> > -Bill
> >
> > On Thu, Mar 4, 2021 at 1:52 AM Shah Alam wrote:
> > >
> > > Hello everyone,
> > >
> > > I am using the "parallel" R package for parallel computation.
> > >
> > > Code:
> > >
> > ># set number of cores
> > > cl <- makeCluster(8, type = "PSOCK") # Mac/Linux need to set as
> > > "FORK"
> > >
> > > # pass functions and objects to the cluster environment and set seed
> > > # all the items exported need to stay in the global environment!!
> > > clusterCall(cl, function() { source("xx.R" )})
> > > clusterExport(cl, list("a", "b", "c", "d",
> > > "5"))
> > > clusterSetRNGStream(cl, 1)
> > >
> > > While parallel processing, I receive the following warning signs. Do I
> > > need to ignore these signs or do they potentially slow the whole process?
> > >
> > > * Warning signs:*
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 19
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 18
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 17
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 16
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 15
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 14
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 13
> > > Warning in for (i in seq_len(Ne + echo)) { :
> > > closing unused connection 12
> > >
> > > Best regards,
> > > Shah Alam
> > >
> > > [[alternative HTML version deleted]]
> > >
> > > __
> > > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > > https://stat.ethz.ch/mailman/listinfo/r-help
> > > PLEASE do read the posting guide
> > > http://www.R-project.org/posting-guide.html
> > > and provide commented, minimal, self-contained, reproducible code.
> >
> > __
> > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Warning messages while parallel computing
I don't think 'parallel' is to blame in this case. Those warnings:
Warning in for (i in seq_len(Ne + echo)) { :
closing unused connection 19
come from base::source()
[https://github.com/wch/r-source/blob/9caddc1eaad1f480283f1e98af34a328699d1869/src/library/base/R/source.R#L166-L244].
Unless there's a bug in source() that leaves connections open, which
is unlikely, I think there's something in the 'xx.R' script that opens
a connection but doesn't close it. Possibly multiple times. A good
check is to see if the same warnings are produced when calling
source("xx.R") sequentially in a for() loop or an lapply() call.
Hope this helps,
Henrik
On Thu, Mar 4, 2021 at 9:58 AM Bill Dunlap wrote:
>
> To avoid the warnings from gc(), call parallel::stopCluster(cl) before
> removing or overwriting cl.
>
> -Bill
>
> On Thu, Mar 4, 2021 at 1:52 AM Shah Alam wrote:
> >
> > Hello everyone,
> >
> > I am using the "parallel" R package for parallel computation.
> >
> > Code:
> >
> ># set number of cores
> > cl <- makeCluster(8, type = "PSOCK") # Mac/Linux need to set as "FORK"
> >
> > # pass functions and objects to the cluster environment and set seed
> > # all the items exported need to stay in the global environment!!
> > clusterCall(cl, function() { source("xx.R" )})
> > clusterExport(cl, list("a", "b", "c", "d",
> > "5"))
> > clusterSetRNGStream(cl, 1)
> >
> > While parallel processing, I receive the following warning signs. Do I
> > need to ignore these signs or do they potentially slow the whole process?
> >
> > * Warning signs:*
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 19
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 18
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 17
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 16
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 15
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 14
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 13
> > Warning in for (i in seq_len(Ne + echo)) { :
> > closing unused connection 12
> >
> > Best regards,
> > Shah Alam
> >
> > [[alternative HTML version deleted]]
> >
> > __
> > R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> > https://stat.ethz.ch/mailman/listinfo/r-help
> > PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> > and provide commented, minimal, self-contained, reproducible code.
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Warning messages while parallel computing
To avoid the warnings from gc(), call parallel::stopCluster(cl) before
removing or overwriting cl.
-Bill
On Thu, Mar 4, 2021 at 1:52 AM Shah Alam wrote:
>
> Hello everyone,
>
> I am using the "parallel" R package for parallel computation.
>
> Code:
>
># set number of cores
> cl <- makeCluster(8, type = "PSOCK") # Mac/Linux need to set as "FORK"
>
> # pass functions and objects to the cluster environment and set seed
> # all the items exported need to stay in the global environment!!
> clusterCall(cl, function() { source("xx.R" )})
> clusterExport(cl, list("a", "b", "c", "d",
> "5"))
> clusterSetRNGStream(cl, 1)
>
> While parallel processing, I receive the following warning signs. Do I
> need to ignore these signs or do they potentially slow the whole process?
>
> * Warning signs:*
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 19
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 18
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 17
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 16
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 15
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 14
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 13
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 12
>
> Best regards,
> Shah Alam
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
Re: [R] Warning messages while parallel computing
Hi Shah,
The error usually means you started a cluster and forgot to close it. From
the code you post, I cannot see any problem. Maybe you run `makeCluster`
twice by accident?
Best,
Jiefei
On Thu, Mar 4, 2021 at 5:53 PM Shah Alam wrote:
> Hello everyone,
>
> I am using the "parallel" R package for parallel computation.
>
> Code:
>
># set number of cores
> cl <- makeCluster(8, type = "PSOCK") # Mac/Linux need to set as "FORK"
>
> # pass functions and objects to the cluster environment and set seed
> # all the items exported need to stay in the global environment!!
> clusterCall(cl, function() { source("xx.R" )})
> clusterExport(cl, list("a", "b", "c", "d",
> "5"))
> clusterSetRNGStream(cl, 1)
>
> While parallel processing, I receive the following warning signs. Do I
> need to ignore these signs or do they potentially slow the whole process?
>
> * Warning signs:*
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 19
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 18
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 17
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 16
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 15
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 14
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 13
> Warning in for (i in seq_len(Ne + echo)) { :
> closing unused connection 12
>
> Best regards,
> Shah Alam
>
> [[alternative HTML version deleted]]
>
> __
> R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
> https://stat.ethz.ch/mailman/listinfo/r-help
> PLEASE do read the posting guide
> http://www.R-project.org/posting-guide.html
> and provide commented, minimal, self-contained, reproducible code.
>
[[alternative HTML version deleted]]
__
R-help@r-project.org mailing list -- To UNSUBSCRIBE and more, see
https://stat.ethz.ch/mailman/listinfo/r-help
PLEASE do read the posting guide http://www.R-project.org/posting-guide.html
and provide commented, minimal, self-contained, reproducible code.
