Re: [pymvpa] Connecitvity Hyperalignment and SeedQuerryEngine
On Mon, 15 Oct 2018, Vincent Taschereau-Dumouchel wrote:
> Thanks a lot for your help!
> It seems that for now we have to specify a seed_queryEngine because of this
> in the connectivity_hyperalignment script:
> if params.seed_queryengines is None:
> raise NotImplementedError("For now, we need seed queryengines.”)
> But if we define an empty seed_queryEngine and use a seed_radius of 20 it
> seems to work. Just like Feilong suggested it will be defined in
> get_trained_queryengines.
> One last question if I may: it seems that I always run out of RAM when I try
> to run the script. Even if I use only 10 resting state dataset of 10 minutes
> (~ 1GB) it will end up crashing after 24 hours because it ran out of the
> ~50GB I allocated for the analysis.
I guess without specifying seed_indices pretty much all of your
voxels/nodes are used as seeds, which leads to huge connectivity
matrices. Feilong - am I right? is there a way to just say to evenly
sample the space?
> Is that normal and do you guys have a rule of thumbs to determine how many
> RAM the analysis will require? I might have to find another environment to
> run my analyses as a function of what this requires.
in general to track RAM consumption you could use following env vars:
MVPA_DEBUG=CHPAL -- to enable debug output from connectivity hyperalignment
MVPA_DEBUG_METRICS=vmem -- to also include current memory consumption
this way you could see between which steps memory consumption grows,
and possibly get a clue what to tune etc. You might actually make
it
MVPA_DEBUG=.*HPAL,SLC for more relevant "debug targets"
--
Yaroslav O. Halchenko
Center for Open Neuroscience http://centerforopenneuroscience.org
Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
WWW: http://www.linkedin.com/in/yarik
___
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] Connecitvity Hyperalignment and SeedQuerryEngine
Hi Vincent,
You probably want to use sparse searchlights to compute the connectivity
targets by specifying center_ids. For example, instead of centering a
searchlight around each voxel, you pick only 10% of the searchlights that
are evenly distributed across the brain. This will make the connectivity
matrices much smaller, and potentially help you fit everything in the RAM.
Note that that's another reason we want to have a separate SeedQueryEngine
-- to make sure the number of connectivity targets are within a proper
range, and so that we can afford the computations.
Projecting your data onto surface will also help, with the same resolution
data on surface are also smaller in size than data in volume.
Best,
Feilong
On Mon, Oct 15, 2018 at 9:35 AM Vincent Taschereau-Dumouchel <
vincen...@ucla.edu> wrote:
> Thanks a lot for your help!
>
> It seems that for now we have to specify a seed_queryEngine because of
> this in the connectivity_hyperalignment script:
>
> if params.seed_queryengines is None:
> raise NotImplementedError("For now, we need seed
> queryengines.”)
>
> But if we define an empty seed_queryEngine and use a seed_radius of 20 it
> seems to work. Just like Feilong suggested it will be defined in
> get_trained_queryengines.
>
> One last question if I may: it seems that I always run out of RAM when I
> try to run the script. Even if I use only 10 resting state dataset of 10
> minutes (~ 1GB) it will end up crashing after 24 hours because it ran out
> of the ~50GB I allocated for the analysis.
>
> Is that normal and do you guys have a rule of thumbs to determine how many
> RAM the analysis will require? I might have to find another environment to
> run my analyses as a function of what this requires.
>
> Thanks a lot!
>
> Vincent
>
> On Oct 12, 2018, at 3:52 PM, Feilong Ma wrote:
>
> Hi Vincent,
>
> The main query engine for searchlight connectivity hyperalignment should
> be similar to that used for searchlight response hyperalignment, and we
> usually use a 20 mm radius, which is the same one used in Swaroop's 2016
> searchlight hyperalignment paper. Sometimes I use a slightly smaller
> radius (e.g., 15 mm, 10 mm) to speed up the computations, but in general 20
> mm radius usually works well.
>
> The other query engine (SeedQueryEngine) is specific for connectivity
> hyperalignment, and is used to compute connectivity targets (sometimes also
> called connectivity seeds) by averaging responses in each searchlight. In
> the connectivity hyperalignment paper we used connectivity seeds from a
> low-resolution surface (AFNI's ico8), and that in combination with a 13 mm
> radius can make sure the entire cortical surface is covered by the
> searchlights.
>
> If you want to perform a quick analysis with data in volume, you can use a
> query engine that will generate searchlights in volume space. Below is an
> example usage
>
> https://github.com/PyMVPA/PyMVPA/blob/5b01da5529c8653da948e9e03c9361168d954482/mvpa2/algorithms/searchlight_hyperalignment.py#L683-L685
>
> Best,
> Feilong
>
> On Fri, Oct 12, 2018 at 11:39 AM Yaroslav Halchenko
> wrote:
>
>>
>> On Fri, 12 Oct 2018, Vincent Taschereau-Dumouchel wrote:
>>
>> > Hi everyone,
>>
>> > First of, thanks a lot for sharing the connectivity hyperalignment
>> code. You guys are doing awesome work.
>> > I am starting to play around with it and was wondering about 2 things:
>>
>> > - Do you have an example of the arguments that needs to be provided?
>> One of them is the SeedQueryEngine. I went through the code a little and I
>> am having a hard time determining what this is exactly.
>>
>> seed_queryengines are needed for figuring out neighborhood to average
>> for estimating connectivity to it. I am not quite sure (CCing the
>> original author Swaroop and Feilong who might know better) why your
>> "main" queryengine could/shouldn't be used (the test uses the
>> same) if you are working in the same "domain" (e.g. in the volume for
>> both the searchlight and then the seeds). If so, may be code could be
>> adjusted to avoid demanding specification of the seed_queryengines?
>>
>> > - Also, can we use it not with surfaces but with typical volume data? I
>> understand the problems associated with running a searchlight on volume
>> data but I was just trying to take a quick peak at the function before
>> processing surfaces and all.
>>
>> AFAIK, due to all those "query engines" flexibility, it could be used in
>> a variety of scenarios:
>>
>> - everything in the volume
>> - everything (projected) on the surface
>> - having surface guiding selection of the voxels in the volume
>>
>>
>> --
>> Yaroslav O. Halchenko
>> Center for Open Neuroscience http://centerforopenneuroscience.org
>> Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755
>> Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419
>> WWW: http://www.linkedin.com/in/yarik
>>
>
>
___
Pkg-ExpPs
Re: [pymvpa] Connecitvity Hyperalignment and SeedQuerryEngine
Thanks a lot for your help!
It seems that for now we have to specify a seed_queryEngine because of this in
the connectivity_hyperalignment script:
if params.seed_queryengines is None:
raise NotImplementedError("For now, we need seed queryengines.”)
But if we define an empty seed_queryEngine and use a seed_radius of 20 it seems
to work. Just like Feilong suggested it will be defined in
get_trained_queryengines.
One last question if I may: it seems that I always run out of RAM when I try to
run the script. Even if I use only 10 resting state dataset of 10 minutes (~
1GB) it will end up crashing after 24 hours because it ran out of the ~50GB I
allocated for the analysis.
Is that normal and do you guys have a rule of thumbs to determine how many RAM
the analysis will require? I might have to find another environment to run my
analyses as a function of what this requires.
Thanks a lot!
Vincent
> On Oct 12, 2018, at 3:52 PM, Feilong Ma wrote:
>
> Hi Vincent,
>
> The main query engine for searchlight connectivity hyperalignment should be
> similar to that used for searchlight response hyperalignment, and we usually
> use a 20 mm radius, which is the same one used in Swaroop's 2016 searchlight
> hyperalignment paper. Sometimes I use a slightly smaller radius (e.g., 15
> mm, 10 mm) to speed up the computations, but in general 20 mm radius usually
> works well.
>
> The other query engine (SeedQueryEngine) is specific for connectivity
> hyperalignment, and is used to compute connectivity targets (sometimes also
> called connectivity seeds) by averaging responses in each searchlight. In
> the connectivity hyperalignment paper we used connectivity seeds from a
> low-resolution surface (AFNI's ico8), and that in combination with a 13 mm
> radius can make sure the entire cortical surface is covered by the
> searchlights.
>
> If you want to perform a quick analysis with data in volume, you can use a
> query engine that will generate searchlights in volume space. Below is an
> example usage
> https://github.com/PyMVPA/PyMVPA/blob/5b01da5529c8653da948e9e03c9361168d954482/mvpa2/algorithms/searchlight_hyperalignment.py#L683-L685
>
>
>
> Best,
> Feilong
>
> On Fri, Oct 12, 2018 at 11:39 AM Yaroslav Halchenko > wrote:
>
> On Fri, 12 Oct 2018, Vincent Taschereau-Dumouchel wrote:
>
> > Hi everyone,
>
> > First of, thanks a lot for sharing the connectivity hyperalignment code.
> > You guys are doing awesome work.
> > I am starting to play around with it and was wondering about 2 things:
>
> > - Do you have an example of the arguments that needs to be provided? One of
> > them is the SeedQueryEngine. I went through the code a little and I am
> > having a hard time determining what this is exactly.
>
> seed_queryengines are needed for figuring out neighborhood to average
> for estimating connectivity to it. I am not quite sure (CCing the
> original author Swaroop and Feilong who might know better) why your
> "main" queryengine could/shouldn't be used (the test uses the
> same) if you are working in the same "domain" (e.g. in the volume for
> both the searchlight and then the seeds). If so, may be code could be
> adjusted to avoid demanding specification of the seed_queryengines?
>
> > - Also, can we use it not with surfaces but with typical volume data? I
> > understand the problems associated with running a searchlight on volume
> > data but I was just trying to take a quick peak at the function before
> > processing surfaces and all.
>
> AFAIK, due to all those "query engines" flexibility, it could be used in
> a variety of scenarios:
>
> - everything in the volume
> - everything (projected) on the surface
> - having surface guiding selection of the voxels in the volume
>
>
> --
> Yaroslav O. Halchenko
> Center for Open Neuroscience http://centerforopenneuroscience.org
>
>
___
Pkg-ExpPsy-PyMVPA mailing list
Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net
https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] Connecitvity Hyperalignment and SeedQuerryEngine
Hi Vincent, The main query engine for searchlight connectivity hyperalignment should be similar to that used for searchlight response hyperalignment, and we usually use a 20 mm radius, which is the same one used in Swaroop's 2016 searchlight hyperalignment paper. Sometimes I use a slightly smaller radius (e.g., 15 mm, 10 mm) to speed up the computations, but in general 20 mm radius usually works well. The other query engine (SeedQueryEngine) is specific for connectivity hyperalignment, and is used to compute connectivity targets (sometimes also called connectivity seeds) by averaging responses in each searchlight. In the connectivity hyperalignment paper we used connectivity seeds from a low-resolution surface (AFNI's ico8), and that in combination with a 13 mm radius can make sure the entire cortical surface is covered by the searchlights. If you want to perform a quick analysis with data in volume, you can use a query engine that will generate searchlights in volume space. Below is an example usage https://github.com/PyMVPA/PyMVPA/blob/5b01da5529c8653da948e9e03c9361168d954482/mvpa2/algorithms/searchlight_hyperalignment.py#L683-L685 Best, Feilong On Fri, Oct 12, 2018 at 11:39 AM Yaroslav Halchenko wrote: > > On Fri, 12 Oct 2018, Vincent Taschereau-Dumouchel wrote: > > > Hi everyone, > > > First of, thanks a lot for sharing the connectivity hyperalignment code. > You guys are doing awesome work. > > I am starting to play around with it and was wondering about 2 things: > > > - Do you have an example of the arguments that needs to be provided? One > of them is the SeedQueryEngine. I went through the code a little and I am > having a hard time determining what this is exactly. > > seed_queryengines are needed for figuring out neighborhood to average > for estimating connectivity to it. I am not quite sure (CCing the > original author Swaroop and Feilong who might know better) why your > "main" queryengine could/shouldn't be used (the test uses the > same) if you are working in the same "domain" (e.g. in the volume for > both the searchlight and then the seeds). If so, may be code could be > adjusted to avoid demanding specification of the seed_queryengines? > > > - Also, can we use it not with surfaces but with typical volume data? I > understand the problems associated with running a searchlight on volume > data but I was just trying to take a quick peak at the function before > processing surfaces and all. > > AFAIK, due to all those "query engines" flexibility, it could be used in > a variety of scenarios: > > - everything in the volume > - everything (projected) on the surface > - having surface guiding selection of the voxels in the volume > > > -- > Yaroslav O. Halchenko > Center for Open Neuroscience http://centerforopenneuroscience.org > Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 > Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 > WWW: http://www.linkedin.com/in/yarik > ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
Re: [pymvpa] Connecitvity Hyperalignment and SeedQuerryEngine
On Fri, 12 Oct 2018, Vincent Taschereau-Dumouchel wrote: > Hi everyone, > First of, thanks a lot for sharing the connectivity hyperalignment code. You > guys are doing awesome work. > I am starting to play around with it and was wondering about 2 things: > - Do you have an example of the arguments that needs to be provided? One of > them is the SeedQueryEngine. I went through the code a little and I am having > a hard time determining what this is exactly. seed_queryengines are needed for figuring out neighborhood to average for estimating connectivity to it. I am not quite sure (CCing the original author Swaroop and Feilong who might know better) why your "main" queryengine could/shouldn't be used (the test uses the same) if you are working in the same "domain" (e.g. in the volume for both the searchlight and then the seeds). If so, may be code could be adjusted to avoid demanding specification of the seed_queryengines? > - Also, can we use it not with surfaces but with typical volume data? I > understand the problems associated with running a searchlight on volume data > but I was just trying to take a quick peak at the function before processing > surfaces and all. AFAIK, due to all those "query engines" flexibility, it could be used in a variety of scenarios: - everything in the volume - everything (projected) on the surface - having surface guiding selection of the voxels in the volume -- Yaroslav O. Halchenko Center for Open Neuroscience http://centerforopenneuroscience.org Dartmouth College, 419 Moore Hall, Hinman Box 6207, Hanover, NH 03755 Phone: +1 (603) 646-9834 Fax: +1 (603) 646-1419 WWW: http://www.linkedin.com/in/yarik ___ Pkg-ExpPsy-PyMVPA mailing list Pkg-ExpPsy-PyMVPA@alioth-lists.debian.net https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-exppsy-pymvpa
