i write a python script which has itertools.combinations(initlist, 2) but it got error when number of elements in initlist over 14,000
is it possible to use spark to do this work? i have seen yatel can do this, is spark and yatel using hard disk as memory? if so, which need to change in python code ?