Sure, here it is: data = np.zeros((10,8), dtype=np.uint8) out = pa.array(list(data)) out.type # ListType(list<item: uint8>)
data = np.zeros((3,4,6), dtype=np.uint8) out = pa.array(list(data)) # Throws error ArrowInvalid: Can only convert 1-dimensional array values Even though it's working on 2D numpy arrays perfectly, it doesn't work on N-Dimensional numpy arrays (where N > 2). Why is it so? On Thu, 11 Feb 2021 at 21:18, Wes McKinney <[email protected]> wrote: > Can you provide more detail about what you are trying? You've showed > some exception here but haven't showed the exact code that results in > those exceptions > > On Thu, Feb 11, 2021 at 4:34 AM Bhavitvya Malik > <[email protected]> wrote: > > > > Hi, > > It's a follow up question for #9462. Rewriting the issue here: > > > >> I came to know that pyarrow has this limitation of not storing > N-dimensional array. After looking into this issue, I decided to represent > a N-dimensional array as a list of arrays i.e. > >> data = np.zeros((5, 3), dtype=np.uint8) > >> data = list(data) > >> inorder to preserve the dtype but when it comes to typecasting and > writing it into array (from list) pyarrow.array(data, type=type) it gives > the following error: > >> pyarrow.lib.ArrowInvalid: Could not convert [0 0 0] with type > numpy.ndarray: tried to convert to int > >> Is there any way to avoid this issue? I just want to preserve the dtype > from numpy array before converting it to list so that while writing it to > pyarrow array format I can recognise its dtype and subsequently write it in > that numpy dtype format. > > > > > > I tried it with a 3D numpy array and it gave me this error even though > it's working fine with 2D numpy arrays. Can you please look into this? > > > > pyarrow.lib.ArrowInvalid: Can only convert 1-dimensional array values > > > > > > My current pyarrow version is 2.0.0 and i tried it with pyarrow==3.0.0 > too > > > > > > > > Thanks, > > Bhavitvya >
