Rahul, Glad to know you were able to upload a 1GB file and process it. We know the system has a lot of room for performance improvement. You are very welcome to do some investigation and identify bottlenecks.
I am including two colleagues from ASU, Jaykumar Rajeshbhai Tandel < [email protected]> and Jia Zou <[email protected]>. Recently they are investigating related issues, and may be able to chime in to provide insights. We will be happy to work with you on an optimization project. Chen Li PPMC Apache Texera (Incubating) On Sun, Aug 30, 2026 at 8:14 AM Rahul P <[email protected]> wrote: > Hi Chen Li / Team > > Thanks for your valuable feedback. > > I have installed Texera on a remote Linux server and created the 1 GB test > file on that server. Since I am accessing the Texera UI from my Windows > machine, I manually used Texera's multipart upload API from the Linux > server to upload the file as a Texera dataset. > > During the upload, I initially encountered a file size limit issue, which > I resolved by increasing the configured limit. After that, the file was > uploaded successfully. > > During execution, I observed that the *File Scan operator was reading the > file in small chunks and writing the data in Iceberg format*. I increased > the chunk size and restarted the execution. > > However, the *Python UDF operator*, which reads one line at a time, > parses it as a CSV row, converts it into six structured fields, and outputs > the tuple, took around *11 hours* to process the 27 million records. > > I am using *3 worker nodes*, and the Python UDF was also running with *3 > workers*. > > Is this expected behavior for Texera, or am I missing any configuration or > optimization that could improve the Python UDF performance? > > [image: image.png] > > > Thanks & Regards > Rahul > > On Fri, Aug 28, 2026 at 11:17 AM Chen Li <[email protected]> wrote: > >> Rahul, >> >> Sorry for the late reply. Currently the CSVFileScan operator doesn't >> support multiple workers to run in parallel. So reading a large file may >> take time. If you are interested in developing a parallel reader, you are >> very welcome to do so and the team can provide support. >> >> The Python UDF operator allows a user to specify the number of workers. >> >> The backend engine is doing in-memory computation. Data transfer between >> operators can be pipelined. >> >> The backend storage layer supports large files using solutions such as >> MinIO, LakeFS, and Iceberg. >> >> The system had done a lot of optimization in uploading large files >> efficiently (e.g., in the range of TBs). >> >> Our team will be happy to answer questions related to your particular use >> case. >> >> Thank you, >> >> Chen Li >> >> >> >> >> On Thu, Aug 27, 2026 at 9:17 PM Rahul P <[email protected]> wrote: >> >> > Hi Team >> > >> > Can I please get an update on this. >> > >> > Thanks & Regards >> > Rahul >> > >> > On Thu, Aug 27, 2026 at 12:09 PM Rahul P <[email protected]> >> wrote: >> > >> > > Hi Team, >> > > >> > > I have installed Texera and am able to execute workflows >> successfully. I >> > > would like to understand how large-scale data processing is handled in >> > > Texera. >> > > >> > > If the input data size is very large, will Texera process it reliably >> > > without crashing, or are there any known limitations related to file >> > size, >> > > memory, or cluster resources? >> > > >> > > I also could not find any option to configure parallelism for the file >> > > fetch/file scan operator. How can parallelism be configured for this >> > > operator? >> > > >> > > Thanks & Regards >> > > Rahul >> > > >> > >> >
