Thank you, Jay, It will be great to have your optimization integrated into the system.
Chen On Mon, Aug 31, 2026 at 3:38 PM Jaykumar Rajeshbhai Tandel <[email protected]> wrote: > Hi Dr. Li and Rahul, > > Yes, we observed the same single-worker limitation on the CSV reader. In > our case, though, it was not the bottleneck. For UDF-centric workloads, we > found the dominant costs were the distribution of work to downstream UDF > workers and the per-tuple build/read/serialize for the Python UDF. > > For reference, our setup: two neural-network UDFs on CPU over a 2.25 GB > dataset, on a machine with 48 cores and 125 GB of memory. The best observed > end-to-end time was 11 minutes 8 seconds. The two UDFs and a link to the > dataset are attached. > > Two changes made the largest difference: > > *Use more workers.* Each Texera worker runs as a separate process, so UDF > work parallelizes well when the machine has cores to spare. We used 20 > workers for each UDF. > > *Use a smaller data transfer batch size.* A batch only ships once a > buffer fills, and each sending worker keeps a separate buffer per receiving > worker, so a large batch size can delay downstream workers considerably. We > saw the best result at a 400 batch size if parallelization is needed. > > Rahul, with 3 workers and 27M records, this is worth checking. Texera does > not yet allocate resources for UDFs automatically, so worker counts and > batch size have to be set manually, based on observations and the resources > available at your end. > > We are currently working on making that overlap independent of the batch > size, so that pipelining also holds at larger batch sizes in multi-UDF > workflows. We plan to raise a PR for the parallelining issue by Sept 9, > and one for the per-tuple cost by Sept 15, so the wider community can > benefit. > Please feel free to reach out if you have more queries. > > Thanks and Regards, > Jay. > > creditcard_x16.csv > <https://drive.google.com/open?id=12n5TRN64sJDFtsa-3igbGguQqYopo1AY> > > UDF Code > <https://drive.google.com/open?id=1jgBexhIhxnoxx0GC_ZPMwI2eJ-ThtsDPnJfYjvQgrxg> > > On Sun, Aug 30, 2026 at 9:27 PM Chen Li <[email protected]> wrote: > >> Rahul, >> >> Glad to know you were able to upload a 1GB file and process it. We know >> the system has a lot of room for performance improvement. You are very >> welcome to do some investigation and identify bottlenecks. >> >> I am including two colleagues from ASU, Jaykumar Rajeshbhai Tandel < >> [email protected]> and Jia Zou <[email protected]>. Recently they are >> investigating related issues, and may be able to chime in to >> provide insights. >> >> We will be happy to work with you on an optimization project. >> >> Chen Li >> PPMC >> Apache Texera (Incubating) >> >> On Sun, Aug 30, 2026 at 8:14 AM Rahul P <[email protected]> wrote: >> >>> Hi Chen Li / Team >>> >>> Thanks for your valuable feedback. >>> >>> I have installed Texera on a remote Linux server and created the 1 GB >>> test file on that server. Since I am accessing the Texera UI from my >>> Windows machine, I manually used Texera's multipart upload API from the >>> Linux server to upload the file as a Texera dataset. >>> >>> During the upload, I initially encountered a file size limit issue, >>> which I resolved by increasing the configured limit. After that, the file >>> was uploaded successfully. >>> >>> During execution, I observed that the *File Scan operator was reading >>> the file in small chunks and writing the data in Iceberg format*. I >>> increased the chunk size and restarted the execution. >>> >>> However, the *Python UDF operator*, which reads one line at a time, >>> parses it as a CSV row, converts it into six structured fields, and outputs >>> the tuple, took around *11 hours* to process the 27 million records. >>> >>> I am using *3 worker nodes*, and the Python UDF was also running with *3 >>> workers*. >>> >>> Is this expected behavior for Texera, or am I missing any configuration >>> or optimization that could improve the Python UDF performance? >>> >>> [image: image.png] >>> >>> >>> Thanks & Regards >>> Rahul >>> >>> On Fri, Aug 28, 2026 at 11:17 AM Chen Li <[email protected]> wrote: >>> >>>> Rahul, >>>> >>>> Sorry for the late reply. Currently the CSVFileScan operator doesn't >>>> support multiple workers to run in parallel. So reading a large file >>>> may >>>> take time. If you are interested in developing a parallel reader, you >>>> are >>>> very welcome to do so and the team can provide support. >>>> >>>> The Python UDF operator allows a user to specify the number of workers. >>>> >>>> The backend engine is doing in-memory computation. Data transfer >>>> between >>>> operators can be pipelined. >>>> >>>> The backend storage layer supports large files using solutions such as >>>> MinIO, LakeFS, and Iceberg. >>>> >>>> The system had done a lot of optimization in uploading large files >>>> efficiently (e.g., in the range of TBs). >>>> >>>> Our team will be happy to answer questions related to your particular >>>> use >>>> case. >>>> >>>> Thank you, >>>> >>>> Chen Li >>>> >>>> >>>> >>>> >>>> On Thu, Aug 27, 2026 at 9:17 PM Rahul P <[email protected]> >>>> wrote: >>>> >>>> > Hi Team >>>> > >>>> > Can I please get an update on this. >>>> > >>>> > Thanks & Regards >>>> > Rahul >>>> > >>>> > On Thu, Aug 27, 2026 at 12:09 PM Rahul P <[email protected]> >>>> wrote: >>>> > >>>> > > Hi Team, >>>> > > >>>> > > I have installed Texera and am able to execute workflows >>>> successfully. I >>>> > > would like to understand how large-scale data processing is handled >>>> in >>>> > > Texera. >>>> > > >>>> > > If the input data size is very large, will Texera process it >>>> reliably >>>> > > without crashing, or are there any known limitations related to file >>>> > size, >>>> > > memory, or cluster resources? >>>> > > >>>> > > I also could not find any option to configure parallelism for the >>>> file >>>> > > fetch/file scan operator. How can parallelism be configured for this >>>> > > operator? >>>> > > >>>> > > Thanks & Regards >>>> > > Rahul >>>> > > >>>> > >>>> >>>
