Hi When is the optimization expected to be integrated?
Thanks & Regards Rahul On Tue, 1 Sept, 2026, 08:13 Chen Li, <[email protected]> wrote: > Thank you, Jay, It will be great to have your optimization integrated into > the system. > > Chen > > On Mon, Aug 31, 2026 at 3:38 PM Jaykumar Rajeshbhai Tandel < > [email protected]> > wrote: > > > Hi Dr. Li and Rahul, > > > > Yes, we observed the same single-worker limitation on the CSV reader. In > > our case, though, it was not the bottleneck. For UDF-centric workloads, > we > > found the dominant costs were the distribution of work to downstream UDF > > workers and the per-tuple build/read/serialize for the Python UDF. > > > > For reference, our setup: two neural-network UDFs on CPU over a 2.25 GB > > dataset, on a machine with 48 cores and 125 GB of memory. The best > observed > > end-to-end time was 11 minutes 8 seconds. The two UDFs and a link to the > > dataset are attached. > > > > Two changes made the largest difference: > > > > *Use more workers.* Each Texera worker runs as a separate process, so UDF > > work parallelizes well when the machine has cores to spare. We used 20 > > workers for each UDF. > > > > *Use a smaller data transfer batch size.* A batch only ships once a > > buffer fills, and each sending worker keeps a separate buffer per > receiving > > worker, so a large batch size can delay downstream workers considerably. > We > > saw the best result at a 400 batch size if parallelization is needed. > > > > Rahul, with 3 workers and 27M records, this is worth checking. Texera > does > > not yet allocate resources for UDFs automatically, so worker counts and > > batch size have to be set manually, based on observations and the > resources > > available at your end. > > > > We are currently working on making that overlap independent of the batch > > size, so that pipelining also holds at larger batch sizes in multi-UDF > > workflows. We plan to raise a PR for the parallelining issue by Sept 9, > > and one for the per-tuple cost by Sept 15, so the wider community can > > benefit. > > Please feel free to reach out if you have more queries. > > > > Thanks and Regards, > > Jay. > > > > creditcard_x16.csv > > <https://drive.google.com/open?id=12n5TRN64sJDFtsa-3igbGguQqYopo1AY> > > > > UDF Code > > < > https://drive.google.com/open?id=1jgBexhIhxnoxx0GC_ZPMwI2eJ-ThtsDPnJfYjvQgrxg > > > > > > On Sun, Aug 30, 2026 at 9:27 PM Chen Li <[email protected]> wrote: > > > >> Rahul, > >> > >> Glad to know you were able to upload a 1GB file and process it. We know > >> the system has a lot of room for performance improvement. You are very > >> welcome to do some investigation and identify bottlenecks. > >> > >> I am including two colleagues from ASU, Jaykumar Rajeshbhai Tandel < > >> [email protected]> and Jia Zou <[email protected]>. Recently they are > >> investigating related issues, and may be able to chime in to > >> provide insights. > >> > >> We will be happy to work with you on an optimization project. > >> > >> Chen Li > >> PPMC > >> Apache Texera (Incubating) > >> > >> On Sun, Aug 30, 2026 at 8:14 AM Rahul P <[email protected]> > wrote: > >> > >>> Hi Chen Li / Team > >>> > >>> Thanks for your valuable feedback. > >>> > >>> I have installed Texera on a remote Linux server and created the 1 GB > >>> test file on that server. Since I am accessing the Texera UI from my > >>> Windows machine, I manually used Texera's multipart upload API from the > >>> Linux server to upload the file as a Texera dataset. > >>> > >>> During the upload, I initially encountered a file size limit issue, > >>> which I resolved by increasing the configured limit. After that, the > file > >>> was uploaded successfully. > >>> > >>> During execution, I observed that the *File Scan operator was reading > >>> the file in small chunks and writing the data in Iceberg format*. I > >>> increased the chunk size and restarted the execution. > >>> > >>> However, the *Python UDF operator*, which reads one line at a time, > >>> parses it as a CSV row, converts it into six structured fields, and > outputs > >>> the tuple, took around *11 hours* to process the 27 million records. > >>> > >>> I am using *3 worker nodes*, and the Python UDF was also running with > *3 > >>> workers*. > >>> > >>> Is this expected behavior for Texera, or am I missing any configuration > >>> or optimization that could improve the Python UDF performance? > >>> > >>> [image: image.png] > >>> > >>> > >>> Thanks & Regards > >>> Rahul > >>> > >>> On Fri, Aug 28, 2026 at 11:17 AM Chen Li <[email protected]> wrote: > >>> > >>>> Rahul, > >>>> > >>>> Sorry for the late reply. Currently the CSVFileScan operator doesn't > >>>> support multiple workers to run in parallel. So reading a large file > >>>> may > >>>> take time. If you are interested in developing a parallel reader, you > >>>> are > >>>> very welcome to do so and the team can provide support. > >>>> > >>>> The Python UDF operator allows a user to specify the number of > workers. > >>>> > >>>> The backend engine is doing in-memory computation. Data transfer > >>>> between > >>>> operators can be pipelined. > >>>> > >>>> The backend storage layer supports large files using solutions such as > >>>> MinIO, LakeFS, and Iceberg. > >>>> > >>>> The system had done a lot of optimization in uploading large files > >>>> efficiently (e.g., in the range of TBs). > >>>> > >>>> Our team will be happy to answer questions related to your particular > >>>> use > >>>> case. > >>>> > >>>> Thank you, > >>>> > >>>> Chen Li > >>>> > >>>> > >>>> > >>>> > >>>> On Thu, Aug 27, 2026 at 9:17 PM Rahul P <[email protected]> > >>>> wrote: > >>>> > >>>> > Hi Team > >>>> > > >>>> > Can I please get an update on this. > >>>> > > >>>> > Thanks & Regards > >>>> > Rahul > >>>> > > >>>> > On Thu, Aug 27, 2026 at 12:09 PM Rahul P <[email protected]> > >>>> wrote: > >>>> > > >>>> > > Hi Team, > >>>> > > > >>>> > > I have installed Texera and am able to execute workflows > >>>> successfully. I > >>>> > > would like to understand how large-scale data processing is > handled > >>>> in > >>>> > > Texera. > >>>> > > > >>>> > > If the input data size is very large, will Texera process it > >>>> reliably > >>>> > > without crashing, or are there any known limitations related to > file > >>>> > size, > >>>> > > memory, or cluster resources? > >>>> > > > >>>> > > I also could not find any option to configure parallelism for the > >>>> file > >>>> > > fetch/file scan operator. How can parallelism be configured for > this > >>>> > > operator? > >>>> > > > >>>> > > Thanks & Regards > >>>> > > Rahul > >>>> > > > >>>> > > >>>> > >>> >
