GitHub user swaroopchenchu created a discussion: Pipeline processing speeds become slower as output grown.
Hop Version: 2.7.0 heap space: i deployed via helm - name: HOP_OPTIONS value: -Xmx8g -XX:+AggressiveHeap I have simple pipeline where we pick from mssql server and place it in monogDB we have almost 134M records, processing speeds decreasing and becoming extremely slow as output grows. we have tried removing all indexes and put off journal option in mongoDB metadata and made write concern w=1 <img width="512" height="215" alt="image" src="https://github.com/user-attachments/assets/a886dfc2-fe07-4db8-97a8-543ea669899f"; /> it taking 30 hrs to finish, I dont have migration window that long. Need to bring around 2hrs pipeline is plain and simple <img width="620" height="252" alt="image" src="https://github.com/user-attachments/assets/a6c808ec-a78b-4168-9a61-554513f4ffb5"; /> Please suggest any options we make try to make pipeline faster Many Thanks GitHub link: https://github.com/apache/hop/discussions/6070 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
