[GitHub] [incubator-uniffle] fhalde commented on issue #955: [DOCS] Improving the description of several parameters
fhalde commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-159449 > Yes, buffer size should be 0.7 of XMX_SIZE. ok, so can i set both `rss.server.buffer.capacity` and `rss.server.read.buffer.capacity` to 0.7 of xmx size or should they be split into 0.35 each ? is `rss.server.buffer.capacity` for write and `rss.server.read.buffer.capacity` for read ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-uniffle] fhalde commented on issue #955: [DOCS] Improving the description of several parameters
fhalde commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-1594492938 > It's enough for several TBs shuffle with 1000 executors to use 9 shuffle server. very impressive. how much storage ( disk ) do you typically attach per shuffle server? we have some jobs that shuffle almost 150TB of data. One could argue that the job needs to be re-written but as a platform we mostly have no control over when the job gets fixed to reduce the shuffle -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-uniffle] fhalde commented on issue #955: [DOCS] Improving the description of several parameters
fhalde commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-1593306745 ok. what do you mean by > We should set it according to the partition number of shuffle server -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-uniffle] fhalde commented on issue #955: [DOCS] Improving the description of several parameters
fhalde commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-1593244337 the assignment process warrants better documentation. let me first try to understand it by myself -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-uniffle] fhalde commented on issue #955: [DOCS] Improving the description of several parameters
fhalde commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-1593240180 > We will assign 5 shuffle nodes to the first shuffle. We will assign another 10 shuffle nodes to the second shuffle oh i see, is this to reduce the spread of the data? does the rss server merge partitions coming from different mappers? > We should according to the partition number of shuffle server is this configurable? is this the same as `rss.coordinator.shuffle.nodes.max` ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [incubator-uniffle] fhalde commented on issue #955: [DOCS] Improving the description of several parameters
fhalde commented on issue #955: URL: https://github.com/apache/incubator-uniffle/issues/955#issuecomment-1593233428 thanks so much, also in `rss-env.sh`. there's a reference to `XMX_SIZE`. should the buffer sizes defined influence the JVM size of the shuffle server? or does allocation happen off heap? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: dev-unsubscr...@uniffle.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
