[GitHub] [echarts] MeetzhDing opened a new issue, #17711: [Bug] dataset DataStore['_initDataFromProvider'] cost too many memory when using encode

[GitBox]

MeetzhDing opened a new issue, #17711:
URL: https://github.com/apache/echarts/issues/17711

   ### Version
   
   5.4.0
   
   ### Link to Minimal Reproduction
   
   https://codepen.io/meetzhding/pen/YzLeNbj
   
   ### Steps to Reproduce
   
   store all data in dataset.source, then draw it in line chart using encode.
   
   ```js
   var container = document.getElementById('main');
   var chart = echarts.init(container);
   
   
   function genTimeSeriesDataset() {
     const dataset = {
       source: {},
       dimensions: [],
     };
     const timePointSize = 3 * 60 * 60
     const lineCount = 100;
   
     dataset.dimensions.push({
       name: 'time1',
       type: 'time',
       displayName: 'time1',
     });
     dataset.source['time1'] = []
     const timeEnd = +new Date().setMilliseconds(0);
     const timeStart = timeEnd - timePointSize * 1000;
     for(let i = 0; i<timePointSize; i++) {
       dataset.source['time1'].push(timeStart + i * 1000)
     }
   
     for(let i=1;i<=lineCount;i++) {
       const dimensionName = `number${i}`
       dataset.dimensions.push({
         name: dimensionName,
         type: 'number',
         displayName: dimensionName,
       });
       dataset.source[dimensionName] = []
       for(let i = 0; i<timePointSize; i++) {
         dataset.source[dimensionName].push(Math.floor(Math.random()*100))
       }
     }
   
     return dataset
   }
   
   const dataset = genTimeSeriesDataset();
   
   console.time('render');
   chart.setOption({
     xAxis: {
       type: 'time',
     },
     yAxis: {
       type: 'value'
     },
     dataset: dataset,
     series: dataset.dimensions.slice(1).map(dimension => {
       return {
         type: 'line',
         name: dimension.name,
         animation: false,
         showSymbol: false,
         encode: {
           x: 'time1',
           y: dimension.name
         },
         stack: 'main',
         lineStyle: {
           width: 0.5
         }
       }
     })
   })
   console.timeEnd('render');
   
   
   ```
   
   ### Current Behavior
   
   <img width="2048" alt="image" 
src="https://user-images.githubusercontent.com/12808432/192789705-a0da3ff8-1ff6-4602-93d4-54058b009cf7.png";>
   
   
   Raw Data Size: 
   ```js
     const timePointSize = 3 * 60 * 60
     const lineCount = 100
     rawMem = timePointSize * (1 + lineCount) * 8Byte = 6240000Bytes  ~=  6M
   ```
   
   Page Memory Cost：=  800M+
   
   ---
   
   In file `src/data/DataStore.ts`,  _initDataFromProvider will copy the raw 
dataset.source, then the memory cost is equal to `lineCount * rawDatasetSize`
   
   
https://github.com/apache/echarts/blob/213f7ebd8b5f8c6dbb6ae1d11c3a7b97505dc798/src/data/DataStore.ts#L374-L433
   
   ### Expected Behavior
   
   Reduce memory cost.
   
   When using encode, it should copy the raw dataset mentioned in encode only, 
it will save lots of memory.
   
   Or DataStore should use raw dataset field directly, then copy raw data in 
transform step only.
   
   
   
   ### Environment
   
   ```markdown
   - OS: MacOS
   - Browser: Chrome
   - Framework: any
   ```
   
   
   ### Any additional comments?
   
   this question may be related to 
https://github.com/apache/echarts/issues/11907


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscr...@echarts.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@echarts.apache.org
For additional commands, e-mail: commits-h...@echarts.apache.org