I second everything Erik just said, spot on! On Tue, Aug 24, 2021 at 11:45 AM Erik Ritter <[email protected]> wrote:
> Hi Sriram, > > I would strongly recommend against using the upload CSV functionality in > Superset as a major part of your data warehouse pipeline, and instead use a > tool like Apache Airflow to write and schedule ingestion pipelines, or use > CLIs/GUIs provided by your warehouse's DB to load data in one-off use > cases. The upload CSV feature is primarily intended for importing small > amounts of data from external sources into your data warehouse and not for > large CSV files or creating core tables. > > As for visualizing trillions of rows of data, a lot of that will fall on > your warehouse's DB query engine to handle. I assume you don't want to > display trillions of datapoints in the browser, but instead show > aggregations of those rows. Superset should perform fine, as the actual > aggregations are computed in the warehouse, and Superset only handles the > aggregated result set (and includes default row limits on said aggregations > to prevent overloading of the browser or the webserver). > > I hope this helps! > Erik Ritter > > On Tue, Aug 24, 2021 at 8:28 AM Sriram V <[email protected]> wrote: > > > Hi Team, > > We are evaluating superset for visualizing trillion rows of data. > > > > 1. Is there any size limitation for superset to upload the data as > csv? > > We could not even upload the csv file of size 500MB (With 2 million rows > of > > data) to superset. > > 2. If there is no size limitation, can you provide some user stories > > that superset is currently performing by handling huge data. > > 3. Is there any direct way to upload or connect data in the form of > orc > > ,parquet and delta lake format without connecting to data bricks? > > > > Thanks, > > > > Regards, > > Sriram V > > Engineer > > Tata Consultancy Services, > > Chennai > > =====-----=====-----===== > > Notice: The information contained in this e-mail > > message and/or attachments to it may contain > > confidential or privileged information. If you are > > not the intended recipient, any dissemination, use, > > review, distribution, printing or copying of the > > information contained in this e-mail message > > and/or attachments to it are strictly prohibited. If > > you have received this communication in error, > > please notify us by reply e-mail or telephone and > > immediately and permanently delete the message > > and any attachments. Thank you > > > > > > >
