Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

github-actions[bot] commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2056210344

   Ephemeral environment shutdown and build artifacts deleted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar merged PR #27840:
URL: https://github.com/apache/superset/pull/27840


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

github-actions[bot] commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2052107239

   @eschutho Ephemeral environment spinning up at http://34.216.119.147:8080. 
Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping 
and startup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2052090948

   /testenv up


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2052032745

   > > Agreed @dpgaspar. We can definitely do a separate PR just to make all 
these modals look the same and use the same component.
   > 
   > @dpgaspar Assuming this , I think the only design changes left would be 
to fix the spacing between inputs
   > 
   > https://private-user-images.githubusercontent.com/70410625/321994905-f40f8a5b-4eba-4b12-b7bf-b16e86dc85da.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI5MjgzMDQsIm5iZiI6MTcxMjkyODAwNCwicGF0aCI6Ii83MDQxMDYyNS8zMjE5OTQ5MDUtZjQwZjhhNWItNGViYS00YjEyLWI3YmYtYjE2ZTg2ZGM4NWRhLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDEyVDEzMjAwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWQwMTI3OTk1NmMyMDAwZGRkZDYwODA0NGVmN2YwZDg2NjQwOTQyZDA4MmRjZTc0MmM0MDYxMDI0MDFiMDNjNGEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.XWjybesvoWyB7Rn0cHaPmL-mgqHwthmw2BaEUCsjhG4;>
 https://private-user-images.githubusercontent.co
 
m/70410625/321994561-090892d0-0a4d-4275-9e56-ed58e89bc6e0.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI5MjgzMDQsIm5iZiI6MTcxMjkyODAwNCwicGF0aCI6Ii83MDQxMDYyNS8zMjE5OTQ1NjEtMDkwODkyZDAtMGE0ZC00Mjc1LTllNTYtZWQ1OGU4OWJjNmUwLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDEyVDEzMjAwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdlZTI4OTg0Y2EzNGExNmI1OTYxMWM1YWIxZmY4ZDRiMTQ1OGQyODkwMGYwM2ZhMTEyYzc1NDFkYzkwOWNjMDUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.mFkmB0DQHk-Y2dovUoRW6uH19nnKrtp1pFGuaEI4rmo">
   > and make the tooltip icons vertically aligned.
   > 
   > https://private-user-images.githubusercontent.com/70410625/321995222-ee05b721-86da-4534-9924-fd8e3ec590fc.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI5MjgzMDQsIm5iZiI6MTcxMjkyODAwNCwicGF0aCI6Ii83MDQxMDYyNS8zMjE5OTUyMjItZWUwNWI3MjEtODZkYS00NTM0LTk5MjQtZmQ4ZTNlYzU5MGZjLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTIlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDEyVDEzMjAwNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTU0MTU2NjlkMzcxMDIwNTcyN2NmMTg1ZjNlYjI5OTcyZmQ5NGQ4YTQwYmJlOThkMTQ5YjAxNjM4NmJkNzc5MDkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.rIpwYsy9im1GVIdqmmHFY0KiS8G0IWbH9xek0Bvof_M;>
   
   ok, done!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2051750163

   > Agreed @dpgaspar. We can definitely do a separate PR just to make all 
these modals look the same and use the same component.
   
   @dpgaspar Assuming this , I think the only design changes left would be to 
fix the spacing between inputs 
   
   
   
   
   https://github.com/apache/superset/assets/70410625/f40f8a5b-4eba-4b12-b7bf-b16e86dc85da;>
   
   
   https://github.com/apache/superset/assets/70410625/090892d0-0a4d-4275-9e56-ed58e89bc6e0;>
   
   
   
   
   and make the tooltip icons vertically aligned.
   
   https://github.com/apache/superset/assets/70410625/ee05b721-86da-4534-9924-fd8e3ec590fc;>
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2049855643

   > 2 - Totally agree, I think that the correct design should be imposed at 
the component level, then we can remove most of the custom styles.ts. So, I 
don't think that doing more style tweaking on this PR makes sense
   
   Agreed @dpgaspar. We can definitely do a separate PR just to make all these 
modals look the same and use the same component.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2049848087

   > Thank you for the improvements @dpgaspar. Here are the comments for the 
second review:
   > 
   > > 1 - Fixed the display disabled at the component level
   > 
   > The cursor is correctly displaying the forbidden sign but the items are 
still becoming blue when hovered which I don't think should be the case because 
it gives the impression that they leave the forbidden state and are now 
clickable. I also noticed that the first item is not disabled. https://private-user-images.githubusercontent.com/70410625/321634223-1a66afee-3859-4071-8fc7-069451a1c4db.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI4NDUyOTQsIm5iZiI6MTcxMjg0NDk5NCwicGF0aCI6Ii83MDQxMDYyNS8zMjE2MzQyMjMtMWE2NmFmZWUtMzg1OS00MDcxLThmYzctMDY5NDUxYTFjNGRiLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDExVDE0MTYzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTgzODVkMDlhNjkyMzg3OWIxMmEyYmJmO
 
DgzMWU3ZDk5YWQ0MTg4NDEwMDk1ZjkxM2U1N2IzOTM3ODk3YWM3MTEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.wQzD65nqFREJF2zL2I2nwEmgULwtT3VwzIFIdiFhTzY">
 | https://private-user-images.githubusercontent.com/70410625/321634275-bb8dbce6-3852-414e-bc39-245e3a7cefd5.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI4NDUyOTQsIm5iZiI6MTcxMjg0NDk5NCwicGF0aCI6Ii83MDQxMDYyNS8zMjE2MzQyNzUtYmI4ZGJjZTYtMzg1Mi00MTRlLWJjMzktMjQ1ZTNhN2NlZmQ1LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDExVDE0MTYzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTI2YzlkNDZiZDkzYTE5OGQzZDU2NGQ0OThlNmZmNDc3OTNkZGUwMzQ2NGU1MjAzNDQwZWE3Y2NmMzdiMmRmMWUmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkP
 TAmcmVwb19pZD0wIn0.wOCgy0WRAxmRA6j1aQ6HHrFmaATSJPak02t4azcrY-k">
   > 
   > > 2 - fixed the width, did not fix the dropdown icons etc because I want 
to avoid customizing the colapse panels (so it follows database modal)
   > 
   > @kasiazjc @eschutho @dpgaspar We just need to decide which design is the 
correct one and apply it consistently:
   > 
   > https://private-user-images.githubusercontent.com/70410625/321631593-4210b0e3-46e0-433d-bd86-e45461717fc3.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI4NDUyOTQsIm5iZiI6MTcxMjg0NDk5NCwicGF0aCI6Ii83MDQxMDYyNS8zMjE2MzE1OTMtNDIxMGIwZTMtNDZlMC00MzNkLWJkODYtZTQ1NDYxNzE3ZmMzLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDExVDE0MTYzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTJmMjdhZDg2ZmViM2VlZTI2NGM0ODI3MjdhMDBhYTIxZGM0ZTkxNDEzZTRhYzA2MzFiNTdiZjYxNTRiZmE5MGMmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.XqQ_ry-YJiHS6dgTtOPuN5H_4Fe1p4cBX0rs-2RzsNA;>
 https://private-user-images.githubusercontent.co
 
m/70410625/321631717-ba2777c9-6b69-4630-8ca1-0d54c78033ef.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI4NDUyOTQsIm5iZiI6MTcxMjg0NDk5NCwicGF0aCI6Ii83MDQxMDYyNS8zMjE2MzE3MTctYmEyNzc3YzktNmI2OS00NjMwLThjYTEtMGQ1NGM3ODAzM2VmLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MTElMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDExVDE0MTYzNFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTFlYzM0NDJmMzMxM2U3NmUxZjRiYjcwMGZkNTJmNWRlNmJlMjg2ODcwNjdiYTcyMDU2NWIyOTI1MmE4ZmIzMmYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.GhF9hN30LTm-3G8TxI3_qKkkxTKI4phDbOUClF6b66I">
   > We need to match spacing between header and description, borders, arrow 
position, the X color. These modals should all be using the same component to 
render the collapse panels.
   > 
   > > 3 - done
   > 
   > Would be possible to move all toggles to last, grouping them, to improve 
the design? I think you can do this for all collapse panels.
   > 
   > 

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

github-actions[bot] commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2048542268

   @eschutho Ephemeral environment spinning up at http://54.214.102.139:8080. 
Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping 
and startup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2048531023

   /testenv up
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1559608941


##
superset/migrations/versions/2024-04-08_15-43_5ad7321c2169_mig_new_csv_upload_perm.py:
##
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""mig new csv upload perm
+
+Revision ID: 5ad7321c2169
+Revises: 678eefb4ab44
+Create Date: 2024-04-08 15:43:29.682687
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "5ad7321c2169"
+down_revision = "678eefb4ab44"
+
+from alembic import op
+from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.orm import Session
+
+from superset.migrations.shared.security_converge import (
+add_pvms,
+get_reversed_new_pvms,
+get_reversed_pvm_map,
+migrate_roles,
+Pvm,
+)
+
+NEW_PVMS = {"Database": ("can_csv_upload",)}
+
+PVM_MAP = {
+Pvm("CsvToDatabaseView", "can_this_form_post"): (
+Pvm("Database", "can_csv_upload"),
+),
+Pvm("CsvToDatabaseView", "can_this_form_get"): (Pvm("Database", 
"can_csv_upload"),),
+}
+
+
+def do_upgrade(session: Session) -> None:
+add_pvms(session, NEW_PVMS)

Review Comment:
   I don't think their breaking, since users don't have to act, the old 
permissions are migrated to the new ones on all existing roles, including 
custom ones.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1559499209


##
superset/migrations/versions/2024-04-08_15-43_5ad7321c2169_mig_new_csv_upload_perm.py:
##
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""mig new csv upload perm
+
+Revision ID: 5ad7321c2169
+Revises: 678eefb4ab44
+Create Date: 2024-04-08 15:43:29.682687
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "5ad7321c2169"
+down_revision = "678eefb4ab44"
+
+from alembic import op
+from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.orm import Session
+
+from superset.migrations.shared.security_converge import (
+add_pvms,
+get_reversed_new_pvms,
+get_reversed_pvm_map,
+migrate_roles,
+Pvm,
+)
+
+NEW_PVMS = {"Database": ("can_csv_upload",)}
+
+PVM_MAP = {
+Pvm("CsvToDatabaseView", "can_this_form_post"): (
+Pvm("Database", "can_csv_upload"),
+),
+Pvm("CsvToDatabaseView", "can_this_form_get"): (Pvm("Database", 
"can_csv_upload"),),
+}
+
+
+def do_upgrade(session: Session) -> None:
+add_pvms(session, NEW_PVMS)

Review Comment:
   The problem is that if this PR contains breaking changes, it will be blocked 
until the next breaking window. There are two ways to avoid this:
   - Make it backward compatible
   - Remove the permission changes from this PR and do them during the breaking 
window



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1559489636


##
superset/migrations/versions/2024-04-08_15-43_5ad7321c2169_mig_new_csv_upload_perm.py:
##
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""mig new csv upload perm
+
+Revision ID: 5ad7321c2169
+Revises: 678eefb4ab44
+Create Date: 2024-04-08 15:43:29.682687
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "5ad7321c2169"
+down_revision = "678eefb4ab44"
+
+from alembic import op
+from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.orm import Session
+
+from superset.migrations.shared.security_converge import (
+add_pvms,
+get_reversed_new_pvms,
+get_reversed_pvm_map,
+migrate_roles,
+Pvm,
+)
+
+NEW_PVMS = {"Database": ("can_csv_upload",)}
+
+PVM_MAP = {
+Pvm("CsvToDatabaseView", "can_this_form_post"): (
+Pvm("Database", "can_csv_upload"),
+),
+Pvm("CsvToDatabaseView", "can_this_form_get"): (Pvm("Database", 
"can_csv_upload"),),
+}
+
+
+def do_upgrade(session: Session) -> None:
+add_pvms(session, NEW_PVMS)

Review Comment:
   I guess you could say it's a breaking change, on `UPDATING.md` should I add 
a note to `## Next` or `### Breaking changes` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1559476034


##
superset-frontend/src/pages/DatabaseList/index.tsx:
##
@@ -230,6 +233,16 @@ function DatabaseList({
 {
   label: t('Upload file to database'),
   childs: [
+{
+  label: t('New Upload CSV'),
+  name: 'Upload CSV file',
+  url: '#',
+  onClick: () => {
+setCsvUploadModalOpen(true);
+  },

Review Comment:
   could not find an easy way without refactoring the component



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1559474828


##
superset-frontend/src/features/home/SubMenu.tsx:
##
@@ -162,6 +162,7 @@ const StyledHeader = styled.div`
 
 const styledDisabled = (theme: SupersetTheme) => css`
   color: ${theme.colors.grayscale.light1};
+  cursor: not-allowed;

Review Comment:
   @michael-s-molina  Disabling the cursor on a menu disabled item



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2047587867

   @michael-s-molina @kasiazjc I've addressed basically all your comments:
   
   @michael-s-molina 
   1 - Fixed the display disabled at the component level
   2 - fixed the width, did not fix the dropdown icons etc because I want to 
avoid customizing the colapse panels (so it follows database modal)
   3 - done
   4 - done
   5 - Followed @kasiazjc comment
   
   ![Screenshot 2024-04-10 at 14 43 
23](https://github.com/apache/superset/assets/4025227/24e640e0-59e2-4ea3-9363-a69c0f0fc250)
   
   6 - Agree removed the code
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2045651454

   > @kasiazjc @dpgaspar One thing I forgot to mention was about the loaded 
columns. Apart from their names, their types are equality important because 
they will inform users if they need to apply any transformation before loading 
them. It's the same principle we have when we create a dataset:
   > 
   > https://private-user-images.githubusercontent.com/70410625/320922545-f357162a-de93-4d9d-a573-b0c90a0c30b8.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2ODAxNDEsIm5iZiI6MTcxMjY3OTg0MSwicGF0aCI6Ii83MDQxMDYyNS8zMjA5MjI1NDUtZjM1NzE2MmEtZGU5My00ZDlkLWE1NzMtYjBjOTBhMGMzMGI4LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE2MjQwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWYwNjE0MTdkOGVkMTA2NTkzM2M1YzlhNjc1MzRlOGM3MjAzZmUxMGY5ZWI0Yjg1ZmMxYzhiYTcxZGU0YjczYTkmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.XTqs8i8PQ4rX6nVQEDtf7OF2emYTLL3SsH371SSh3X8;>
   > We also have a Column Data Types field that allows type manipulation:
   > 
   > https://private-user-images.githubusercontent.com/70410625/320923933-b4a5f1d7-9405-435a-a70b-76de86d86214.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2ODAxNDEsIm5iZiI6MTcxMjY3OTg0MSwicGF0aCI6Ii83MDQxMDYyNS8zMjA5MjM5MzMtYjRhNWYxZDctOTQwNS00MzVhLWE3MGItNzZkZTg2ZDg2MjE0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE2MjQwMVomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWI2MjNiZWIzMmJlOWYxNTQ4ZWJhMjM3YzkxNmM5MDg5NDhjZDczOTdmZTI2YzJiNjEzYzUzY2E5MmI4NzNjMzcmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.OkIgtHw2m9C3XqvFIY-gHy1qipjluAT2ntC6WvofXVQ;>
   > @kasiazjc I think we can resolve both requirements if we present a similar 
table that allows users to edit their types as they would using the Column Data 
Types field.
   
   On the loaded columns we can't infer the type yet, as for the Column Data 
Types field I would vote on improving that field on another PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2045601881

   @kasiazjc @dpgaspar One thing I forgot to mention was about the loaded 
columns. Apart from their names, their types are equality important because 
they will inform users if they need to apply any transformation before loading 
them. It's the same principle we have when we create a dataset:
   
   https://github.com/apache/superset/assets/70410625/f357162a-de93-4d9d-a573-b0c90a0c30b8;>
   
   We also have a Column Data Types field that allows type manipulation:
   
   https://github.com/apache/superset/assets/70410625/b4a5f1d7-9405-435a-a70b-76de86d86214;>
   
   @kasiazjc I think we can resolve both requirements if we present a similar 
table that allows users to edit their types as they would using the Column Data 
Types field.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

kasiazjc commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2045477799

   Thanks @michael-s-molina for the review and @dpgaspar for working on this! 
   > 
   > 1 - I know it's not part of this PR, but could you use this opportunity to 
improve how the menu is displaying the disabled state? Right now, when 
hovering, it's showing the text as clickable, and when you click, nothing 
happens. In my opinion, disabled items should be always grey. @kasiazjc
   
   agreed, disabled items should always be grayed out and show  cursor instead 
of pointy finger, not sure if it can be easily done globally though?  
   
   > 2 - Can you match the modal design with the alerts/reports?
   > 
   > I believe it's important to match the same width, dropdown arrow 
positions, etc. @kasiazjc
   > 
   > https://private-user-images.githubusercontent.com/70410625/320631143-bd81a3ce-41e2-4052-b622-fac6fb1f09a4.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2NzUxMTYsIm5iZiI6MTcxMjY3NDgxNiwicGF0aCI6Ii83MDQxMDYyNS8zMjA2MzExNDMtYmQ4MWEzY2UtNDFlMi00MDUyLWI2MjItZmFjNmZiMWYwOWE0LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE1MDAxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWY1OGUxZDEwOGVlYzliNTNjMzljYTNiNzUyYmQ5YzNmYTBiY2FiMDNjOTRiOWYyOGE4YzU3MTJmMDA4MDNkZTImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.mFrY397mYqulIocW5RzZA96d6BPG294eIXTLmP7tBxQ;>
 https://private-user-images.githubusercontent.co
 
m/70410625/320631168-cdec7e2a-e7da-47dd-be40-dafc70ba11cd.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2NzUxMTYsIm5iZiI6MTcxMjY3NDgxNiwicGF0aCI6Ii83MDQxMDYyNS8zMjA2MzExNjgtY2RlYzdlMmEtZTdkYS00N2RkLWJlNDAtZGFmYzcwYmExMWNkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE1MDAxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPWNmMDdiOTBjZWJkMWE2NjI3NWM1Y2MwZDIxY2YwYjJkOTAxNmQ2YTg1NDYyZGM1NDk4NjExYzEwODNlYzliYWImWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.dYLiE9duF-CIvawuq34KcCT_JhZdhMVmfr6orhzhFl4">
   
   yes, all of the properties should be the same as in the alerts/reports. 
Additionally, each select/input should be in its own row (like in alerts modal)
   
   > 4 - The Select button should be one of our predefined buttons and it looks 
a little lost on the screen. I think once we change the modal width and 
rearrange the fields, it will look better. @kasiazjc
   > 
   > https://private-user-images.githubusercontent.com/70410625/320633416-61ebb606-2d7f-4303-80c8-7d6b32e5204d.png?jwt=eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3MTI2NzUxMTYsIm5iZiI6MTcxMjY3NDgxNiwicGF0aCI6Ii83MDQxMDYyNS8zMjA2MzM0MTYtNjFlYmI2MDYtMmQ3Zi00MzAzLTgwYzgtN2Q2YjMyZTUyMDRkLnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNDA0MDklMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjQwNDA5VDE1MDAxNlomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTk2OWY3NWYxMTZjOTZhNGE3OWY5YWQyYjdlODRkOTMzNThiMTViMDViYTBhYTY3YjUyYzNkM2Q3YTlkYTZlNTYmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0JmFjdG9yX2lkPTAma2V5X2lkPTAmcmVwb19pZD0wIn0.P5RDwcfVb_KLmcDp5qDsJDgbUO6sQFN_bg6if7idJEY;>
   
   I think it should be tertiary button (outlined) so that it's still visible 
and easy to spot.
   
   > 5 - How does loaded columns work with 30+ columns? @kasiazjc can we make 
sure we cover this case?
   > 
   @dpgaspar I think columns should be under the loaded file, with ability to 
uncollapse/collapse the columns (if more than fits in a row) so it would be 
something like this (perfect scenario):
   
   https://github.com/apache/superset/assets/36897697/cd654a16-baed-4f06-88d1-718a0b4da4cb;>
   
   I would also skip "basic" header (if it's not needed from you point of view) 
+ descriptions under the fields in general section (for schema we can move it 
to the tooltip)
   
   What do you think? Is it doable? (also do not mind things like labels not 
looking like what we have in the code)
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, 

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2045213575

   > This looks great! It gets me wondering about "upload columnar file to 
database" and "upload Excel file to database." My users mostly use the Excel 
one. Would some aspects of this overhaul be applicable to those upload options?
   
   I'll do the excel and columnar modals next, and yes they should be 
applicable then


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1557653728


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

mistercrunch commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556624571


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

yousoph commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556499783


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

yousoph commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556499378


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

yousoph commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556498542


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

yousoph commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556493843


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

yousoph commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556492195


##
superset/commands/database/csv_import.py:
##
@@ -0,0 +1,164 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Optional, TypedDict
+
+import pandas as pd
+from flask_babel import lazy_gettext as _
+from sqlalchemy.exc import SQLAlchemyError
+
+from superset import db
+from superset.commands.base import BaseCommand
+from superset.commands.database.exceptions import (
+DatabaseNotFoundError,
+DatabaseSchemaUploadNotAllowed,
+DatabaseUploadFailed,
+)
+from superset.connectors.sqla.models import SqlaTable
+from superset.daos.database import DatabaseDAO
+from superset.models.core import Database
+from superset.sql_parse import Table
+from superset.utils.core import get_user
+from superset.views.database.validators import schema_allows_file_upload
+
+logger = logging.getLogger(__name__)
+
+READ_CSV_CHUNK_SIZE = 1000
+
+
+class CSVImportOptions(TypedDict, total=False):
+schema: str
+delimiter: str
+already_exists: str
+column_data_types: dict[str, str]
+column_dates: list[str]
+column_labels: str
+columns_read: list[str]
+dataframe_index: str
+day_first: bool
+decimal_character: str
+header_row: int
+index_column: str
+null_values: list[str]
+overwrite_duplicates: bool
+rows_to_read: int
+skip_blank_lines: bool
+skip_initial_space: bool
+skip_rows: bool
+
+
+class CSVImportCommand(BaseCommand):
+def __init__(
+self,
+model_id: int,
+table_name: str,
+file: Any,
+options: CSVImportOptions,
+) -> None:
+self._model_id = model_id
+self._model: Optional[Database] = None
+self._table_name = table_name
+self._schema = options.get("schema")
+self._file = file
+self._options = options
+
+def run(self) -> None:
+self.validate()
+if not self._model:
+return
+try:
+df = pd.concat(
+pd.read_csv(
+chunksize=READ_CSV_CHUNK_SIZE,
+encoding="utf-8",
+filepath_or_buffer=self._file,
+header=self._options.get("header_row", 0),
+index_col=self._options.get("index_column"),
+dayfirst=self._options.get("day_first", False),
+iterator=True,
+keep_default_na=not self._options.get("null_values"),
+usecols=self._options.get("columns_read")
+if self._options.get("columns_read")
+else None,
+na_values=self._options.get("null_values")
+if self._options.get("null_values")
+else None,
+nrows=self._options.get("rows_to_read"),
+parse_dates=self._options.get("column_dates"),
+sep=self._options.get("delimiter", ","),
+skip_blank_lines=self._options.get("skip_blank_lines", 
False),
+skipinitialspace=self._options.get("skip_initial_space", 
False),
+skiprows=self._options.get("skip_rows", 0),
+dtype=self._options.get("column_data_types")
+if self._options.get("column_data_types")
+else {},
+)
+)
+csv_table = Table(table=self._table_name, schema=self._schema)
+self._model.db_engine_spec.df_to_sql(
+self._model,
+csv_table,
+df,
+to_sql_kwargs={
+"chunksize": READ_CSV_CHUNK_SIZE,
+"if_exists": self._options.get("already_exists", "fail"),
+"index": self._options.get("index_column"),
+"index_label": self._options.get("column_labels"),
+},
+)
+except ValueError as ex:
+raise DatabaseUploadFailed(
+message=_(
+"Table already exists. You can change your"
+   

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556355099


##
superset-frontend/src/features/databases/CSVUploadModal/index.test.tsx:
##
@@ -0,0 +1,330 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React from 'react';
+import fetchMock from 'fetch-mock';
+import CSVUploadModal from 'src/features/databases/CSVUploadModal';
+import { act, cleanup, render, screen } from 'spec/helpers/testing-library';
+import userEvent from '@testing-library/user-event';
+import { waitFor } from '@testing-library/react';
+
+fetchMock.post('glob:*api/v1/database/1/csv_upload/', {});
+
+fetchMock.get(
+  
'glob:*api/v1/database/?q=(filters:!((col:allow_file_upload,opr:eq,value:!t)),page:0,page_size:100)',
+  {
+result: [
+  {
+id: 1,
+database_name: 'database1',
+  },
+  {
+id: 2,
+database_name: 'database2',
+  },
+],
+  },
+);
+
+fetchMock.get('glob:*api/v1/database/1/schemas/', {
+  result: ['information_schema', 'public'],
+});
+
+fetchMock.get('glob:*api/v1/database/2/schemas/', {
+  result: ['schema1', 'schema2'],
+});
+
+const csvProps = {
+  show: true,
+  onHide: () => {},
+};
+
+describe('CSVUploadModal', () => {

Review Comment:
   Could you avoid using `describe` here and just use `test`?



##
superset-frontend/src/features/databases/CSVUploadModal/index.test.tsx:
##
@@ -0,0 +1,330 @@
+/**

Review Comment:
   Following our other components, the test should be named 
`CSVUploadModalTest.test.tsx`.



##
superset/migrations/versions/2024-04-08_15-43_5ad7321c2169_mig_new_csv_upload_perm.py:
##
@@ -0,0 +1,85 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+"""mig new csv upload perm
+
+Revision ID: 5ad7321c2169
+Revises: 678eefb4ab44
+Create Date: 2024-04-08 15:43:29.682687
+
+"""
+
+# revision identifiers, used by Alembic.
+revision = "5ad7321c2169"
+down_revision = "678eefb4ab44"
+
+from alembic import op
+from sqlalchemy.exc import SQLAlchemyError
+from sqlalchemy.orm import Session
+
+from superset.migrations.shared.security_converge import (
+add_pvms,
+get_reversed_new_pvms,
+get_reversed_pvm_map,
+migrate_roles,
+Pvm,
+)
+
+NEW_PVMS = {"Database": ("can_csv_upload",)}
+
+PVM_MAP = {
+Pvm("CsvToDatabaseView", "can_this_form_post"): (
+Pvm("Database", "can_csv_upload"),
+),
+Pvm("CsvToDatabaseView", "can_this_form_get"): (Pvm("Database", 
"can_csv_upload"),),
+}
+
+
+def do_upgrade(session: Session) -> None:
+add_pvms(session, NEW_PVMS)

Review Comment:
   Can any of the migrations here be considered breaking changes?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

sfirke commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2043247127

   This looks great!  It gets me wondering about "upload columnar file to 
database" and "upload Excel file to database."  My users mostly use the Excel 
one.  Would some aspects of this overhaul be applicable to those upload options?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556136195


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,

Review Comment:
   We shouldn't import a button directly from Ant Design but use one of our 
Button styles.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

michael-s-molina commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1556133882


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1555775226


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

rusackas commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554279700


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554036084


##
superset/commands/database/csv_import.py:
##
@@ -0,0 +1,164 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+import logging
+from typing import Any, Optional, TypedDict
+
+import pandas as pd
+from flask_babel import lazy_gettext as _
+from sqlalchemy.exc import SQLAlchemyError
+
+from superset import db
+from superset.commands.base import BaseCommand
+from superset.commands.database.exceptions import (
+DatabaseNotFoundError,
+DatabaseSchemaUploadNotAllowed,
+DatabaseUploadFailed,
+)
+from superset.connectors.sqla.models import SqlaTable
+from superset.daos.database import DatabaseDAO
+from superset.models.core import Database
+from superset.sql_parse import Table
+from superset.utils.core import get_user
+from superset.views.database.validators import schema_allows_file_upload
+
+logger = logging.getLogger(__name__)
+
+READ_CSV_CHUNK_SIZE = 1000
+
+
+class CSVImportOptions(TypedDict, total=False):
+schema: str
+delimiter: str
+already_exists: str
+column_data_types: dict[str, str]
+column_dates: list[str]
+column_labels: str
+columns_read: list[str]
+dataframe_index: str
+day_first: bool
+decimal_character: str
+header_row: int
+index_column: str
+null_values: list[str]
+overwrite_duplicates: bool
+rows_to_read: int
+skip_blank_lines: bool
+skip_initial_space: bool
+skip_rows: bool
+
+
+class CSVImportCommand(BaseCommand):
+def __init__(
+self,
+model_id: int,
+table_name: str,
+file: Any,
+options: CSVImportOptions,
+) -> None:
+self._model_id = model_id
+self._model: Optional[Database] = None
+self._table_name = table_name
+self._schema = options.get("schema")
+self._file = file
+self._options = options
+
+def run(self) -> None:
+self.validate()
+if not self._model:
+return
+try:
+df = pd.concat(
+pd.read_csv(
+chunksize=READ_CSV_CHUNK_SIZE,
+encoding="utf-8",
+filepath_or_buffer=self._file,
+header=self._options.get("header_row", 0),
+index_col=self._options.get("index_column"),
+dayfirst=self._options.get("day_first", False),
+iterator=True,
+keep_default_na=not self._options.get("null_values"),
+usecols=self._options.get("columns_read")
+if self._options.get("columns_read")
+else None,
+na_values=self._options.get("null_values")
+if self._options.get("null_values")
+else None,
+nrows=self._options.get("rows_to_read"),
+parse_dates=self._options.get("column_dates"),
+sep=self._options.get("delimiter", ","),
+skip_blank_lines=self._options.get("skip_blank_lines", 
False),
+skipinitialspace=self._options.get("skip_initial_space", 
False),
+skiprows=self._options.get("skip_rows", 0),
+dtype=self._options.get("column_data_types")
+if self._options.get("column_data_types")
+else {},
+)
+)
+csv_table = Table(table=self._table_name, schema=self._schema)
+self._model.db_engine_spec.df_to_sql(
+self._model,
+csv_table,
+df,
+to_sql_kwargs={
+"chunksize": READ_CSV_CHUNK_SIZE,
+"if_exists": self._options.get("already_exists", "fail"),
+"index": self._options.get("index_column"),
+"index_label": self._options.get("column_labels"),
+},
+)
+except ValueError as ex:
+raise DatabaseUploadFailed(
+message=_(
+"Table already exists. You can change your"
+  

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554034363


##
superset-frontend/src/pages/DatabaseList/index.tsx:
##
@@ -230,6 +233,16 @@ function DatabaseList({
 {
   label: t('Upload file to database'),
   childs: [
+{
+  label: t('New Upload CSV'),
+  name: 'Upload CSV file',
+  url: '#',
+  onClick: () => {
+setCsvUploadModalOpen(true);
+  },

Review Comment:
   is there a simple way to make this a button instead of an href?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554030824


##
superset-frontend/src/features/databases/CSVUploadModal/styles.ts:
##
@@ -0,0 +1,77 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import { FormItem } from 'src/components/Form';
+import { css, styled, SupersetTheme } from '@superset-ui/core';
+
+const MODAL_BODY_HEIGHT = 180.5;
+const anticonHeight = 12;
+
+export const StyledFormItem = styled(FormItem)`
+  margin-bottom: 0;
+`;
+
+export const antdCollapseStyles = (theme: SupersetTheme) => css`
+  .ant-collapse-header {
+padding-top: ${theme.gridUnit * 3.5}px;
+padding-bottom: ${theme.gridUnit * 2.5}px;
+.anticon.ant-collapse-arrow {
+  top: calc(50% - ${anticonHeight / 2}px);
+}
+.helper {
+  color: ${theme.colors.grayscale.base};
+}
+  }
+  h4 {
+font-size: 16px;

Review Comment:
   can we use the theme values here?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554030212


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554029456


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554027632


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554024631


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

eschutho commented on code in PR #27840:
URL: https://github.com/apache/superset/pull/27840#discussion_r1554025634


##
superset-frontend/src/features/databases/CSVUploadModal/index.tsx:
##
@@ -0,0 +1,805 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+import React, { FunctionComponent, useEffect, useMemo, useState } from 'react';
+import {
+  getClientErrorObject,
+  SupersetClient,
+  SupersetTheme,
+  t,
+} from '@superset-ui/core';
+import Modal from 'src/components/Modal';
+import Collapse from 'src/components/Collapse';
+import {
+  Upload,
+  AntdForm,
+  Col,
+  Row,
+  AntdButton as Button,
+  AsyncSelect,
+  Select,
+  Typography,
+} from 'src/components';
+import { Divider, Switch } from 'antd';
+import { UploadOutlined } from '@ant-design/icons';
+import { Input, InputNumber } from 'src/components/Input';
+import rison from 'rison';
+import { UploadChangeParam, UploadFile } from 'antd/lib/upload/interface';
+import withToasts from 'src/components/MessageToasts/withToasts';
+import {
+  antDModalStyles,
+  antDModalNoPaddingStyles,
+  antdCollapseStyles,
+  StyledFormItem,
+} from './styles';
+
+interface CSVUploadModalProps {
+  addDangerToast: (msg: string) => void;
+  addSuccessToast: (msg: string) => void;
+  onHide: () => void;
+  show: boolean;
+}
+
+interface UploadInfo {
+  database_id: number;
+  table_name: string;
+  schema: string;
+  delimiter: string;
+  already_exists: string;
+  skip_initial_space: boolean;
+  skip_blank_lines: boolean;
+  day_first: boolean;
+  decimal_character: string;
+  null_values: Array;
+  header_row: string;
+  rows_to_read: string | null;
+  skip_rows: string;
+  column_dates: Array;
+  index_column: string | null;
+  dataframe_index: boolean;
+  column_labels: string;
+  columns_read: Array;
+  overwrite_duplicates: boolean;
+  column_data_types: string;
+}
+
+const defaultUploadInfo: UploadInfo = {
+  database_id: 0,
+  table_name: '',
+  schema: '',
+  delimiter: ',',
+  already_exists: 'fail',
+  skip_initial_space: false,
+  skip_blank_lines: false,
+  day_first: false,
+  decimal_character: '.',
+  null_values: [],
+  header_row: '0',
+  rows_to_read: null,
+  skip_rows: '0',
+  column_dates: [],
+  index_column: null,
+  dataframe_index: false,
+  column_labels: '',
+  columns_read: [],
+  overwrite_duplicates: false,
+  column_data_types: '',
+};
+
+const CSVUploadModal: FunctionComponent = ({
+  addDangerToast,
+  addSuccessToast,
+  onHide,
+  show,
+}) => {
+  const [form] = AntdForm.useForm();
+  // Declare states here
+  const [currentDatabaseId, setCurrentDatabaseId] = useState(0);
+  const [fileList, setFileList] = useState([]);
+  const [columns, setColumns] = React.useState([]);
+  const [delimiter, setDelimiter] = useState(',');
+  const [isLoading, setIsLoading] = useState(false);
+  const [currentSchema, setCurrentSchema] = useState('');
+
+  const nullValuesOptions = [
+{
+  value: '""',
+  label: 'Empty Strings ""',
+},
+{
+  value: 'None',
+  label: 'None',
+},
+{
+  value: 'nan',
+  label: 'nan',
+},
+{
+  value: 'null',
+  label: 'null',
+},
+{
+  value: 'N/A',
+  label: 'N/A',
+},
+  ];
+
+  const delimiterOptions = [
+{
+  value: ',',
+  label: 'Comma ","',
+},
+{
+  value: ';',
+  label: 'Semicolon ";"',
+},
+{
+  value: '\t',
+  label: 'Tab "\\t"',
+},
+{
+  value: '|',
+  label: 'Pipe',
+},
+  ];
+
+  const tableAlreadyExistsOptions = [
+{
+  value: 'fail',
+  label: 'Fail',
+},
+{
+  value: 'replace',
+  label: 'Replace',
+},
+{
+  value: 'append',
+  label: 'Append',
+},
+  ];
+
+  const onChangeDatabase = (database: { value: number; label: string }) => {
+setCurrentDatabaseId(database?.value);
+  };
+
+  const onChangeSchema = (schema: { value: string; label: string }) => {
+setCurrentSchema(schema?.value);
+  };
+
+  const onChangeDelimiter = (value: string) => {
+setDelimiter(value);
+  };
+
+  const clearModal = () => {
+setFileList([]);
+setColumns([]);
+setCurrentSchema('');
+setCurrentDatabaseId(0);
+

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

github-actions[bot] commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2040049322

   @dpgaspar Ephemeral environment spinning up at http://52.24.73.183:8080. 
Credentials are `admin`/`admin`. Please allow several minutes for bootstrapping 
and startup.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

Re: [PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar commented on PR #27840:
URL: https://github.com/apache/superset/pull/27840#issuecomment-2040031248

   /testenv up


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org

[PR] feat: new CSV upload form and API [superset]

[via GitHub]

dpgaspar opened a new pull request, #27840:
URL: https://github.com/apache/superset/pull/27840

   
   
   ### SUMMARY
   
   
   ### BEFORE/AFTER SCREENSHOTS OR ANIMATED GIF
   
   
   ### TESTING INSTRUCTIONS
   
   
   ### ADDITIONAL INFORMATION
   
   
   - [ ] Has associated issue:
   - [ ] Required feature flags:
   - [ ] Changes UI
   - [ ] Includes DB Migration (follow approval process in 
[SIP-59](https://github.com/apache/superset/issues/13351))
 - [ ] Migration is atomic, supports rollback & is backwards-compatible
 - [ ] Confirm DB migration upgrade and downgrade tested
 - [ ] Runtime estimates and downtime expectations provided
   - [ ] Introduces new feature or API
   - [ ] Removes existing feature or API
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


-
To unsubscribe, e-mail: notifications-unsubscr...@superset.apache.org
For additional commands, e-mail: notifications-h...@superset.apache.org