[jira] [Commented] (SPARK-8386) DataFrame and JDBC regression
[ https://issues.apache.org/jira/browse/SPARK-8386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724756#comment-14724756 ] Vishal Bagga commented on SPARK-8386: - I just ran into the same issue today. This should addressed and looks like a sever limitation. > DataFrame and JDBC regression > - > > Key: SPARK-8386 > URL: https://issues.apache.org/jira/browse/SPARK-8386 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.0 > Environment: RHEL 7.1 >Reporter: Peter Haumer >Priority: Critical > > I have an ETL app that appends to a JDBC table new results found at each run. > In 1.3.1 I did this: > testResultsDF.insertIntoJDBC(CONNECTION_URL, TABLE_NAME, false); > When I do this now in 1.4 it complains that the "object" 'TABLE_NAME' already > exists. I get this even if I switch the overwrite to true. I also tried this > now: > testResultsDF.write().mode(SaveMode.Append).jdbc(CONNECTION_URL, TABLE_NAME, > connectionProperties); > getting the same error. It works running the first time creating the new > table and adding data successfully. But, running it a second time it (the > jdbc driver) will tell me that the table already exists. Even > SaveMode.Overwrite will give me the same error. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8386) DataFrame and JDBC regression
[ https://issues.apache.org/jira/browse/SPARK-8386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724756#comment-14724756 ] Vishal Bagga edited comment on SPARK-8386 at 9/1/15 4:05 AM: - I just ran into the same issue today. This should be addressed and looks like a sever limitation for JDBC writers. was (Author: vishal b): I just ran into the same issue today. This should addressed and looks like a sever limitation. > DataFrame and JDBC regression > - > > Key: SPARK-8386 > URL: https://issues.apache.org/jira/browse/SPARK-8386 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 1.4.0 > Environment: RHEL 7.1 >Reporter: Peter Haumer >Priority: Critical > > I have an ETL app that appends to a JDBC table new results found at each run. > In 1.3.1 I did this: > testResultsDF.insertIntoJDBC(CONNECTION_URL, TABLE_NAME, false); > When I do this now in 1.4 it complains that the "object" 'TABLE_NAME' already > exists. I get this even if I switch the overwrite to true. I also tried this > now: > testResultsDF.write().mode(SaveMode.Append).jdbc(CONNECTION_URL, TABLE_NAME, > connectionProperties); > getting the same error. It works running the first time creating the new > table and adding data successfully. But, running it a second time it (the > jdbc driver) will tell me that the table already exists. Even > SaveMode.Overwrite will give me the same error. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9041) Support reading of delimited text files as DataFrames
[ https://issues.apache.org/jira/browse/SPARK-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724752#comment-14724752 ] Vishal Bagga commented on SPARK-9041: - [~srowen] I checked this and set priority to Major according to the wiki URL above: Major: a small minority of users are missing important functionality without this, and there is a workaround The component is set to SQL and that seems to be okay? > Support reading of delimited text files as DataFrames > - > > Key: SPARK-9041 > URL: https://issues.apache.org/jira/browse/SPARK-9041 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 1.4.0 >Reporter: Vishal Bagga > > It would be a great addition to support reading of delimited text files > directly into DataFrames with automatic schema inference etc. There is a > package out there (spark-csv) but I believe that this functionality really > belongs in core spark/spark sql since this is such a common use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-9041) Support reading of delimited text files as DataFrames
[ https://issues.apache.org/jira/browse/SPARK-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Bagga updated SPARK-9041: Priority: Major (was: Minor) > Support reading of delimited text files as DataFrames > - > > Key: SPARK-9041 > URL: https://issues.apache.org/jira/browse/SPARK-9041 > Project: Spark > Issue Type: New Feature > Components: SQL >Affects Versions: 1.4.0 >Reporter: Vishal Bagga > > It would be a great addition to support reading of delimited text files > directly into DataFrames with automatic schema inference etc. There is a > package out there (spark-csv) but I believe that this functionality really > belongs in core spark/spark sql since this is such a common use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-9041) Support reading of text/csv files as DataFrames
Vishal Bagga created SPARK-9041: --- Summary: Support reading of text/csv files as DataFrames Key: SPARK-9041 URL: https://issues.apache.org/jira/browse/SPARK-9041 Project: Spark Issue Type: New Feature Affects Versions: 1.4.0 Reporter: Vishal Bagga It would be a great addition to support reading of delimited text files directly into DataFrames with automatic schema inference etc. There is a package out there (spark-csv) but I believe that this functionality really belongs in core spark/spark sql since this is such a common use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-9041) Support reading of delimited files as DataFrames
[ https://issues.apache.org/jira/browse/SPARK-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Bagga updated SPARK-9041: Summary: Support reading of delimited files as DataFrames (was: Support reading of text/csv files as DataFrames) > Support reading of delimited files as DataFrames > > > Key: SPARK-9041 > URL: https://issues.apache.org/jira/browse/SPARK-9041 > Project: Spark > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Vishal Bagga > > It would be a great addition to support reading of delimited text files > directly into DataFrames with automatic schema inference etc. There is a > package out there (spark-csv) but I believe that this functionality really > belongs in core spark/spark sql since this is such a common use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-9041) Support reading of delimited text files as DataFrames
[ https://issues.apache.org/jira/browse/SPARK-9041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vishal Bagga updated SPARK-9041: Summary: Support reading of delimited text files as DataFrames (was: Support reading of delimited files as DataFrames) > Support reading of delimited text files as DataFrames > - > > Key: SPARK-9041 > URL: https://issues.apache.org/jira/browse/SPARK-9041 > Project: Spark > Issue Type: New Feature >Affects Versions: 1.4.0 >Reporter: Vishal Bagga > > It would be a great addition to support reading of delimited text files > directly into DataFrames with automatic schema inference etc. There is a > package out there (spark-csv) but I believe that this functionality really > belongs in core spark/spark sql since this is such a common use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
