[jira] [Comment Edited] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path
[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017820#comment-17017820 ] Sivakumar edited comment on SPARK-30542 at 1/20/20 12:36 PM: - Hi Jungtaek, I think this might be a feature that should be added to structured streaming. Earlier with Spark Dstreams two jobs can have a same base path. But with Spark structured streaming I don't have that flexibility. I guess this should be a feature that structured streaming should support. Also Please lemme know If you have any work around for this. was (Author: sparksiva): Hi Jungtaek, I thought this might be a feature that should be added to structured streaming. Earlier with Spark Dstreams two jobs can have a same base path. But with Spark structured streaming I don't have that flexibility. I guess this should be a feature that structured streaming should support. Also Please lemme know If you have any work around for this. > Two Spark structured streaming jobs cannot write to same base path > -- > > Key: SPARK-30542 > URL: https://issues.apache.org/jira/browse/SPARK-30542 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Sivakumar >Priority: Major > > Hi All, > Spark Structured Streaming doesn't allow two structured streaming jobs to > write data to the same base directory which is possible with using dstreams. > As __spark___metadata directory will be created by default for one job, > second job cannot use the same directory as base path as already > _spark__metadata directory is created by other job, It is throwing exception. > Is there any workaround for this, other than creating separate base path's > for both the jobs. > Is it possible to create the __spark__metadata directory else where or > disable without any data loss. > If I had to change the base path for both the jobs, then my whole framework > will get impacted, So i don't want to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path
[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017820#comment-17017820 ] Sivakumar edited comment on SPARK-30542 at 1/17/20 12:40 PM: - Hi Jungtaek, I thought this might be a feature that should be added to structured streaming. Earlier with Spark Dstreams two jobs can have a same base path. But with Spark structured streaming I don't have that flexibility. I guess this should be a feature that structured streaming should support. Also Please lemme know If you have any work around for this. was (Author: sparksiva): Earlier with Spark Dstreams two jobs can have a same base path. But with Spark structured streaming I don't have that flexibility. I guess this should be a feature that structured streaming should support. > Two Spark structured streaming jobs cannot write to same base path > -- > > Key: SPARK-30542 > URL: https://issues.apache.org/jira/browse/SPARK-30542 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Sivakumar >Priority: Major > > Hi All, > Spark Structured Streaming doesn't allow two structured streaming jobs to > write data to the same base directory which is possible with using dstreams. > As __spark___metadata directory will be created by default for one job, > second job cannot use the same directory as base path as already > _spark__metadata directory is created by other job, It is throwing exception. > Is there any workaround for this, other than creating separate base path's > for both the jobs. > Is it possible to create the __spark__metadata directory else where or > disable without any data loss. > If I had to change the base path for both the jobs, then my whole framework > will get impacted, So i don't want to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-30542) Two Spark structured streaming jobs cannot write to same base path
[ https://issues.apache.org/jira/browse/SPARK-30542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017967#comment-17017967 ] Sivakumar edited comment on SPARK-30542 at 1/17/20 12:40 PM: - Hi Jungtaek, I thought this might be a feature that should be added to structured streaming. Also Please lemme know If you have any work around for this. was (Author: sparksiva): Hi Jungtaek, I thought this might be a feature that should be added to structured streaming. Also Please lemme know If you have any work around for this. > Two Spark structured streaming jobs cannot write to same base path > -- > > Key: SPARK-30542 > URL: https://issues.apache.org/jira/browse/SPARK-30542 > Project: Spark > Issue Type: Bug > Components: Structured Streaming >Affects Versions: 2.3.0 >Reporter: Sivakumar >Priority: Major > > Hi All, > Spark Structured Streaming doesn't allow two structured streaming jobs to > write data to the same base directory which is possible with using dstreams. > As __spark___metadata directory will be created by default for one job, > second job cannot use the same directory as base path as already > _spark__metadata directory is created by other job, It is throwing exception. > Is there any workaround for this, other than creating separate base path's > for both the jobs. > Is it possible to create the __spark__metadata directory else where or > disable without any data loss. > If I had to change the base path for both the jobs, then my whole framework > will get impacted, So i don't want to do that. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
