Steve Loughran created SPARK-20448: -------------------------------------- Summary: Document how FileInputDStream works with object storage Key: SPARK-20448 URL: https://issues.apache.org/jira/browse/SPARK-20448 Project: Spark Issue Type: Documentation Components: Documentation Affects Versions: 2.1.0 Reporter: Steve Loughran Priority: Minor
Object stores work differently from filesystems: intermediate writes not visible, renames are really O(data) copies, not O(1) transactions. This makes working with them as DStreams fundamentally different: you can write straight into the destination. 1. Document how FileinputDStreams scan directories for changes 2. Document how object stores behave differently, and the implications for users. -- This message was sent by Atlassian JIRA (v6.3.15#6346) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org