I have a single source of data. The processing of records have to be directed to multiple destinations. i.e
1. read the source data 2. based on condition route to the following sources 1. Kafka for error records 2. store success records with certain condition in s3 bucket, bucket name : "A", folder - "a" 4. store success records with certain condition in s3 bucket, bucket name : "A", folder - "b" 3. store success records with certain condition in a different s3 bucket How can I achieve this in pyspark? Are there any resources on the design patterns or common industry followed architectural patterns for apache spark? | Source | Destination | | ----------- | ----------- | | Single | Single | | Multiple | Single | | Single | Multiple | | Multiple | Multiple | -- Sent from: http://apache-spark-user-list.1001560.n3.nabble.com/ --------------------------------------------------------------------- To unsubscribe e-mail: user-unsubscr...@spark.apache.org