-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 I would suggest creating one DStream per directory and then using StreamingContext#union(...) to get a union DStream.
- -- Ankur
On 13/05/2015 00:53, hotdog wrote:
> I want to use use fileStream in spark streaming to monitor multi
> hdfs directories, such as:
>
> val list_join_action_stream = ssc.fileStream[LongWritable, Text,
> TextInputFormat]("/user/root/*/*", check_valid_file(_),
> false).map(_._2.toString).print
>
>
> Buy the way, i could not under the meaning of the three class :
> LongWritable, Text, TextInputFormat
>
> but it doesn't work...
>
>
>
> -- View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/how-to-monitor-mul
ti-directories-in-spark-streaming-task-tp22863.html
>
>
Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
>
>
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
-----BEGIN PGP SIGNATURE-----
iQEcBAEBAgAGBQJVUxJ1AAoJEOSJAMhvLp3L2f4IAKK+ouQ2VD7H6s/5w/YGbt2P
uBGJPQ92Hb5REq3f4gK4YecygtAlSAwsqXGCoAaaoPAC7vUMs9RM+slqse1gmUPU
pbORTIB9dv3iVxjPtZ6R8EX14BAlxcIOR6ni2RBHuQTL+dgIEUekmCg0IhFa5lVF
Kt5in8rY5PSnX5l/dX9Yu8LI3uC4TLQ+eJXjjOGXoCHys+SaZWJckA3gVcF9GQdB
dwdhv4UCIYVFj3QIVlLf0+B8FgA0DnRfBC+5ZfS88gcWMc4065sDdx5LkySy4oZB
tB8IpC4yaY3Mqiu8jdvhcw+SevlYan5YkkkutSvKH7nL/0d1WIkEkHxPBjRqAmY=
=U0oQ
-----END PGP SIGNATURE-----
0x6D461C4A.asc
Description: application/pgp-keys
0x6D461C4A.asc.sig
Description: Binary data
--------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
