Hi,
Currently I am trying to count on a document with multiple filter.
Let say, here is my document:
//user field1 field2 field3
user1 0 0 1
user2 0 1 0
user3 0 0 0
I want to count on user.log for some filters like this:
Filter1: field1 == 0 & field 2 = 0
Filter2: field1 == 0 & field 3 = 1
Filter3: field1 == 0 & field 3 = 0
...
and total line.
I have tried and I found that I couldn't use "group by" or "map then reduce"
because a line could match two or more filter.
My idea now is "foreach" line and then maintain a outsite counter service.
Forexample:
JavaRDD<String> textFile = sc.textFile(hdfs, 10);
long start = System.currentTimeMillis();
textFile.foreach(new VoidFunction<String>() {
public void call(String s) {
foreach(MyFilter filter: MyFilters){
if(filter.match(s)) filter.increaseOwnCounter();
}
}
});
I would happy if there have another way to do it, any help is appreciate.
Thanks in advance.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Multiple-Filter-Effiency-tp20701.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
