[jira] [Commented] (SPARK-9162) Implement code generation for ScalaUDF
[ https://issues.apache.org/jira/browse/SPARK-9162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634615#comment-14634615 ] Tarek Auel commented on SPARK-9162: --- I can take it. Implement code generation for ScalaUDF -- Key: SPARK-9162 URL: https://issues.apache.org/jira/browse/SPARK-9162 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9163) Implement code generation for Conv
[ https://issues.apache.org/jira/browse/SPARK-9163?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14634070#comment-14634070 ] Tarek Auel commented on SPARK-9163: --- This might be a duplicate of https://issues.apache.org/jira/browse/SPARK-9132 Implement code generation for Conv -- Key: SPARK-9163 URL: https://issues.apache.org/jira/browse/SPARK-9163 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-9178) UTF8String empty string method
Tarek Auel created SPARK-9178: - Summary: UTF8String empty string method Key: SPARK-9178 URL: https://issues.apache.org/jira/browse/SPARK-9178 Project: Spark Issue Type: New Feature Components: SQL Reporter: Tarek Auel Create a method in UTF8String that returns an empty string, in order to avoid calls of UTF8String.fromString() -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-9177) Reuse Calendar instance in WeekOfYear
[ https://issues.apache.org/jira/browse/SPARK-9177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14632823#comment-14632823 ] Tarek Auel commented on SPARK-9177: --- Sure. Reuse Calendar instance in WeekOfYear - Key: SPARK-9177 URL: https://issues.apache.org/jira/browse/SPARK-9177 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Right now WeekOfYear creates a new Calendar instance for every record, both in code gen and in interpreted mode. We should just reuse the same Calendar instance (i.e. initialize it as a member variable in interpreted mode, and use mutable state in codegen mode). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-9115) date/time function: dayInYear
Tarek Auel created SPARK-9115: - Summary: date/time function: dayInYear Key: SPARK-9115 URL: https://issues.apache.org/jira/browse/SPARK-9115 Project: Spark Issue Type: New Feature Components: SQL Affects Versions: 1.5.0 Reporter: Tarek Auel dayInyear(date): Int Returns the number of the day in the year of a given date. e.g. dayInYear(2015-01-01) == 1, dayInYear(2015-12-31) == 365 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-8995) Cast date strings with date, date and time and just time information to DateType and TimestampTzpe
[ https://issues.apache.org/jira/browse/SPARK-8995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tarek Auel updated SPARK-8995: -- Description: Tests of https://github.com/apache/spark/pull/6981 fail, because we can not cast strings like '13:18:08' to a valid date and extract the hours later. It's not possible to parse strings that contains date and time information to date, like '2015-03-18 12:25:49' (was: Tests of https://github.com/apache/spark/pull/6981 fails, because we can not cast strings like '13:18:08' to a valid date and extract the hours later. It's not possible to parse strings that contains date and time information to date, like '2015-03-18 12:25:49') Cast date strings with date, date and time and just time information to DateType and TimestampTzpe -- Key: SPARK-8995 URL: https://issues.apache.org/jira/browse/SPARK-8995 Project: Spark Issue Type: Improvement Components: SQL Reporter: Tarek Auel Tests of https://github.com/apache/spark/pull/6981 fail, because we can not cast strings like '13:18:08' to a valid date and extract the hours later. It's not possible to parse strings that contains date and time information to date, like '2015-03-18 12:25:49' -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8864) Date/time function and data type design
[ https://issues.apache.org/jira/browse/SPARK-8864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14619838#comment-14619838 ] Tarek Auel commented on SPARK-8864: --- Thanks Reynold. The function date_format(date, format) has been removed? Date/time function and data type design --- Key: SPARK-8864 URL: https://issues.apache.org/jira/browse/SPARK-8864 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Reynold Xin Fix For: 1.5.0 Attachments: SparkSQLdatetimeudfs (1).pdf Please see the attached design doc. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8830) levenshtein directly on top of UTF8String
[ https://issues.apache.org/jira/browse/SPARK-8830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14614590#comment-14614590 ] Tarek Auel commented on SPARK-8830: --- I am working on this levenshtein directly on top of UTF8String - Key: SPARK-8830 URL: https://issues.apache.org/jira/browse/SPARK-8830 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin We currently rely on commons-lang's levenshtein implementation. Ideally, we should have our own implementation to: 1. Reduce external dependency 2. Work directly against UTF8String so we don't need to convert to/from java.lang.String back and forth. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-8727) Add missing python api
Tarek Auel created SPARK-8727: - Summary: Add missing python api Key: SPARK-8727 URL: https://issues.apache.org/jira/browse/SPARK-8727 Project: Spark Issue Type: Improvement Components: SQL Reporter: Tarek Auel Add the python api that is missing for https://issues.apache.org/jira/browse/SPARK-8248 https://issues.apache.org/jira/browse/SPARK-8234 https://issues.apache.org/jira/browse/SPARK-8217 https://issues.apache.org/jira/browse/SPARK-8215 https://issues.apache.org/jira/browse/SPARK-8212 -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8212) math function: e
[ https://issues.apache.org/jira/browse/SPARK-8212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607589#comment-14607589 ] Tarek Auel commented on SPARK-8212: --- I added the python api: https://issues.apache.org/jira/browse/SPARK-8727 math function: e Key: SPARK-8212 URL: https://issues.apache.org/jira/browse/SPARK-8212 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Adrian Wang Labels: missing-python Fix For: 1.5.0 e(): double Returns the value of e. We should make this foldable so it gets folded by the optimizer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8234) misc function: md5
[
https://issues.apache.org/jira/browse/SPARK-8234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607586#comment-14607586
]
Tarek Auel commented on SPARK-8234:
---
I added the python api:
https://issues.apache.org/jira/browse/SPARK-8727
misc function: md5
--
Key: SPARK-8234
URL: https://issues.apache.org/jira/browse/SPARK-8234
Project: Spark
Issue Type: Sub-task
Components: SQL
Reporter: Reynold Xin
Assignee: Qian, Shilei
Labels: missing-python
Fix For: 1.5.0
md5(string/binary): string
Calculates an MD5 128-bit checksum for the string or binary (as of Hive
1.3.0). The value is returned as a string of 32 hex digits, or NULL if the
argument was NULL. Example: md5('ABC') = '902fbdd2b1df0c4f70b4a5d23525e932'.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8248) string function: length
[ https://issues.apache.org/jira/browse/SPARK-8248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607585#comment-14607585 ] Tarek Auel commented on SPARK-8248: --- I added the python api: https://issues.apache.org/jira/browse/SPARK-8727 string function: length --- Key: SPARK-8248 URL: https://issues.apache.org/jira/browse/SPARK-8248 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Cheng Hao Labels: missing-python Fix For: 1.5.0 length(string A): int Returns the length of the string. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8217) math function: log2
[ https://issues.apache.org/jira/browse/SPARK-8217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607587#comment-14607587 ] Tarek Auel commented on SPARK-8217: --- I added the python api: https://issues.apache.org/jira/browse/SPARK-8727 math function: log2 --- Key: SPARK-8217 URL: https://issues.apache.org/jira/browse/SPARK-8217 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Adrian Wang Labels: missing-python Fix For: 1.5.0 log2(double a): double Returns the base-2 logarithm of the argument a. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8215) math function: pi
[ https://issues.apache.org/jira/browse/SPARK-8215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607588#comment-14607588 ] Tarek Auel commented on SPARK-8215: --- I added the python api: https://issues.apache.org/jira/browse/SPARK-8727 math function: pi - Key: SPARK-8215 URL: https://issues.apache.org/jira/browse/SPARK-8215 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Adrian Wang Labels: missing-python Fix For: 1.5.0 pi(): double Returns the value of pi. We should make sure foldable = true so it gets folded by the optimizer. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8363) Move sqrt into math
[ https://issues.apache.org/jira/browse/SPARK-8363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14607592#comment-14607592 ] Tarek Auel commented on SPARK-8363: --- [~rxin] I guess the python api does exist: https://github.com/apache/spark/blob/master/python/pyspark/sql/functions.py#L99 Move sqrt into math --- Key: SPARK-8363 URL: https://issues.apache.org/jira/browse/SPARK-8363 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin Assignee: Liang-Chi Hsieh Labels: missing-python Fix For: 1.5.0 It doesn't really belong in Arithmetic. It should also extend UnaryMathExpression. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8668) expr function to convert SQL expression into a Column
[ https://issues.apache.org/jira/browse/SPARK-8668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14606067#comment-14606067 ] Tarek Auel commented on SPARK-8668: --- Hi, just to get it right: selectExpr of the dataframe api takes at the moment varargs as arguments. This should be enhanced in order to parse ONE string argument that contains multiple expressions, shouldn't it? Or do I get it wrong? expr function to convert SQL expression into a Column - Key: SPARK-8668 URL: https://issues.apache.org/jira/browse/SPARK-8668 Project: Spark Issue Type: Sub-task Components: SQL Reporter: Reynold Xin selectExpr uses the expression parser to parse a string expressions. would be great to create an expr function in functions.scala/functions.py that converts a string into an expression (or a list of expressions separated by comma). -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8301) Improve UTF8String substring/startsWith/endsWith/contains performance
[ https://issues.apache.org/jira/browse/SPARK-8301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582691#comment-14582691 ] Tarek Auel edited comment on SPARK-8301 at 6/11/15 11:45 PM: - Another approach could be: (0 until b.length).forall(( i ) = b( i ) == bytes( i )) In theory this could be parallelised, was (Author: tarekauel): Another approach could be: (0 until b.length).forall((i) = b(i) == bytes(i)) In theory this could be parallelised, Improve UTF8String substring/startsWith/endsWith/contains performance - Key: SPARK-8301 URL: https://issues.apache.org/jira/browse/SPARK-8301 Project: Spark Issue Type: Improvement Components: SQL Reporter: Reynold Xin Priority: Critical Many functions in UTF8String are unnecessarily expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8301) Improve UTF8String substring/startsWith/endsWith/contains performance
[ https://issues.apache.org/jira/browse/SPARK-8301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582691#comment-14582691 ] Tarek Auel commented on SPARK-8301: --- Another approach could be: (0 until b.length).forall((i) = b(i) == bytes(i)) In theory this could be parallelised, Improve UTF8String substring/startsWith/endsWith/contains performance - Key: SPARK-8301 URL: https://issues.apache.org/jira/browse/SPARK-8301 Project: Spark Issue Type: Improvement Components: SQL Reporter: Reynold Xin Priority: Critical Many functions in UTF8String are unnecessarily expensive. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-8301) Improve UTF8String substring/startsWith/endsWith/contains performance
[
https://issues.apache.org/jira/browse/SPARK-8301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582609#comment-14582609
]
Tarek Auel commented on SPARK-8301:
---
Hi,
do you have concrete ideas how the implementation should be improved?
I would like to take this item. One suggestion for startWith could be:
val b = prefix.getBytes
if (b.length bytes.length) {
return false
}
var i = 0
while (i b.length (bytes(i) == b(i))) {
i += 1
}
i == b.length
What do you think? Do you have already ideas in your mind how the performance
should be improved?
Improve UTF8String substring/startsWith/endsWith/contains performance
-
Key: SPARK-8301
URL: https://issues.apache.org/jira/browse/SPARK-8301
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Reynold Xin
Priority: Critical
Many functions in UTF8String are unnecessarily expensive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-8301) Improve UTF8String substring/startsWith/endsWith/contains performance
[
https://issues.apache.org/jira/browse/SPARK-8301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14582609#comment-14582609
]
Tarek Auel edited comment on SPARK-8301 at 6/11/15 10:25 PM:
-
Hi,
do you have concrete ideas how the implementation should be improved?
I would like to take this item. One suggestion for startWith could be:
val b = prefix.getBytes
if (b.length bytes.length) {
return false
}
var i = 0
while (i b.length (bytes ( i ) == b ( i ))) {
i += 1
}
i == b.length
What do you think? Do you have already ideas in your mind how the performance
should be improved?
was (Author: tarekauel):
Hi,
do you have concrete ideas how the implementation should be improved?
I would like to take this item. One suggestion for startWith could be:
val b = prefix.getBytes
if (b.length bytes.length) {
return false
}
var i = 0
while (i b.length (bytes(i) == b(i))) {
i += 1
}
i == b.length
What do you think? Do you have already ideas in your mind how the performance
should be improved?
Improve UTF8String substring/startsWith/endsWith/contains performance
-
Key: SPARK-8301
URL: https://issues.apache.org/jira/browse/SPARK-8301
Project: Spark
Issue Type: Improvement
Components: SQL
Reporter: Reynold Xin
Priority: Critical
Many functions in UTF8String are unnecessarily expensive.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-7999) Graph complement
Tarek Auel created SPARK-7999: - Summary: Graph complement Key: SPARK-7999 URL: https://issues.apache.org/jira/browse/SPARK-7999 Project: Spark Issue Type: Improvement Reporter: Tarek Auel Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7999) Graph complement
[ https://issues.apache.org/jira/browse/SPARK-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tarek Auel updated SPARK-7999: -- Issue Type: Sub-task (was: Improvement) Parent: SPARK-7893 Graph complement Key: SPARK-7999 URL: https://issues.apache.org/jira/browse/SPARK-7999 Project: Spark Issue Type: Sub-task Reporter: Tarek Auel Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7999) Graph complement
[ https://issues.apache.org/jira/browse/SPARK-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tarek Auel updated SPARK-7999: -- Description: This task is for implementing the complement operation (compare to parent task). http://techieme.in/complex-graph-operations/ Graph complement Key: SPARK-7999 URL: https://issues.apache.org/jira/browse/SPARK-7999 Project: Spark Issue Type: Sub-task Reporter: Tarek Auel Priority: Minor This task is for implementing the complement operation (compare to parent task). http://techieme.in/complex-graph-operations/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-7999) Graph complement
[ https://issues.apache.org/jira/browse/SPARK-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566999#comment-14566999 ] Tarek Auel commented on SPARK-7999: --- I would propose def complement(attr: ED): Graph[VD, ED] as interface Graph complement Key: SPARK-7999 URL: https://issues.apache.org/jira/browse/SPARK-7999 Project: Spark Issue Type: Sub-task Reporter: Tarek Auel Priority: Minor This task is for implementing the complement operation (compare to parent task). http://techieme.in/complex-graph-operations/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-7999) Graph complement
[ https://issues.apache.org/jira/browse/SPARK-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14566999#comment-14566999 ] Tarek Auel edited comment on SPARK-7999 at 6/1/15 7:04 AM: --- I would propose def complement(attr: ED, selfLoops: Boolean = false): Graph[VD, ED] as interface. The self-loop parameter defines whether self loops (A--A) should be created or not. was (Author: tarekauel): I would propose def complement(attr: ED): Graph[VD, ED] as interface Graph complement Key: SPARK-7999 URL: https://issues.apache.org/jira/browse/SPARK-7999 Project: Spark Issue Type: Sub-task Reporter: Tarek Auel Priority: Minor This task is for implementing the complement operation (compare to parent task). http://techieme.in/complex-graph-operations/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-7999) Graph complement function in GraphX
[ https://issues.apache.org/jira/browse/SPARK-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tarek Auel updated SPARK-7999: -- Summary: Graph complement function in GraphX (was: Graph complement) Graph complement function in GraphX --- Key: SPARK-7999 URL: https://issues.apache.org/jira/browse/SPARK-7999 Project: Spark Issue Type: Sub-task Components: GraphX Reporter: Tarek Auel Priority: Minor This task is for implementing the complement operation (compare to parent task). http://techieme.in/complex-graph-operations/ -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-7966) add Spreading Activation algorithm to GraphX
Tarek Auel created SPARK-7966: - Summary: add Spreading Activation algorithm to GraphX Key: SPARK-7966 URL: https://issues.apache.org/jira/browse/SPARK-7966 Project: Spark Issue Type: Improvement Components: GraphX Reporter: Tarek Auel Priority: Minor I'm wondering if you would like to add the Spreading Activation algorithm to GraphX. I have implemented it, using the Pregel-API and would love to share it with the community. Spreading activation is a algorithm that was invented to search in associative networks. The basic idea is, that you have one (or multiple) starting nodes. The activation spreads out from these nodes to the neighbours and the neighbours of the neighbours. The activation decreases after every hop. Nodes that were reached by many activations will have a higher total activation level. Spreading Activation is for many use cases useful. Imagine you have the social network of two people. If you apply the spreading activation to this social graph with the two people as starting nodes, you will get the nodes that are most important for both. Some resources: http://www.websci11.org/fileadmin/websci/posters/105_paper.pdf https://webfiles.uci.edu/eloftus/CollinsLoftus_PsychReview_75.pdf?uniq=20ou4w -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
