[jira] [Commented] (SPARK-4727) Add "dimensional" RDDs (time series, spatial)
[ https://issues.apache.org/jira/browse/SPARK-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14609095#comment-14609095 ] David Sabater commented on SPARK-4727: -- [~sandyr] is actually working on something similar: https://github.com/cloudera/spark-timeseries I am actually interested in ARIMA models which needs timeSeriesRDD to be implemented. > Add "dimensional" RDDs (time series, spatial) > - > > Key: SPARK-4727 > URL: https://issues.apache.org/jira/browse/SPARK-4727 > Project: Spark > Issue Type: Brainstorming > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: RJ Nowling > > Certain types of data (times series, spatial) can benefit from specialized > RDDs. I'd like to open a discussion about this. > For example, time series data should be ordered by time and would benefit > from operations like: > * Subsampling (taking every n data points) > * Signal processing (correlations, FFTs, filtering) > * Windowing functions > Spatial data benefits from ordering and partitioning along a 2D or 3D grid. > For example, path finding algorithms can optimized by only comparing points > within a set distance, which can be computed more efficiently by partitioning > data into a grid. > Although the operations on time series and spatial data may be different, > there is some commonality in the sense of the data having ordered dimensions > and the implementations may overlap. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4727) Add "dimensional" RDDs (time series, spatial)
[ https://issues.apache.org/jira/browse/SPARK-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14384213#comment-14384213 ] Simon Ouellette commented on SPARK-4727: This is a great idea, I need a TimeSeriesRDD concept for the work I'm doing at the moment -- and I believe a specialized RDD would be helpful (right now I'm treating time series as RDD[(Long, Double)] where the Long is my time dimension -- which is definitely less than ideal). I need lag functions, differentials, and more advanced statistics such as augmented dickey-fuller tests, etc. I'd be interested in contributing to this effort. A good first step would be to translate to Scala the work done in Thunder, and then see if we can extend the RDD base class to implement optimal partitioning. > Add "dimensional" RDDs (time series, spatial) > - > > Key: SPARK-4727 > URL: https://issues.apache.org/jira/browse/SPARK-4727 > Project: Spark > Issue Type: Brainstorming > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: RJ Nowling > > Certain types of data (times series, spatial) can benefit from specialized > RDDs. I'd like to open a discussion about this. > For example, time series data should be ordered by time and would benefit > from operations like: > * Subsampling (taking every n data points) > * Signal processing (correlations, FFTs, filtering) > * Windowing functions > Spatial data benefits from ordering and partitioning along a 2D or 3D grid. > For example, path finding algorithms can optimized by only comparing points > within a set distance, which can be computed more efficiently by partitioning > data into a grid. > Although the operations on time series and spatial data may be different, > there is some commonality in the sense of the data having ordered dimensions > and the implementations may overlap. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4727) Add "dimensional" RDDs (time series, spatial)
[ https://issues.apache.org/jira/browse/SPARK-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234399#comment-14234399 ] RJ Nowling commented on SPARK-4727: --- Thanks, Jeremy! Your work may cover my needs, and if not, it seems like a great place to contribute to! Was there some talk about encouraging people to build Spark libraries and putting together a community list? I'd love to see this sort of work advertised more. > Add "dimensional" RDDs (time series, spatial) > - > > Key: SPARK-4727 > URL: https://issues.apache.org/jira/browse/SPARK-4727 > Project: Spark > Issue Type: Brainstorming > Components: Spark Core >Affects Versions: 1.1.0 >Reporter: RJ Nowling > > Certain types of data (times series, spatial) can benefit from specialized > RDDs. I'd like to open a discussion about this. > For example, time series data should be ordered by time and would benefit > from operations like: > * Subsampling (taking every n data points) > * Signal processing (correlations, FFTs, filtering) > * Windowing functions > Spatial data benefits from ordering and partitioning along a 2D or 3D grid. > For example, path finding algorithms can optimized by only comparing points > within a set distance, which can be computed more efficiently by partitioning > data into a grid. > Although the operations on time series and spatial data may be different, > there is some commonality in the sense of the data having ordered dimensions > and the implementations may overlap. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-4727) Add "dimensional" RDDs (time series, spatial)
[
https://issues.apache.org/jira/browse/SPARK-4727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14234269#comment-14234269
]
Jeremy Freeman commented on SPARK-4727:
---
Great to brainstorm about this RJ!
To some extent, we've been doing this over on the
[Thunder|http://thefreemanlab.com/thunder/docs/] project. In particular, check
out the {{TimeSeries}} and {{Images}} classes
[here|https://github.com/freeman-lab/thunder/tree/master/python/thunder/rdds],
which are essentially wrappers for specialized RDDs. Our basic abstraction is
RDDs of ndarrays (1D for time series, 2D or 3D for images/volumes), with
metadeta (lazily propagated) for things like dimensionality and time base,
coordinates embedded in keys, and useful methods on these objects like the ones
you menion (e.g. filtering, fourier, cross-correlation). We've also worked on
transformations between representations, for the common case of sequences of
images corresponding to different time points.
We haven't worked on custom partition strategies yet, I think that will be most
important for image tiles drawn from a much larger image. There's cool work
ongoing for that in GeoTrellis, see the
[repo|https://github.com/geotrellis/geotrellis/tree/master/spark/src/main] and
a
[talk|http://spark-summit.org/2014/talk/geotrellis-adding-geospatial-capabilities-to-spark]
from Rob.
FWIW, when we started it seemed more appropriate to build this into a
specialized library, rather than Spark core. It's also something that benefits
from using Python, due to a bevy of existing libraries for temporal and image
data (though there are certainly analogs in Java/Scala). But it would be great
to probe the community for general interest in these kinds of abstractions and
methods.
> Add "dimensional" RDDs (time series, spatial)
> -
>
> Key: SPARK-4727
> URL: https://issues.apache.org/jira/browse/SPARK-4727
> Project: Spark
> Issue Type: Brainstorming
> Components: Spark Core
>Affects Versions: 1.1.0
>Reporter: RJ Nowling
>
> Certain types of data (times series, spatial) can benefit from specialized
> RDDs. I'd like to open a discussion about this.
> For example, time series data should be ordered by time and would benefit
> from operations like:
> * Subsampling (taking every n data points)
> * Signal processing (correlations, FFTs, filtering)
> * Windowing functions
> Spatial data benefits from ordering and partitioning along a 2D or 3D grid.
> For example, path finding algorithms can optimized by only comparing points
> within a set distance, which can be computed more efficiently by partitioning
> data into a grid.
> Although the operations on time series and spatial data may be different,
> there is some commonality in the sense of the data having ordered dimensions
> and the implementations may overlap.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
