[
https://issues.apache.org/jira/browse/SPARK-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14531229#comment-14531229
]
Alexander Ulanov commented on SPARK-7316:
-----------------------------------------
I would say that the major use case is practical considerations :)
In my case it is time series analysis of sensor data. It does not make sense to
analyze time windows with step 1 because it is high-frequency sensor (1024 Hz).
Also, even if we want to do it, the size of the resulting data gets enormous.
For example, I have 2B data points (542 hours) of size 23GB binary data. If I
apply sliding window with size 1024 and step 1, it will result in
1024*23=23.5TB of data which I am not able to process with Spark currently
(honestly speaking my disk space is only 10TB). If you store data in HDFS than
it will be tripled, i.e. 70TB.
> Add step capability to RDD sliding window
> -----------------------------------------
>
> Key: SPARK-7316
> URL: https://issues.apache.org/jira/browse/SPARK-7316
> Project: Spark
> Issue Type: Improvement
> Components: MLlib
> Affects Versions: 1.3.0
> Reporter: Alexander Ulanov
> Fix For: 1.4.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> RDDFunctions in MLlib contains sliding window implementation with step 1.
> User should be able to define step. This capability should be implemented.
> Although one can generate sliding windows with step 1 and then filter every
> Nth window, it might take much more time and disk space depending on the step
> size. For example, if your window is 1000 then you will generate the amount
> of data thousand times bigger than your initial dataset. It does not make
> sense if you need just every Nth window, so the data generated will be 1000/N
> smaller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
