Re: [Dev] about wso2-ml

[Nirmal Fernando]

Hi Roger,

Thanks for your interest in WSO2 ML.

On Mon, Feb 6, 2017 at 8:35 PM, Roger Giuffre  wrote:

> I'm studying your WSO2-ml implementation because it seems a very interesting
> starting point for this new generation of spark centric applications. But
> they encounter any questions  about the DataProcessors you have
> implemented. For example, about the DasDataProcessor where do you save
> the inputStream?
>

DASDatasetProcessor is a special dataset processor where it sets the
pointers to
WSO2 Data Analytics Server tables. Then, when we want to retrieve data we
are loading DAS tables
directly into Spark [2].

If you take our FileDatasetProcessor [1], you can see that we have
basically consumed
the input stream and persist in a server side. (this is done assuming that
files are small
in size, if they are stored in the file system).

In the HdfsDatasetProcessor, again we don't persist the file at the server
side.

Hope this helps. Let us know if you need further clarifications.

[1]
https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/impl/FileDatasetProcessor.java
[2]
https://github.com/wso2/carbon-ml/blob/master/components/ml/org.wso2.carbon.ml.core/src/main/java/org/wso2/carbon/ml/core/utils/MLUtils.java#L127

>
>
> Roger
>
> ___
> Dev mailing list
> Dev@wso2.org
> http://wso2.org/cgi-bin/mailman/listinfo/dev
>
>


-- 

Thanks & regards,
Nirmal

Technical Lead - Analytics Team, WSO2 Inc.
Mobile: +94715779733
Blog: http://nirmalfdo.blogspot.com/
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev

[Dev] about wso2-ml

[Roger Giuffre]

I'm studying your WSO2-ml implementation because it seems a very interesting
starting point for this new generation of spark centric applications. But
they encounter any questions  about the DataProcessors you have
implemented. For
example, about the DasDataProcessor where do you save the inputStream?

Roger
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev