Re: [Dev] [ML] Document - Integrating DL to WSO2-ML
Hi, I couldn't attach the file itself (Apparently, they only support images) I've commented with a link to the file (Google Drive). On Mon, Aug 24, 2015 at 3:08 PM, Thushan Ganegedara wrote: > Hi, > > Yup. I'll do that > > On Mon, Aug 24, 2015 at 3:06 PM, Nirmal Fernando wrote: > >> Thushan, one more thing, can you please attach this in the PR too? Thanks. >> >> On Mon, Aug 24, 2015 at 10:15 AM, Thushan Ganegedara >> wrote: >> >>> Yes, sure I'll do that >>> >>> On Mon, Aug 24, 2015 at 2:39 PM, Nirmal Fernando >>> wrote: >>> >>>> Thushan, >>>> >>>> Can you send a PR to carbon-ml repo? >>>> >>>> On Mon, Aug 24, 2015 at 8:14 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> Please find the final document attached with. >>>>> >>>>> Furthremore, the code has been merged with the latest version of the >>>>> upstream repository (wos2/carbon-ml) and can be found here. >>>>> https://github.com/thushv89/carbon-ml >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Thushan Ganegedara >>>>> School of IT >>>>> University of Sydney, Australia >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Document - Integrating DL to WSO2-ML
Hi, Yup. I'll do that On Mon, Aug 24, 2015 at 3:06 PM, Nirmal Fernando wrote: > Thushan, one more thing, can you please attach this in the PR too? Thanks. > > On Mon, Aug 24, 2015 at 10:15 AM, Thushan Ganegedara > wrote: > >> Yes, sure I'll do that >> >> On Mon, Aug 24, 2015 at 2:39 PM, Nirmal Fernando wrote: >> >>> Thushan, >>> >>> Can you send a PR to carbon-ml repo? >>> >>> On Mon, Aug 24, 2015 at 8:14 AM, Thushan Ganegedara >>> wrote: >>> >>>> Hi, >>>> >>>> Please find the final document attached with. >>>> >>>> Furthremore, the code has been merged with the latest version of the >>>> upstream repository (wos2/carbon-ml) and can be found here. >>>> https://github.com/thushv89/carbon-ml >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Document - Integrating DL to WSO2-ML
Yes, sure I'll do that On Mon, Aug 24, 2015 at 2:39 PM, Nirmal Fernando wrote: > Thushan, > > Can you send a PR to carbon-ml repo? > > On Mon, Aug 24, 2015 at 8:14 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> Please find the final document attached with. >> >> Furthremore, the code has been merged with the latest version of the >> upstream repository (wos2/carbon-ml) and can be found here. >> https://github.com/thushv89/carbon-ml >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Potential Improvement for Predicting test data
Hi, I assume the Jira is, https://wso2.org/jira/browse/ML Isn't this an exclusive JIRA? I don't see an option to create an issue here. On Mon, Aug 24, 2015 at 2:22 PM, Nirmal Fernando wrote: > Thanks Thushan for the suggestion. Can you please create a Jira with this > suggestion? > > On Mon, Aug 24, 2015 at 5:33 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> The current prediction logic requires the prediction test file to have 1 >> column less than of the training file (i.e. removing the response column). >> >> But I think it will be more convenient for users if we do not enforce >> that rule. Instead, perform the following check in prediction logic. >> >> >> If column_count_predict == column_count_train - 1 >> >> Follow normal logic >> >> else if column_count_predict == column_count_train >> >> Get the response column ID/Index >> Remove the corresponding column from the predict file >> Warn the user about the operation >> Follow normal logic >> >> Else >> >> Produce an error >> >> Because something, test files deliberately include the classes column to >> make it easier to test the accuracy >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Potential Improvement for Predicting test data
Hi, The current prediction logic requires the prediction test file to have 1 column less than of the training file (i.e. removing the response column). But I think it will be more convenient for users if we do not enforce that rule. Instead, perform the following check in prediction logic. If column_count_predict == column_count_train - 1 Follow normal logic else if column_count_predict == column_count_train Get the response column ID/Index Remove the corresponding column from the predict file Warn the user about the operation Follow normal logic Else Produce an error Because something, test files deliberately include the classes column to make it easier to test the accuracy -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] scrollVibrateFix() not defined in login.jag
Never mind. Found that it's defined in the wso2.ml.util.js For some reason, the app had referred to an old version of that file which didn't have the method in it. Think it was a caching issue On Mon, Aug 24, 2015 at 7:44 AM, Thushan Ganegedara wrote: > Also the home.jag > > On Mon, Aug 24, 2015 at 7:42 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> I got the latest pull from the carbon-ml and the login.jag ( >> https://github.com/wso2/carbon-ml/blob/master/apps/ml/site/home/login.jag) >> doesn't have the method scrollVibrateFix() defined. Hence, the following >> error, >> *Uncaught ReferenceError: scrollVibrateFix is not defined* >> >> So the login page doesn't work. Could someone please look into it? >> Meanwhile, I'm gonna introduce it in my fork. >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] scrollVibrateFix() not defined in login.jag
Also the home.jag On Mon, Aug 24, 2015 at 7:42 AM, Thushan Ganegedara wrote: > Hi, > > I got the latest pull from the carbon-ml and the login.jag ( > https://github.com/wso2/carbon-ml/blob/master/apps/ml/site/home/login.jag) > doesn't have the method scrollVibrateFix() defined. Hence, the following > error, > *Uncaught ReferenceError: scrollVibrateFix is not defined* > > So the login page doesn't work. Could someone please look into it? > Meanwhile, I'm gonna introduce it in my fork. > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] scrollVibrateFix() not defined in login.jag
Hi, I got the latest pull from the carbon-ml and the login.jag ( https://github.com/wso2/carbon-ml/blob/master/apps/ml/site/home/login.jag) doesn't have the method scrollVibrateFix() defined. Hence, the following error, *Uncaught ReferenceError: scrollVibrateFix is not defined* So the login page doesn't work. Could someone please look into it? Meanwhile, I'm gonna introduce it in my fork. -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Completed DL Visualization component
Hi,
Thank you very much for all the support given to make this project a
success :)
1. We need the code base
I haven't got the latest pull from the upstream. I'll be doing that today
and merging (and resolving conflicts)
2. A documentation on the design, implementation, dependencies, future
improvements, any other relevant thing.
Yes, I'll try to finish and send this by today.
3. Anything else, that you think we might need to know :-)
I can't think of anything else at the moment. If I remember anything, I'll
make sure that I add it to the document.
Thank you
On Fri, Aug 21, 2015 at 3:24 PM, Nirmal Fernando wrote:
> Great! Thanks Thushan for all the hard work on this project. Please, we
> need few things from you;
>
> 1. We need the code base
> 2. A documentation on the design, implementation, dependencies, future
> improvements, any other relevant thing.
> 3. Anything else, that you think we might need to know :-)
>
> Thanks !
>
> On Thu, Aug 20, 2015 at 7:25 PM, Thushan Ganegedara
> wrote:
>
>> Hi,
>>
>> Please find the screenshot for view-model attached
>>
>> On Thu, Aug 20, 2015 at 10:12 PM, Nirmal Fernando
>> wrote:
>>
>>> I'd prefer at the bottom, to be consistent with other Algos.
>>>
>>> On Thu, Aug 20, 2015 at 5:40 PM, Thushan Ganegedara
>>> wrote:
>>>
>>>> Hello CD,
>>>>
>>>> Yes, seems there was a bit of miscommunication :)
>>>>
>>>> I'm migrating the code to that.
>>>>
>>>> Also, where do you think we should place the visualization. I'm
>>>> attaching a screenshot of the current view-model with red arrows for
>>>> potential placements.
>>>>
>>>>
>>>>
>>>> On Thu, Aug 20, 2015 at 10:07 PM, CD Athuraliya
>>>> wrote:
>>>>
>>>>> Hi Thushan,
>>>>>
>>>>> Now do we have the visualization in both parameters.jag and model.jag?
>>>>> I think Nirmal was suggesting to show it in model summery page. You can
>>>>> find it here analysis/view-model.jag.
>>>>>
>>>>> Regards,
>>>>> CD
>>>>>
>>>>> On Thu, Aug 20, 2015 at 5:29 PM, Thushan Ganegedara
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Now the visualization is included in the models.jag.
>>>>>>
>>>>>> Screenshots are attached
>>>>>>
>>>>>> On Thu, Aug 20, 2015 at 9:35 PM, Nirmal Fernando
>>>>>> wrote:
>>>>>>
>>>>>>> Yes, api/analyses/{analysisId}/hyperParameters
>>>>>>>
>>>>>>> On Thu, Aug 20, 2015 at 5:03 PM, Thushan Ganegedara <
>>>>>>> thu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Thanks for the inputs. I'm working on that.
>>>>>>>>
>>>>>>>> By the way, is there a way to get the hyperparameter values
>>>>>>>> (specifically, Layer_Sizes parameter) to the model.jag page with an API
>>>>>>>> call? I checked the API documentation, but couldn't find one.
>>>>>>>>
>>>>>>>> On Thu, Aug 20, 2015 at 2:47 PM, Nirmal Fernando
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> May be we need it in both places.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 20, 2015 at 9:39 AM, Thushan Ganegedara <
>>>>>>>>> thu...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> This was put in parameters page, as it was how it was decided
>>>>>>>>>> during the discussion.
>>>>>>>>>>
>>>>>>>>>> The argument was that, the user will get a better insight to
>>>>>>>>>> changing layers sizes as necessary. However, there wasn't a strong
>>>>>>>>>> view on
>>>>>>>>>> which page we should have it.
>>>>>>>>>>
>>>>>>>>>> The reason I put it on the parameter page is that, user can
>>>>>>>>>> visualize the network and chan
Re: [Dev] [ML] Completed DL Visualization component
Hi,
Please find the screenshot for view-model attached
On Thu, Aug 20, 2015 at 10:12 PM, Nirmal Fernando wrote:
> I'd prefer at the bottom, to be consistent with other Algos.
>
> On Thu, Aug 20, 2015 at 5:40 PM, Thushan Ganegedara
> wrote:
>
>> Hello CD,
>>
>> Yes, seems there was a bit of miscommunication :)
>>
>> I'm migrating the code to that.
>>
>> Also, where do you think we should place the visualization. I'm attaching
>> a screenshot of the current view-model with red arrows for potential
>> placements.
>>
>>
>>
>> On Thu, Aug 20, 2015 at 10:07 PM, CD Athuraliya
>> wrote:
>>
>>> Hi Thushan,
>>>
>>> Now do we have the visualization in both parameters.jag and model.jag? I
>>> think Nirmal was suggesting to show it in model summery page. You can find
>>> it here analysis/view-model.jag.
>>>
>>> Regards,
>>> CD
>>>
>>> On Thu, Aug 20, 2015 at 5:29 PM, Thushan Ganegedara
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Now the visualization is included in the models.jag.
>>>>
>>>> Screenshots are attached
>>>>
>>>> On Thu, Aug 20, 2015 at 9:35 PM, Nirmal Fernando
>>>> wrote:
>>>>
>>>>> Yes, api/analyses/{analysisId}/hyperParameters
>>>>>
>>>>> On Thu, Aug 20, 2015 at 5:03 PM, Thushan Ganegedara
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> Thanks for the inputs. I'm working on that.
>>>>>>
>>>>>> By the way, is there a way to get the hyperparameter values
>>>>>> (specifically, Layer_Sizes parameter) to the model.jag page with an API
>>>>>> call? I checked the API documentation, but couldn't find one.
>>>>>>
>>>>>> On Thu, Aug 20, 2015 at 2:47 PM, Nirmal Fernando
>>>>>> wrote:
>>>>>>
>>>>>>> May be we need it in both places.
>>>>>>>
>>>>>>> On Thu, Aug 20, 2015 at 9:39 AM, Thushan Ganegedara <
>>>>>>> thu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> This was put in parameters page, as it was how it was decided
>>>>>>>> during the discussion.
>>>>>>>>
>>>>>>>> The argument was that, the user will get a better insight to
>>>>>>>> changing layers sizes as necessary. However, there wasn't a strong
>>>>>>>> view on
>>>>>>>> which page we should have it.
>>>>>>>>
>>>>>>>> The reason I put it on the parameter page is that, user can
>>>>>>>> visualize the network and change the layers sizes without navigating
>>>>>>>> between pages (as opposed to having it in the models page)
>>>>>>>>
>>>>>>>> But I'm open to suggestions
>>>>>>>>
>>>>>>>> On Thu, Aug 20, 2015 at 1:56 PM, Nirmal Fernando
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> Hi Thushan,
>>>>>>>>>
>>>>>>>>> Thanks for the update. Any special reason for having it in
>>>>>>>>> parameters page? I'd prefer to have it in the model summary page.
>>>>>>>>>
>>>>>>>>> On Thu, Aug 20, 2015 at 5:28 AM, Thushan Ganegedara <
>>>>>>>>> thu...@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hello,
>>>>>>>>>>
>>>>>>>>>> Please find an attached screen capture of the completed DL
>>>>>>>>>> visualization component
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Regards,
>>>>>>>>>>
>>>>>>>>>> Thushan Ganegedara
>>>>>>>>>> School of IT
>>>>>>>>>> University of Sydney, Australia
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>&g
Re: [Dev] [ML] Completed DL Visualization component
Hello CD,
Yes, seems there was a bit of miscommunication :)
I'm migrating the code to that.
Also, where do you think we should place the visualization. I'm attaching a
screenshot of the current view-model with red arrows for potential
placements.
On Thu, Aug 20, 2015 at 10:07 PM, CD Athuraliya wrote:
> Hi Thushan,
>
> Now do we have the visualization in both parameters.jag and model.jag? I
> think Nirmal was suggesting to show it in model summery page. You can find
> it here analysis/view-model.jag.
>
> Regards,
> CD
>
> On Thu, Aug 20, 2015 at 5:29 PM, Thushan Ganegedara
> wrote:
>
>> Hi,
>>
>> Now the visualization is included in the models.jag.
>>
>> Screenshots are attached
>>
>> On Thu, Aug 20, 2015 at 9:35 PM, Nirmal Fernando wrote:
>>
>>> Yes, api/analyses/{analysisId}/hyperParameters
>>>
>>> On Thu, Aug 20, 2015 at 5:03 PM, Thushan Ganegedara
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> Thanks for the inputs. I'm working on that.
>>>>
>>>> By the way, is there a way to get the hyperparameter values
>>>> (specifically, Layer_Sizes parameter) to the model.jag page with an API
>>>> call? I checked the API documentation, but couldn't find one.
>>>>
>>>> On Thu, Aug 20, 2015 at 2:47 PM, Nirmal Fernando
>>>> wrote:
>>>>
>>>>> May be we need it in both places.
>>>>>
>>>>> On Thu, Aug 20, 2015 at 9:39 AM, Thushan Ganegedara
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> This was put in parameters page, as it was how it was decided during
>>>>>> the discussion.
>>>>>>
>>>>>> The argument was that, the user will get a better insight to
>>>>>> changing layers sizes as necessary. However, there wasn't a strong view
>>>>>> on
>>>>>> which page we should have it.
>>>>>>
>>>>>> The reason I put it on the parameter page is that, user can visualize
>>>>>> the network and change the layers sizes without navigating between pages
>>>>>> (as opposed to having it in the models page)
>>>>>>
>>>>>> But I'm open to suggestions
>>>>>>
>>>>>> On Thu, Aug 20, 2015 at 1:56 PM, Nirmal Fernando
>>>>>> wrote:
>>>>>>
>>>>>>> Hi Thushan,
>>>>>>>
>>>>>>> Thanks for the update. Any special reason for having it in
>>>>>>> parameters page? I'd prefer to have it in the model summary page.
>>>>>>>
>>>>>>> On Thu, Aug 20, 2015 at 5:28 AM, Thushan Ganegedara <
>>>>>>> thu...@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hello,
>>>>>>>>
>>>>>>>> Please find an attached screen capture of the completed DL
>>>>>>>> visualization component
>>>>>>>>
>>>>>>>> --
>>>>>>>> Regards,
>>>>>>>>
>>>>>>>> Thushan Ganegedara
>>>>>>>> School of IT
>>>>>>>> University of Sydney, Australia
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>>
>>>>>>> Thanks & regards,
>>>>>>> Nirmal
>>>>>>>
>>>>>>> Team Lead - WSO2 Machine Learner
>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>>>>>> Mobile: +94715779733
>>>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Thushan Ganegedara
>>>>>> School of IT
>>>>>> University of Sydney, Australia
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks & regards,
>>>>> Nirmal
>>>>>
>>>>> Team Lead - WSO2 Machine Learner
>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>>>> Mobile: +94715779733
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Thushan Ganegedara
>>>> School of IT
>>>> University of Sydney, Australia
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks & regards,
>>> Nirmal
>>>
>>> Team Lead - WSO2 Machine Learner
>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>> Mobile: +94715779733
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Thushan Ganegedara
>> School of IT
>> University of Sydney, Australia
>>
>
>
>
> --
> *CD Athuraliya*
> Software Engineer
> WSO2, Inc.
> lean . enterprise . middleware
> Mobile: +94 716288847 <94716288847>
> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter
> <https://twitter.com/cdathuraliya> | Blog
> <http://cdathuraliya.tumblr.com/>
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Completed DL Visualization component
Hi,
Now the visualization is included in the models.jag.
Screenshots are attached
On Thu, Aug 20, 2015 at 9:35 PM, Nirmal Fernando wrote:
> Yes, api/analyses/{analysisId}/hyperParameters
>
> On Thu, Aug 20, 2015 at 5:03 PM, Thushan Ganegedara
> wrote:
>
>> Hi,
>>
>> Thanks for the inputs. I'm working on that.
>>
>> By the way, is there a way to get the hyperparameter values
>> (specifically, Layer_Sizes parameter) to the model.jag page with an API
>> call? I checked the API documentation, but couldn't find one.
>>
>> On Thu, Aug 20, 2015 at 2:47 PM, Nirmal Fernando wrote:
>>
>>> May be we need it in both places.
>>>
>>> On Thu, Aug 20, 2015 at 9:39 AM, Thushan Ganegedara
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> This was put in parameters page, as it was how it was decided during
>>>> the discussion.
>>>>
>>>> The argument was that, the user will get a better insight to changing
>>>> layers sizes as necessary. However, there wasn't a strong view on which
>>>> page we should have it.
>>>>
>>>> The reason I put it on the parameter page is that, user can visualize
>>>> the network and change the layers sizes without navigating between pages
>>>> (as opposed to having it in the models page)
>>>>
>>>> But I'm open to suggestions
>>>>
>>>> On Thu, Aug 20, 2015 at 1:56 PM, Nirmal Fernando
>>>> wrote:
>>>>
>>>>> Hi Thushan,
>>>>>
>>>>> Thanks for the update. Any special reason for having it in parameters
>>>>> page? I'd prefer to have it in the model summary page.
>>>>>
>>>>> On Thu, Aug 20, 2015 at 5:28 AM, Thushan Ganegedara
>>>>> wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> Please find an attached screen capture of the completed DL
>>>>>> visualization component
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Thushan Ganegedara
>>>>>> School of IT
>>>>>> University of Sydney, Australia
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks & regards,
>>>>> Nirmal
>>>>>
>>>>> Team Lead - WSO2 Machine Learner
>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>>>> Mobile: +94715779733
>>>>> Blog: http://nirmalfdo.blogspot.com/
>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Thushan Ganegedara
>>>> School of IT
>>>> University of Sydney, Australia
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks & regards,
>>> Nirmal
>>>
>>> Team Lead - WSO2 Machine Learner
>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>> Mobile: +94715779733
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>>
>> Thushan Ganegedara
>> School of IT
>> University of Sydney, Australia
>>
>
>
>
> --
>
> Thanks & regards,
> Nirmal
>
> Team Lead - WSO2 Machine Learner
> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
> Mobile: +94715779733
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Completed DL Visualization component
Hi, Thanks for the inputs. I'm working on that. By the way, is there a way to get the hyperparameter values (specifically, Layer_Sizes parameter) to the model.jag page with an API call? I checked the API documentation, but couldn't find one. On Thu, Aug 20, 2015 at 2:47 PM, Nirmal Fernando wrote: > May be we need it in both places. > > On Thu, Aug 20, 2015 at 9:39 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> This was put in parameters page, as it was how it was decided during the >> discussion. >> >> The argument was that, the user will get a better insight to changing >> layers sizes as necessary. However, there wasn't a strong view on which >> page we should have it. >> >> The reason I put it on the parameter page is that, user can visualize the >> network and change the layers sizes without navigating between pages (as >> opposed to having it in the models page) >> >> But I'm open to suggestions >> >> On Thu, Aug 20, 2015 at 1:56 PM, Nirmal Fernando wrote: >> >>> Hi Thushan, >>> >>> Thanks for the update. Any special reason for having it in parameters >>> page? I'd prefer to have it in the model summary page. >>> >>> On Thu, Aug 20, 2015 at 5:28 AM, Thushan Ganegedara >>> wrote: >>> >>>> Hello, >>>> >>>> Please find an attached screen capture of the completed DL >>>> visualization component >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Completed DL Visualization component
Hi, This was put in parameters page, as it was how it was decided during the discussion. The argument was that, the user will get a better insight to changing layers sizes as necessary. However, there wasn't a strong view on which page we should have it. The reason I put it on the parameter page is that, user can visualize the network and change the layers sizes without navigating between pages (as opposed to having it in the models page) But I'm open to suggestions On Thu, Aug 20, 2015 at 1:56 PM, Nirmal Fernando wrote: > Hi Thushan, > > Thanks for the update. Any special reason for having it in parameters > page? I'd prefer to have it in the model summary page. > > On Thu, Aug 20, 2015 at 5:28 AM, Thushan Ganegedara > wrote: > >> Hello, >> >> Please find an attached screen capture of the completed DL visualization >> component >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Completed DL Visualization component
Hello, Please find an attached screen capture of the completed DL visualization component -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Getting the number of output classes
Also, I got an implementation concern as well,
I have the following ajax calls taking place,
getting number of input features,
getting the response variable
getting the number of output classes using the response variable
then executing the visualization script
Should I use nested ajax calls?
On Wed, Aug 19, 2015 at 11:32 PM, Thushan Ganegedara
wrote:
> Oh, makes sense now.
>
> I need to know the number of output classes. I got it to work.
>
> Thanks
>
> On Wed, Aug 19, 2015 at 11:28 PM, Nirmal Fernando wrote:
>
>> Thushan,
>>
>> What's the feature name? 'Class' was a sample.
>>
>> https://localhost:9443/api/datasets/1/stats?feature=
>> <https://localhost:9443/api/datasets/1/stats?feature=Class>
>>
>>
>> On Wed, Aug 19, 2015 at 6:53 PM, Thushan Ganegedara
>> wrote:
>>
>>> Hi,
>>>
>>> I tried the api call, however, I got the following error
>>>
>>> Error occurred while retrieving summarized stats of feature [name] Class
>>> for the dataset [id] 2 of tenant [id] -1234 and [user] admin . Cause:
>>> org.wso2.carbon.ml.core.exceptions.MLDataProcessingException - An error
>>> occurred while retrieving summary statistics for the dataset: 2: No data is
>>> available [2000-140]
>>>
>>>
>>> On Wed, Aug 19, 2015 at 10:43 PM, Thushan Ganegedara
>>> wrote:
>>>
>>>> Hi CD,
>>>>
>>>> Thanks a lot.
>>>>
>>>> On Wed, Aug 19, 2015 at 10:41 PM, CD Athuraliya
>>>> wrote:
>>>>
>>>>> Hi Thushan,
>>>>>
>>>>> You can retrieve the number of output classes (or class labels of
>>>>> response variable) via this [1] API (e.g.
>>>>> https://localhost:9443/api/datasets/1/stats?feature=Class). Number of
>>>>> unique labels in that feature values are given in the JSON response like
>>>>> below.
>>>>>
>>>>> [
>>>>> {
>>>>> "missing": 0,
>>>>> "median": "0",
>>>>> "values": [
>>>>> [
>>>>> "0",
>>>>> 500
>>>>> ],
>>>>> [
>>>>> "1",
>>>>> 268
>>>>> ]
>>>>> ],
>>>>> *"unique": 2,*
>>>>> "std": "0.477",
>>>>> "mean": "0.349",
>>>>> "bar": true,
>>>>> "key": "Frequency"
>>>>> }
>>>>> ]
>>>>>
>>>>> [1]
>>>>> https://docs.wso2.com/display/ML100/Retrieve+summarized+statistics+of+a+feature+in+a+dataset+via+REST+API
>>>>>
>>>>> Regards,
>>>>> CD
>>>>>
>>>>> On Wed, Aug 19, 2015 at 5:47 PM, Thushan Ganegedara
>>>>> wrote:
>>>>>
>>>>>> Hi all,
>>>>>>
>>>>>> The D3 visualization component is almost ready. I've to tweak few
>>>>>> things. I'm attaching an images of what it looks like. Also there's a
>>>>>> button for updating the visualization whenever the Layer Sizes input text
>>>>>> is changed
>>>>>>
>>>>>> Could someone please tell me a way to obtain the number of classes
>>>>>> with an ajax call maybe? I could obtain the input feature count but not
>>>>>> the
>>>>>> output class count.
>>>>>>
>>>>>> --
>>>>>> Regards,
>>>>>>
>>>>>> Thushan Ganegedara
>>>>>> School of IT
>>>>>> University of Sydney, Australia
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> *CD Athuraliya*
>>>>> Software Engineer
>>>>> WSO2, Inc.
>>>>> lean . enterprise . middleware
>>>>> Mobile: +94 716288847 <94716288847>
>>>>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter
>>>>> <https://twitter.com/cdathuraliya> | Blog
>>>>> <http://cdathuraliya.tumblr.com/>
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Thushan Ganegedara
>>>> School of IT
>>>> University of Sydney, Australia
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Thushan Ganegedara
>>> School of IT
>>> University of Sydney, Australia
>>>
>>
>>
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Team Lead - WSO2 Machine Learner
>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>> Mobile: +94715779733
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
>
>
> --
> Regards,
>
> Thushan Ganegedara
> School of IT
> University of Sydney, Australia
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Getting the number of output classes
Oh, makes sense now.
I need to know the number of output classes. I got it to work.
Thanks
On Wed, Aug 19, 2015 at 11:28 PM, Nirmal Fernando wrote:
> Thushan,
>
> What's the feature name? 'Class' was a sample.
>
> https://localhost:9443/api/datasets/1/stats?feature=
> <https://localhost:9443/api/datasets/1/stats?feature=Class>
>
>
> On Wed, Aug 19, 2015 at 6:53 PM, Thushan Ganegedara
> wrote:
>
>> Hi,
>>
>> I tried the api call, however, I got the following error
>>
>> Error occurred while retrieving summarized stats of feature [name] Class for
>> the dataset [id] 2 of tenant [id] -1234 and [user] admin . Cause:
>> org.wso2.carbon.ml.core.exceptions.MLDataProcessingException - An error
>> occurred while retrieving summary statistics for the dataset: 2: No data is
>> available [2000-140]
>>
>>
>> On Wed, Aug 19, 2015 at 10:43 PM, Thushan Ganegedara
>> wrote:
>>
>>> Hi CD,
>>>
>>> Thanks a lot.
>>>
>>> On Wed, Aug 19, 2015 at 10:41 PM, CD Athuraliya
>>> wrote:
>>>
>>>> Hi Thushan,
>>>>
>>>> You can retrieve the number of output classes (or class labels of
>>>> response variable) via this [1] API (e.g.
>>>> https://localhost:9443/api/datasets/1/stats?feature=Class). Number of
>>>> unique labels in that feature values are given in the JSON response like
>>>> below.
>>>>
>>>> [
>>>> {
>>>> "missing": 0,
>>>> "median": "0",
>>>> "values": [
>>>> [
>>>> "0",
>>>> 500
>>>> ],
>>>> [
>>>> "1",
>>>> 268
>>>> ]
>>>> ],
>>>> *"unique": 2,*
>>>> "std": "0.477",
>>>> "mean": "0.349",
>>>> "bar": true,
>>>> "key": "Frequency"
>>>> }
>>>> ]
>>>>
>>>> [1]
>>>> https://docs.wso2.com/display/ML100/Retrieve+summarized+statistics+of+a+feature+in+a+dataset+via+REST+API
>>>>
>>>> Regards,
>>>> CD
>>>>
>>>> On Wed, Aug 19, 2015 at 5:47 PM, Thushan Ganegedara
>>>> wrote:
>>>>
>>>>> Hi all,
>>>>>
>>>>> The D3 visualization component is almost ready. I've to tweak few
>>>>> things. I'm attaching an images of what it looks like. Also there's a
>>>>> button for updating the visualization whenever the Layer Sizes input text
>>>>> is changed
>>>>>
>>>>> Could someone please tell me a way to obtain the number of classes
>>>>> with an ajax call maybe? I could obtain the input feature count but not
>>>>> the
>>>>> output class count.
>>>>>
>>>>> --
>>>>> Regards,
>>>>>
>>>>> Thushan Ganegedara
>>>>> School of IT
>>>>> University of Sydney, Australia
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> *CD Athuraliya*
>>>> Software Engineer
>>>> WSO2, Inc.
>>>> lean . enterprise . middleware
>>>> Mobile: +94 716288847 <94716288847>
>>>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter
>>>> <https://twitter.com/cdathuraliya> | Blog
>>>> <http://cdathuraliya.tumblr.com/>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Thushan Ganegedara
>>> School of IT
>>> University of Sydney, Australia
>>>
>>
>>
>>
>> --
>> Regards,
>>
>> Thushan Ganegedara
>> School of IT
>> University of Sydney, Australia
>>
>
>
>
> --
>
> Thanks & regards,
> Nirmal
>
> Team Lead - WSO2 Machine Learner
> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
> Mobile: +94715779733
> Blog: http://nirmalfdo.blogspot.com/
>
>
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Getting the number of output classes
Hi,
I tried the api call, however, I got the following error
Error occurred while retrieving summarized stats of feature [name]
Class for the dataset [id] 2 of tenant [id] -1234 and [user] admin .
Cause: org.wso2.carbon.ml.core.exceptions.MLDataProcessingException -
An error occurred while retrieving summary statistics for the dataset:
2: No data is available [2000-140]
On Wed, Aug 19, 2015 at 10:43 PM, Thushan Ganegedara
wrote:
> Hi CD,
>
> Thanks a lot.
>
> On Wed, Aug 19, 2015 at 10:41 PM, CD Athuraliya
> wrote:
>
>> Hi Thushan,
>>
>> You can retrieve the number of output classes (or class labels of
>> response variable) via this [1] API (e.g.
>> https://localhost:9443/api/datasets/1/stats?feature=Class). Number of
>> unique labels in that feature values are given in the JSON response like
>> below.
>>
>> [
>> {
>> "missing": 0,
>> "median": "0",
>> "values": [
>> [
>> "0",
>> 500
>> ],
>> [
>> "1",
>> 268
>> ]
>> ],
>> *"unique": 2,*
>> "std": "0.477",
>> "mean": "0.349",
>> "bar": true,
>> "key": "Frequency"
>> }
>> ]
>>
>> [1]
>> https://docs.wso2.com/display/ML100/Retrieve+summarized+statistics+of+a+feature+in+a+dataset+via+REST+API
>>
>> Regards,
>> CD
>>
>> On Wed, Aug 19, 2015 at 5:47 PM, Thushan Ganegedara
>> wrote:
>>
>>> Hi all,
>>>
>>> The D3 visualization component is almost ready. I've to tweak few
>>> things. I'm attaching an images of what it looks like. Also there's a
>>> button for updating the visualization whenever the Layer Sizes input text
>>> is changed
>>>
>>> Could someone please tell me a way to obtain the number of classes with
>>> an ajax call maybe? I could obtain the input feature count but not the
>>> output class count.
>>>
>>> --
>>> Regards,
>>>
>>> Thushan Ganegedara
>>> School of IT
>>> University of Sydney, Australia
>>>
>>
>>
>>
>> --
>> *CD Athuraliya*
>> Software Engineer
>> WSO2, Inc.
>> lean . enterprise . middleware
>> Mobile: +94 716288847 <94716288847>
>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter
>> <https://twitter.com/cdathuraliya> | Blog
>> <http://cdathuraliya.tumblr.com/>
>>
>
>
>
> --
> Regards,
>
> Thushan Ganegedara
> School of IT
> University of Sydney, Australia
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Getting the number of output classes
Hi CD,
Thanks a lot.
On Wed, Aug 19, 2015 at 10:41 PM, CD Athuraliya wrote:
> Hi Thushan,
>
> You can retrieve the number of output classes (or class labels of response
> variable) via this [1] API (e.g.
> https://localhost:9443/api/datasets/1/stats?feature=Class). Number of
> unique labels in that feature values are given in the JSON response like
> below.
>
> [
> {
> "missing": 0,
> "median": "0",
> "values": [
> [
> "0",
> 500
> ],
> [
> "1",
> 268
> ]
> ],
> *"unique": 2,*
> "std": "0.477",
> "mean": "0.349",
> "bar": true,
> "key": "Frequency"
> }
> ]
>
> [1]
> https://docs.wso2.com/display/ML100/Retrieve+summarized+statistics+of+a+feature+in+a+dataset+via+REST+API
>
> Regards,
> CD
>
> On Wed, Aug 19, 2015 at 5:47 PM, Thushan Ganegedara
> wrote:
>
>> Hi all,
>>
>> The D3 visualization component is almost ready. I've to tweak few things.
>> I'm attaching an images of what it looks like. Also there's a button for
>> updating the visualization whenever the Layer Sizes input text is changed
>>
>> Could someone please tell me a way to obtain the number of classes with
>> an ajax call maybe? I could obtain the input feature count but not the
>> output class count.
>>
>> --
>> Regards,
>>
>> Thushan Ganegedara
>> School of IT
>> University of Sydney, Australia
>>
>
>
>
> --
> *CD Athuraliya*
> Software Engineer
> WSO2, Inc.
> lean . enterprise . middleware
> Mobile: +94 716288847 <94716288847>
> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter
> <https://twitter.com/cdathuraliya> | Blog
> <http://cdathuraliya.tumblr.com/>
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Getting the number of output classes
Hi all, The D3 visualization component is almost ready. I've to tweak few things. I'm attaching an images of what it looks like. Also there's a button for updating the visualization whenever the Layer Sizes input text is changed Could someone please tell me a way to obtain the number of classes with an ajax call maybe? I could obtain the input feature count but not the output class count. -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] DL Visualization component for parameter.jag
Hello CD, Okey, it makes sense now. Thank you On Tue, Aug 18, 2015 at 2:03 PM, CD Athuraliya wrote: > Hi Thushan, > > Those are Bootstrap classes. We use Bootstrap [1] for styling. If you want > to add custom styling you can add another class separated with a space > (e.g. "container col-md-12 col-centered wr-content my-style") > > [1] http://getbootstrap.com/ > > Regards, > CD > > On Tue, Aug 18, 2015 at 9:22 AM, Thushan Ganegedara > wrote: > >> Thanks CD >> >> However, what I meant was that this. For example in the parameter.jag, >> there's a div with class name "container col-md-12 col-centered >> wr-content" >> >> What does "col-md-12" and "wr-content" mean? >> >> On Tue, Aug 18, 2015 at 1:42 PM, CD Athuraliya >> wrote: >> >>> Hi Thushan, >>> >>> We use lowercase names for id, class attributes. If there are two or >>> more words separate them by a hyphen (e.g. graph-header). >>> >>> Regards, >>> CD >>> >>> On Tue, Aug 18, 2015 at 5:46 AM, Thushan Ganegedara >>> wrote: >>> >>>> Hello CD, >>>> >>>> Thank you very much for pointing me to the resources. So far it seems >>>> quite manageable without any additional libraries. But I'll keep an eye on >>>> them. >>>> >>>> Yes, as you mentioned, it is imperative to keep element size >>>> proportional to actual layer sizes. I'm designing it that way. Furthermore, >>>> I've introduced max and min threshold to make sure the elements doesn't >>>> grow unnecessarily big or shrink too much. >>>> >>>> I've one question though. What is the naming convention used for div >>>> class attributes in jag file? >>>> >>>> On Tue, Aug 18, 2015 at 2:52 AM, CD Athuraliya >>>> wrote: >>>> >>>>> I think this visualization can be made more intuitive by making >>>>> element sizes proportionate to network elements (e.g. no. of nodes, input >>>>> feature vector size). A library like dagre-d3 [1][2][3] might make things >>>>> easier. :) >>>>> >>>>> [1] https://github.com/cpettitt/dagre-d3 >>>>> [2] >>>>> http://cpettitt.github.io/project/dagre-d3/latest/demo/interactive-demo.html >>>>> [3] >>>>> http://cpettitt.github.io/project/dagre-d3/latest/demo/sentence-tokenization.html >>>>> >>>>> On Mon, Aug 17, 2015 at 4:38 PM, Nirmal Fernando >>>>> wrote: >>>>> >>>>>> Looks good for me. >>>>>> >>>>>> On Mon, Aug 17, 2015 at 3:56 PM, Thushan Ganegedara >>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'm currently designing the a small visualization component for DL >>>>>>> which will be displayed on the parameter.jag >>>>>>> >>>>>>> I'm thinking of a very simple visualization as shown in the attached >>>>>>> image. I'm not very aware of the capabilities of D3 but I think this is >>>>>>> attainable. >>>>>>> >>>>>>> I would like to know your thoughts on this. >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Thushan Ganegedara >>>>>>> School of IT >>>>>>> University of Sydney, Australia >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>>> ___ >>>>>> Dev mailing list >>>>>> Dev@wso2.org >>>>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *CD Athuraliya* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> lean . enterprise . middleware >>>>> Mobile: +94 716288847 <94716288847> >>>>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter >>>>> <https://twitter.com/cdathuraliya> | Blog >>>>> <http://cdathuraliya.tumblr.com/> >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> *CD Athuraliya* >>> Software Engineer >>> WSO2, Inc. >>> lean . enterprise . middleware >>> Mobile: +94 716288847 <94716288847> >>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter >>> <https://twitter.com/cdathuraliya> | Blog >>> <http://cdathuraliya.tumblr.com/> >>> >> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <http://cdathuraliya.tumblr.com/> > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] DL Visualization component for parameter.jag
Thanks CD However, what I meant was that this. For example in the parameter.jag, there's a div with class name "container col-md-12 col-centered wr-content" What does "col-md-12" and "wr-content" mean? On Tue, Aug 18, 2015 at 1:42 PM, CD Athuraliya wrote: > Hi Thushan, > > We use lowercase names for id, class attributes. If there are two or more > words separate them by a hyphen (e.g. graph-header). > > Regards, > CD > > On Tue, Aug 18, 2015 at 5:46 AM, Thushan Ganegedara > wrote: > >> Hello CD, >> >> Thank you very much for pointing me to the resources. So far it seems >> quite manageable without any additional libraries. But I'll keep an eye on >> them. >> >> Yes, as you mentioned, it is imperative to keep element size proportional >> to actual layer sizes. I'm designing it that way. Furthermore, I've >> introduced max and min threshold to make sure the elements doesn't grow >> unnecessarily big or shrink too much. >> >> I've one question though. What is the naming convention used for div >> class attributes in jag file? >> >> On Tue, Aug 18, 2015 at 2:52 AM, CD Athuraliya >> wrote: >> >>> I think this visualization can be made more intuitive by making element >>> sizes proportionate to network elements (e.g. no. of nodes, input feature >>> vector size). A library like dagre-d3 [1][2][3] might make things easier. :) >>> >>> [1] https://github.com/cpettitt/dagre-d3 >>> [2] >>> http://cpettitt.github.io/project/dagre-d3/latest/demo/interactive-demo.html >>> [3] >>> http://cpettitt.github.io/project/dagre-d3/latest/demo/sentence-tokenization.html >>> >>> On Mon, Aug 17, 2015 at 4:38 PM, Nirmal Fernando >>> wrote: >>> >>>> Looks good for me. >>>> >>>> On Mon, Aug 17, 2015 at 3:56 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I'm currently designing the a small visualization component for DL >>>>> which will be displayed on the parameter.jag >>>>> >>>>> I'm thinking of a very simple visualization as shown in the attached >>>>> image. I'm not very aware of the capabilities of D3 but I think this is >>>>> attainable. >>>>> >>>>> I would like to know your thoughts on this. >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Thushan Ganegedara >>>>> School of IT >>>>> University of Sydney, Australia >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Team Lead - WSO2 Machine Learner >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>>> _______ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>> >>> >>> -- >>> *CD Athuraliya* >>> Software Engineer >>> WSO2, Inc. >>> lean . enterprise . middleware >>> Mobile: +94 716288847 <94716288847> >>> LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter >>> <https://twitter.com/cdathuraliya> | Blog >>> <http://cdathuraliya.tumblr.com/> >>> >> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <http://cdathuraliya.tumblr.com/> > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] DL Visualization component for parameter.jag
Hello CD, Thank you very much for pointing me to the resources. So far it seems quite manageable without any additional libraries. But I'll keep an eye on them. Yes, as you mentioned, it is imperative to keep element size proportional to actual layer sizes. I'm designing it that way. Furthermore, I've introduced max and min threshold to make sure the elements doesn't grow unnecessarily big or shrink too much. I've one question though. What is the naming convention used for div class attributes in jag file? On Tue, Aug 18, 2015 at 2:52 AM, CD Athuraliya wrote: > I think this visualization can be made more intuitive by making element > sizes proportionate to network elements (e.g. no. of nodes, input feature > vector size). A library like dagre-d3 [1][2][3] might make things easier. :) > > [1] https://github.com/cpettitt/dagre-d3 > [2] > http://cpettitt.github.io/project/dagre-d3/latest/demo/interactive-demo.html > [3] > http://cpettitt.github.io/project/dagre-d3/latest/demo/sentence-tokenization.html > > On Mon, Aug 17, 2015 at 4:38 PM, Nirmal Fernando wrote: > >> Looks good for me. >> >> On Mon, Aug 17, 2015 at 3:56 PM, Thushan Ganegedara >> wrote: >> >>> Hi all, >>> >>> I'm currently designing the a small visualization component for DL which >>> will be displayed on the parameter.jag >>> >>> I'm thinking of a very simple visualization as shown in the attached >>> image. I'm not very aware of the capabilities of D3 but I think this is >>> attainable. >>> >>> I would like to know your thoughts on this. >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <http://cdathuraliya.tumblr.com/> > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] DL Visualization component for parameter.jag
Hi all, I'm currently designing the a small visualization component for DL which will be displayed on the parameter.jag I'm thinking of a very simple visualization as shown in the attached image. I'm not very aware of the capabilities of D3 but I think this is attainable. I would like to know your thoughts on this. -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Progress with Deeplearning Integration
Hi all, Below is a progress report on the Deeplearning integration The previous two weeks I mainly dedicated for testing what is already implemented for errors and potential enhancements. *Getting the deeplearning component to work* After rigorously testing the deeplearning component, it seems to be working well with all the datasets tested so far. There were few bugs in the implementation (i.e. onExBarrierCompletion error, conversion to enum) which have been resolved with the help of WSO2 team and H-2-O team. *Testing the deeplearning component* So far DL component has been tested against 4 datasets. They are, *Iris, MNIST, Leaf, Action Recognition* Deeplearning has shown remarkable performance with all the datasets except for the leaf dataset. (The performance for the leaf dataset does not exceed the Linear Reg - L-BFGS by a lot) I speculate this could be due to a linearity in data (so the model is too complicated). For all the other datasets, DL reaches 95%+ without much effort *Fixing minor bugs and Enhancements* Few minor bugs were fixed such as, Input sanitization error for Stacked Autoencoders Code formatting and changes suggested in the review Enhancements include, Introducing activation function selection -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Categorical or Numerical column?
could >>>>>>> come-up with. I did a bit of research on this too, but only to find no >>>>>>> other solution :( >>>>>>> >>>>>>> Thanks, >>>>>>> Supun >>>>>>> >>>>>>> On Thu, Aug 13, 2015 at 1:49 AM, Nirmal Fernando >>>>>>> wrote: >>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> We have a feature in ML where we suggest a given data column of a >>>>>>>> dataset is categorical or numerical. Currently, how we determine this >>>>>>>> is by >>>>>>>> using a threshold value (The maximum number of categories that can >>>>>>>> have in a non-string categorical feature. If exceeds, the feature >>>>>>>> will be treated as a numerical feature.). But this is not a >>>>>>>> successful measurement for most of the datasets. >>>>>>>> >>>>>>>> Can we use 'skewness' of a distribution as a measurement to >>>>>>>> determine this? Can we say, a column is numerical, if the modulus of >>>>>>>> the >>>>>>>> skewness of the distribution is less than a certain threshold (say >>>>>>>> 0.01) ? >>>>>>>> >>>>>>>> *References*: >>>>>>>> >>>>>>>> http://www.itrcweb.org/gsmc-1/Content/GW%20Stats/5%20Methods%20in%20indiv%20Topics/5%206%20Distributional%20Tests.htm >>>>>>>> >>>>>>>> -- >>>>>>>> >>>>>>>> Thanks & regards, >>>>>>>> Nirmal >>>>>>>> >>>>>>>> Team Lead - WSO2 Machine Learner >>>>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>>>> Mobile: +94715779733 >>>>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> *Supun Sethunga* >>>>>>> Software Engineer >>>>>>> WSO2, Inc. >>>>>>> http://wso2.com/ >>>>>>> lean | enterprise | middleware >>>>>>> Mobile : +94 716546324 >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> >>>>>> Thanks & regards, >>>>>> Nirmal >>>>>> >>>>>> Team Lead - WSO2 Machine Learner >>>>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>>>> Mobile: +94715779733 >>>>>> Blog: http://nirmalfdo.blogspot.com/ >>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>>> -- >>>>> *Supun Sethunga* >>>>> Software Engineer >>>>> WSO2, Inc. >>>>> http://wso2.com/ >>>>> lean | enterprise | middleware >>>>> Mobile : +94 716546324 >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>> Site: http://people.apache.org/~hemapani/ >>>> Photos: http://www.flickr.com/photos/hemapani/ >>>> Phone: 0772360902 >>>> >>> >>> >>> >>> -- >>> >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://people.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> Pruthuvi Maheshakya Wijewardena >> Software Engineer >> WSO2 : http://wso2.com/ >> Email: mahesha...@wso2.com >> Mobile: +94711228855 >> >> >> > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Issue while loading the leaf dataset (misreading classes)
Moreover, I think a hybrid approach as follows might work well. 1. Select a sample 2. Filter columns by the data type and find potential categorical variables (integer / string) 3. Filter further by checking if same values are repeated multiple times in the dataset. On Fri, Aug 14, 2015 at 2:48 PM, Thushan Ganegedara wrote: > Hi, > > Yes, no mater which approach used, there's always going to be outliers > which does not fit the defined rules. But for these corner cases, user > always have to opportunity to change the variable to numerical. > > One more approach is to introduce a measure of replication of values in a > column. If the column shows a repetition of same values many times, imo, it > is a good indicator for detecting categorical variable. > > On Fri, Aug 14, 2015 at 2:41 PM, Nirmal Fernando wrote: > >> >> >> On Fri, Aug 14, 2015 at 10:01 AM, Thushan Ganegedara >> wrote: >> >>> Hi, >>> >>> This was mainly due to the detection of a numerical feature as a >>> categorical one. >>> Oh, it makes sense now. Why don't we try taking a sample of data and if >>> the sample contains only integers (or doubles without any decimals) or >>> strings, consider it as a categorical variable. >>> >> >> I tried that approach too, but there're some datasets like automobile >> dataset normalized-losses feature, which has integer values (0-164) but >> which is probably not categorical. >> >>> >>> We suggested increasing the categorical threshold as a work-around. >>> @thushan did it work? >>> Yes, it worked. After increasing the threshold to 40. >>> >>> On Fri, Aug 14, 2015 at 2:21 PM, Nirmal Fernando >>> wrote: >>> >>>> This was mainly due to the detection of a numerical feature as a >>>> categorical one. >>>> >>>> We suggested increasing the categorical threshold as a work-around. >>>> @thushan did it work? >>>> >>>> On Tue, Aug 11, 2015 at 5:50 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> This issue occurs, if I turn the response variable to a categorical >>>>> variable. If I get the variable as a numerical variable, the values are >>>>> read correctly. >>>>> >>>>> So I presume there is a fault in categorical conversion of the >>>>> variable. >>>>> >>>>> On Tue, Aug 11, 2015 at 7:11 PM, Thushan Ganegedara >>>>> wrote: >>>>> >>>>>> I still get the same result >>>>>> >>>>>> 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >>>>>> 1.0 1.0 1.0 1.0 12.012.012.012.012.0 >>>>>> 12.012.012.012.012.013.013.013.013.0 >>>>>> 13.013.0 >>>>>> 13.013.013.013.014.014.014.014.0 >>>>>> 14.014.014.014.015.015.015.015.015.0 >>>>>> 15.015.015.015.015.015.015.016.016.0 >>>>>> 16.016.0 >>>>>> 16.016.016.016.017.017.017.017.0 >>>>>> 17.017.017.017.017.017.018.018.018.0 >>>>>> 18.018.018.018.018.018.018.018.019.0 >>>>>> 19.019.0 >>>>>> 19.019.019.019.019.019.019.019.0 >>>>>> 19.019.019.02.0 2.0 2.0 2.0 2.0 2.0 >>>>>> 2.0 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 >>>>>> 3.0 3.0 >>>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>>> 3.0 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 >>>>>> 4.0 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 >>>>>> 5.0 5.0 >>>>>> 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 >>>>>> 5.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 >>>>>> 6.0 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 >>>>>> 7.0 7.0 >>>>>> 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 >>>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >&
Re: [Dev] [ML] Issue while loading the leaf dataset (misreading classes)
Hi, Yes, no mater which approach used, there's always going to be outliers which does not fit the defined rules. But for these corner cases, user always have to opportunity to change the variable to numerical. One more approach is to introduce a measure of replication of values in a column. If the column shows a repetition of same values many times, imo, it is a good indicator for detecting categorical variable. On Fri, Aug 14, 2015 at 2:41 PM, Nirmal Fernando wrote: > > > On Fri, Aug 14, 2015 at 10:01 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> This was mainly due to the detection of a numerical feature as a >> categorical one. >> Oh, it makes sense now. Why don't we try taking a sample of data and if >> the sample contains only integers (or doubles without any decimals) or >> strings, consider it as a categorical variable. >> > > I tried that approach too, but there're some datasets like automobile > dataset normalized-losses feature, which has integer values (0-164) but > which is probably not categorical. > >> >> We suggested increasing the categorical threshold as a work-around. >> @thushan did it work? >> Yes, it worked. After increasing the threshold to 40. >> >> On Fri, Aug 14, 2015 at 2:21 PM, Nirmal Fernando wrote: >> >>> This was mainly due to the detection of a numerical feature as a >>> categorical one. >>> >>> We suggested increasing the categorical threshold as a work-around. >>> @thushan did it work? >>> >>> On Tue, Aug 11, 2015 at 5:50 PM, Thushan Ganegedara >>> wrote: >>> >>>> This issue occurs, if I turn the response variable to a categorical >>>> variable. If I get the variable as a numerical variable, the values are >>>> read correctly. >>>> >>>> So I presume there is a fault in categorical conversion of the variable. >>>> >>>> On Tue, Aug 11, 2015 at 7:11 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> I still get the same result >>>>> >>>>> 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >>>>> 1.0 1.0 1.0 1.0 12.012.012.012.012.0 >>>>> 12.012.012.012.012.013.013.013.013.0 >>>>> 13.013.0 >>>>> 13.013.013.013.014.014.014.014.0 >>>>> 14.014.014.014.015.015.015.015.015.0 >>>>> 15.015.015.015.015.015.015.016.016.0 >>>>> 16.016.0 >>>>> 16.016.016.016.017.017.017.017.0 >>>>> 17.017.017.017.017.017.018.018.018.0 >>>>> 18.018.018.018.018.018.018.018.019.0 >>>>> 19.019.0 >>>>> 19.019.019.019.019.019.019.019.0 >>>>> 19.019.019.02.0 2.0 2.0 2.0 2.0 2.0 >>>>> 2.0 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 >>>>> 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 >>>>> 4.0 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 >>>>> 5.0 5.0 >>>>> 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 >>>>> 5.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 >>>>> 6.0 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 >>>>> 7.0 7.0 >>>>> 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>> 3.0 3.0 >>>>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>>>
Re: [Dev] [ML] Issue while loading the leaf dataset (misreading classes)
Hi, This was mainly due to the detection of a numerical feature as a categorical one. Oh, it makes sense now. Why don't we try taking a sample of data and if the sample contains only integers (or doubles without any decimals) or strings, consider it as a categorical variable. We suggested increasing the categorical threshold as a work-around. @thushan did it work? Yes, it worked. After increasing the threshold to 40. On Fri, Aug 14, 2015 at 2:21 PM, Nirmal Fernando wrote: > This was mainly due to the detection of a numerical feature as a > categorical one. > > We suggested increasing the categorical threshold as a work-around. > @thushan did it work? > > On Tue, Aug 11, 2015 at 5:50 PM, Thushan Ganegedara > wrote: > >> This issue occurs, if I turn the response variable to a categorical >> variable. If I get the variable as a numerical variable, the values are >> read correctly. >> >> So I presume there is a fault in categorical conversion of the variable. >> >> On Tue, Aug 11, 2015 at 7:11 PM, Thushan Ganegedara >> wrote: >> >>> I still get the same result >>> >>> 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >>> 1.0 1.0 1.0 12.012.012.012.012.012.0 >>> 12.012.012.012.013.013.013.013.013.013.0 >>> 13.013.013.013.014.014.014.014.014.0 >>> 14.014.014.015.015.015.015.015.015.0 >>> 15.015.015.015.015.015.016.016.016.016.0 >>> 16.016.016.016.017.017.017.017.017.0 >>> 17.017.017.017.017.018.018.018.018.0 >>> 18.018.018.018.018.018.018.019.019.019.0 >>> 19.019.019.019.019.019.019.019.019.0 >>> 19.019.02.0 2.0 2.0 2.0 2.0 2.0 2.0 >>> 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 4.0 >>> 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 5.0 5.0 >>> 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 >>> 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 >>> 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 >>> 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >>> 3.0 3.0 3.0 3.0 >>> >>> On Tue, Aug 11, 2015 at 7:05 PM, Nirmal Fernando >>> wrote: >>> >>>> Can you use following code and try; >>>> >>>> List points = labeledPoints.collect(); >>>> for(int i=0;i>>> System.out.print(points.get(i).label() + "\t"); >>>> } >>>> >>>> On Tue, Aug 11, 2015 at 2:30 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> I used the following snippet >>>>> >>>>> for(int i=0;i>>>> System.out.print(labeledPoints.collect().get(i).label() + >>>>> "\t"); >>&g
Re: [Dev] [ML] Categorical or Numerical column?
Hi all, To add to what Supun said, yes, the normal (or gaussian) distribution is considered to be a common naturally occuring phenomena. There are many ML techniques that assumes gauss distribution and applies really well to the real world problems. For example, Gaussian processes assumes Gaussian noise in the dataset. And also, I don't quite fathom the reason to use a categorical threshold. Why do we need to use a threshold? If the user specifies afield as categorical, shouldn't we convert it to a categorical variable without leveraging a threshold? On Aug 14, 2015 2:09 AM, "Seshika Fernando" wrote: > In addition, there are lots of datasets in economics, stocks, physics that > are normally or approximate normally distributed, which will be used for > predictive modelling > On 13 Aug 2015 20:46, "Supun Sethunga" wrote: > >> When a dataset is large, in general its said to be approximates to a >> Normal Distribution. :) True it Hypothetical, but the point they make is, >> when the datasets are large, then properties of a distribution like >> skewness, variance and etc. become closer to the properties Normal >> Distribution in most cases.. >> >> On Thu, Aug 13, 2015 at 11:07 AM, Nirmal Fernando >> wrote: >> >>> Hi Supun, >>> >>> Thanks for the reply. >>> >>> On Thu, Aug 13, 2015 at 8:09 PM, Supun Sethunga wrote: >>> Hi Nirmal, IMO don't think we would be able to use skewness in this case. Skewness says how symmetric the distribution is. For example, if we consider a numerical/continuous feature (not categorical) which is Normally Distributed, then the skewness would be 0. Also for a categorical (encoded) feature having a systematic distribution, then again the skewness would be 0. >>> >>> What's the probability of you see a normal distribution of a real >>> dataset? IMO it's very less and also since what we're doing here is a >>> suggestion, do you see it as an issue? >>> >>> We did have this concern at the beginning as well, regarding how we could determine whether a feature is categorical or Continuous. Usually this is strictly dependent on the domain of the dataset (i.e. user have to decide this with the knowledge about the data). That was the idea behind letting user change the data type.. But since we needed a default option, we had to go for the threshold thing, which was the olny option we could come-up with. I did a bit of research on this too, but only to find no other solution :( Thanks, Supun On Thu, Aug 13, 2015 at 1:49 AM, Nirmal Fernando wrote: > Hi All, > > We have a feature in ML where we suggest a given data column of a > dataset is categorical or numerical. Currently, how we determine this is > by > using a threshold value (The maximum number of categories that can > have in a non-string categorical feature. If exceeds, the feature > will be treated as a numerical feature.). But this is not a > successful measurement for most of the datasets. > > Can we use 'skewness' of a distribution as a measurement to determine > this? Can we say, a column is numerical, if the modulus of the skewness of > the distribution is less than a certain threshold (say 0.01) ? > > *References*: > > http://www.itrcweb.org/gsmc-1/Content/GW%20Stats/5%20Methods%20in%20indiv%20Topics/5%206%20Distributional%20Tests.htm > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Supun Sethunga* Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Team Lead - WSO2 Machine Learner >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> *Supun Sethunga* >> Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > ___ > Dev mailing list > Dev@wso2.org > http://wso2.org/cgi-bin/mailman/listinfo/dev > > ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Issue while loading the leaf dataset (misreading classes)
This issue occurs, if I turn the response variable to a categorical variable. If I get the variable as a numerical variable, the values are read correctly. So I presume there is a fault in categorical conversion of the variable. On Tue, Aug 11, 2015 at 7:11 PM, Thushan Ganegedara wrote: > I still get the same result > > 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 > 1.0 1.0 1.0 12.012.012.012.012.012.0 > 12.012.012.012.013.013.013.013.013.013.0 > 13.013.013.013.014.014.014.014.014.0 > 14.014.014.015.015.015.015.015.015.0 > 15.015.015.015.015.015.016.016.016.016.0 > 16.016.016.016.017.017.017.017.017.0 > 17.017.017.017.017.018.018.018.018.0 > 18.018.018.018.018.018.018.019.019.019.0 > 19.019.019.019.019.019.019.019.019.0 > 19.019.02.0 2.0 2.0 2.0 2.0 2.0 2.0 > 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 4.0 > 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 5.0 5.0 > 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 > 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 > 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 > 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 > 3.0 3.0 3.0 3.0 > > On Tue, Aug 11, 2015 at 7:05 PM, Nirmal Fernando wrote: > >> Can you use following code and try; >> >> List points = labeledPoints.collect(); >> for(int i=0;i> System.out.print(points.get(i).label() + "\t"); >> } >> >> On Tue, Aug 11, 2015 at 2:30 PM, Thushan Ganegedara >> wrote: >> >>> I used the following snippet >>> >>> for(int i=0;i>> System.out.print(labeledPoints.collect().get(i).label() + >>> "\t"); >>> } >>> >>> in the public MLModel build() throws MLModelBuilderException in >>> DeeplearningModelBuilder.java >>> >>> >>> On Tue, Aug 11, 2015 at 6:17 PM, Nirmal Fernando >>> wrote: >>> >>>> Hi thushan, >>>> >>>> We need more info. What did you exactly print and where? >>>> >>>> On Tue, Aug 11, 2015 at 12:47 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I found the potential cause of the poor accuracy for the leaf dataset. >>>>> It seems the data read into ML is wrong. >>>>> >>>>> I have attached the data file as a CSV (classes are in the last column) >>>>> >>>>> However, when I print out the labels of the read data (classes), it >>>>> looks something like below. Clearly there aren't this many "3.0" classes >>>>> and there should be classes up to 36.0. >>>>> >>>>> Is this caused by a bug? >>>>> >>>>> 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >>>>> 1.0
Re: [Dev] [ML] Issue while loading the leaf dataset (misreading classes)
I still get the same result 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 12.012.012.012.012.012.0 12.012.012.012.013.013.013.013.013.013.0 13.013.013.013.014.014.014.014.014.0 14.014.014.015.015.015.015.015.015.0 15.015.015.015.015.015.016.016.016.016.0 16.016.016.016.017.017.017.017.017.0 17.017.017.017.017.018.018.018.018.0 18.018.018.018.018.018.018.019.019.019.0 19.019.019.019.019.019.019.019.019.0 19.019.02.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 On Tue, Aug 11, 2015 at 7:05 PM, Nirmal Fernando wrote: > Can you use following code and try; > > List points = labeledPoints.collect(); > for(int i=0;i System.out.print(points.get(i).label() + "\t"); > } > > On Tue, Aug 11, 2015 at 2:30 PM, Thushan Ganegedara > wrote: > >> I used the following snippet >> >> for(int i=0;i> System.out.print(labeledPoints.collect().get(i).label() + >> "\t"); >> } >> >> in the public MLModel build() throws MLModelBuilderException in >> DeeplearningModelBuilder.java >> >> >> On Tue, Aug 11, 2015 at 6:17 PM, Nirmal Fernando wrote: >> >>> Hi thushan, >>> >>> We need more info. What did you exactly print and where? >>> >>> On Tue, Aug 11, 2015 at 12:47 PM, Thushan Ganegedara >>> wrote: >>> >>>> Hi, >>>> >>>> I found the potential cause of the poor accuracy for the leaf dataset. >>>> It seems the data read into ML is wrong. >>>> >>>> I have attached the data file as a CSV (classes are in the last column) >>>> >>>> However, when I print out the labels of the read data (classes), it >>>> looks something like below. Clearly there aren't this many "3.0" classes >>>> and there should be classes up to 36.0. >>>> >>>> Is this caused by a bug? >>>> >>>> 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >>>> 1.0 1.0 1.0 12.012.012.012.012.012.0 >>>> 12.012.012.012.013.013.013.013.013.0 >>>> 13.0 >>>> 13.013.013.013.014.014.014.014.014.0 >>>> 14.014.014.015.015.015.015.015.015.0 >>>> 15.015.015.015.015.015.016.016.016.0 >>>> 16.0 >>>> 16.016.016.016.017.017.017.017.017.0 >>>> 17.017.017.017.017.018.018.018.018.0 >>>> 18.018.018.018.01
Re: [Dev] [ML] Issue while loading the leaf dataset (misreading classes)
I used the following snippet for(int i=0;i wrote: > Hi thushan, > > We need more info. What did you exactly print and where? > > On Tue, Aug 11, 2015 at 12:47 PM, Thushan Ganegedara > wrote: > >> Hi, >> >> I found the potential cause of the poor accuracy for the leaf dataset. It >> seems the data read into ML is wrong. >> >> I have attached the data file as a CSV (classes are in the last column) >> >> However, when I print out the labels of the read data (classes), it looks >> something like below. Clearly there aren't this many "3.0" classes and >> there should be classes up to 36.0. >> >> Is this caused by a bug? >> >> 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 >> 1.0 1.0 1.0 12.012.012.012.012.012.0 >> 12.012.012.012.013.013.013.013.013.013.0 >> 13.013.013.013.014.014.014.014.014.0 >> 14.014.014.015.015.015.015.015.015.0 >> 15.015.015.015.015.015.016.016.016.016.0 >> 16.016.016.016.017.017.017.017.017.0 >> 17.017.017.017.017.018.018.018.018.0 >> 18.018.018.018.018.018.018.019.019.019.0 >> 19.019.019.019.019.019.019.019.019.0 >> 19.019.02.0 2.0 2.0 2.0 2.0 2.0 2.0 >> 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 4.0 >> 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 5.0 5.0 >> 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 >> 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 >> 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 >> 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 >> 3.0 3.0 3.0 3.0 >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Issue while loading the leaf dataset (misreading classes)
Hi, I found the potential cause of the poor accuracy for the leaf dataset. It seems the data read into ML is wrong. I have attached the data file as a CSV (classes are in the last column) However, when I print out the labels of the read data (classes), it looks something like below. Clearly there aren't this many "3.0" classes and there should be classes up to 36.0. Is this caused by a bug? 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 1.0 12.012.012.012.012.012.0 12.012.012.012.013.013.013.013.013.013.0 13.013.013.013.014.014.014.014.014.0 14.014.014.015.015.015.015.015.015.0 15.015.015.015.015.015.016.016.016.016.0 16.016.016.016.017.017.017.017.017.0 17.017.017.017.017.018.018.018.018.0 18.018.018.018.018.018.018.019.019.019.0 19.019.019.019.019.019.019.019.019.0 19.019.02.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 2.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 4.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 5.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 6.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 7.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 3.0 -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia 0.72694,1.4742,0.32396,0.98535,1,0.83592,0.0046566,0.0039465,0.04779,0.12795,0.016108,0.0052323,0.00027477,1.1756,1 0.74173,1.5257,0.36116,0.98152,0.99825,0.79867,0.0052423,0.0050016,0.02416,0.090476,0.0081195,0.002708,7.48E-05,0.69659,1 0.76722,1.5725,0.38998,0.97755,1,0.80812,0.0074573,0.010121,0.011897,0.057445,0.0032891,0.00092068,3.79E-05,0.44348,1 0.73797,1.4597,0.35376,0.97566,1,0.81697,0.0068768,0.0086068,0.01595,0.065491,0.0042707,0.0011544,6.63E-05,0.58785,1 0.82301,1.7707,0.44462,0.97698,1,0.75493,0.007428,0.010042,0.0079379,0.045339,0.0020514,0.00055986,2.35E-05,0.34214,1 0.72997,1.4892,0.34284,0.98755,1,0.84482,0.0049451,0.0044506,0.010487,0.058528,0.0034138,0.0011248,2.48E-05,0.34068,1 0.82063,1.7529,0.44458,0.97964,0.99649,0.7677,0.0059279,0.0063954,0.018375,0.080587,0.0064523,0.0022713,4.15E-05,0.53904,1 0.77982,1.6215,0.39222,0.98512,0.99825,0.80816,0.0050987,0.0047314,0.024875,0.089686,0.0079794,0.0024664,0.00014676,0.66975,1 0.83089,1.8199,0.45693,0.9824,1,0.77106,0.0060055,0.006564,0.0072447,0.040616,0.0016469,0.00038812,3.29E-05,0.33696,1 0.90631,2.3906,0.58336,0.97683,0.99825,0.66419,0.0084019,0.012848,0.0070096,0.042347,0.0017901,0.00045889,2.83E-05,0.28082,1 0.7459,1.4927,0.34116,0.98296,1,0.83088,0.0055665,0.0056395,0.0057679,0.036511,0.0013313,0.00030872,3.18E-05,0.25026,1 0.79606,1.6934,0.43387,0.98181,1,0.76985,0.0077992,0.011071,0.013677,0.057832,0.004,0.00081648,0.00013855,0.49751,1 0.93361,2.7582,0.64257,0.98346,1,0.59851,0.0055336,0.0055731,0.029712,0.089889,0.0080153,0.0020648,0.00023883,0.91499,2 0.91186,2.4994,0.60323,0.983,1,0.64916,0.0061494,0.0068823,0.018887,0.072486,0.0052267,0.0014887,8.33E-05,0.67811,2 0.89063,2.2927,0.56667,0.98732,1,0.66427,0.0028365,0.0014643,0.029272,0.091328,0.0082717,0.0022383,0.00020166,0.87177,2 0.86755,2.009,0.51464,0.98691,1,0.70277,0.0054439,0.0053937,0.030348,0.092063,0.0084044,0.0022541,0.000
Re: [Dev] [ML] Progress with Deeplearning Component
Hi CD, No worries. On Tue, Aug 11, 2015 at 5:11 PM, CD Athuraliya wrote: > Hi Nirmal, > > We will be able to fix this issue. > > Thanks Thushan for pointing this out! :) > > > > On Tue, Aug 11, 2015 at 12:32 PM, Nirmal Fernando wrote: > >> @CD, is this something we could fix? can we list features in the order of >> the indices? >> >> On Tue, Aug 11, 2015 at 12:25 PM, Thushan Ganegedara >> wrote: >> >>> Hi, >>> >>> I noticed that, in certain cases, the features don't follow the correct >>> ordering. Any idea why this is happening? >>> >>> For example in this image, V10 appears after V1 >>> >>> On Tue, Aug 11, 2015 at 12:10 PM, Thushan Ganegedara >>> wrote: >>> >>>> Hi all, >>>> >>>> After a daunting struggle, I was able to corner the issue with the poor >>>> accuracy for the specific leaf dataset. The dataset has classes from 1 to >>>> 36. However, there are no classes from 16th - 22nd. i.e. Classes go as >>>> 1,2,..,14,15,23,24,...,35,36 >>>> >>>> Then, while converting these class labels to enums in H-2-O (combined >>>> with the fact that there's very little data for each class) confuses H-2-O >>>> and causes it to *assign different enum values for the same classes in >>>> different datasets*. Which manifest itself as a poor accuracy. >>>> >>>> I suspect that there's a mismatch between the labels provided by >>>> JavaRDD and enums produced by H-2-O as well. I'm looking into this issue >>>> right now. >>>> >>>> Thank you >>>> >>>> On Mon, Aug 10, 2015 at 11:16 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I've been testing the new Deeplearning component with few different >>>>> datasets (mainly leaf dataset) and the leaf dataset seems to be not >>>>> working >>>>> as expected for an unknown reason. >>>>> >>>>> However, I tested the Deeplearning component extensively with the leaf >>>>> dataset and identified several potential problems that might be causing >>>>> the >>>>> poor accuracy. >>>>> >>>>> 1. Need to have higher number of epochs (compared to other datasets) >>>>> to produce a reasonable accuracy. >>>>> >>>>> 2. Too many neurons causing overfitting thereby causing poor accuracy. >>>>> >>>>> 3. Some classes have quite closely related features (Especially the >>>>> latter classes are misclassified often) >>>>> >>>>> I was able to get an accuracy of 86% with Logistic Regression L-BFGS. >>>>> Which is quite reasonable. But I'm having trouble reaching that accuracy >>>>> with Deeplearning (which should be quite easy). Highest accuracy I reached >>>>> so far is 71.xx% >>>>> >>>>> So I'm still looking for any definite issues causing the poor accuracy. >>>>> >>>>> Thank you. >>>>> >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Thushan Ganegedara >>>>> School of IT >>>>> University of Sydney, Australia >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <http://cdathuraliya.tumblr.com/> > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Progress with Deeplearning Component
Hi, I noticed that, in certain cases, the features don't follow the correct ordering. Any idea why this is happening? For example in this image, V10 appears after V1 On Tue, Aug 11, 2015 at 12:10 PM, Thushan Ganegedara wrote: > Hi all, > > After a daunting struggle, I was able to corner the issue with the poor > accuracy for the specific leaf dataset. The dataset has classes from 1 to > 36. However, there are no classes from 16th - 22nd. i.e. Classes go as > 1,2,..,14,15,23,24,...,35,36 > > Then, while converting these class labels to enums in H-2-O (combined with > the fact that there's very little data for each class) confuses H-2-O and > causes it to *assign different enum values for the same classes in > different datasets*. Which manifest itself as a poor accuracy. > > I suspect that there's a mismatch between the labels provided by JavaRDD > and enums produced by H-2-O as well. I'm looking into this issue right now. > > Thank you > > On Mon, Aug 10, 2015 at 11:16 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I've been testing the new Deeplearning component with few different >> datasets (mainly leaf dataset) and the leaf dataset seems to be not working >> as expected for an unknown reason. >> >> However, I tested the Deeplearning component extensively with the leaf >> dataset and identified several potential problems that might be causing the >> poor accuracy. >> >> 1. Need to have higher number of epochs (compared to other datasets) to >> produce a reasonable accuracy. >> >> 2. Too many neurons causing overfitting thereby causing poor accuracy. >> >> 3. Some classes have quite closely related features (Especially the >> latter classes are misclassified often) >> >> I was able to get an accuracy of 86% with Logistic Regression L-BFGS. >> Which is quite reasonable. But I'm having trouble reaching that accuracy >> with Deeplearning (which should be quite easy). Highest accuracy I reached >> so far is 71.xx% >> >> So I'm still looking for any definite issues causing the poor accuracy. >> >> Thank you. >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Introducing new Activation hyper parameter for Deeplearning
Hi, Thank you very much. Yes, I'll do so. On Tue, Aug 11, 2015 at 1:54 PM, Nirmal Fernando wrote: > Hi Thushan, > > This is a current limitation of us, ideally we should be able to > accommodate options of a parameter in machine-learner.xml. For now, please > add all the viable options in a tool-tip against the text field. > > On Thu, Aug 6, 2015 at 7:27 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I've changed the machine-learner.xml to accommodate the new "Activation >> Type" parameter. >> >> However, I feel it's unwise to let the user type the activation type as a >> string. A combo-box would be more appropriate. There are about six >> different activation types (max out, max out w dropout, tanh, ...) >> >> But how should I pass these values to parameters.jag to create them on >> the go. One option is to embed them in parameter.jag (buildInput method) >> >> I would like to know your opinion on this. >> >> Thank you >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Progress with Deeplearning Component
Hi all, After a daunting struggle, I was able to corner the issue with the poor accuracy for the specific leaf dataset. The dataset has classes from 1 to 36. However, there are no classes from 16th - 22nd. i.e. Classes go as 1,2,..,14,15,23,24,...,35,36 Then, while converting these class labels to enums in H-2-O (combined with the fact that there's very little data for each class) confuses H-2-O and causes it to *assign different enum values for the same classes in different datasets*. Which manifest itself as a poor accuracy. I suspect that there's a mismatch between the labels provided by JavaRDD and enums produced by H-2-O as well. I'm looking into this issue right now. Thank you On Mon, Aug 10, 2015 at 11:16 AM, Thushan Ganegedara wrote: > Hi all, > > I've been testing the new Deeplearning component with few different > datasets (mainly leaf dataset) and the leaf dataset seems to be not working > as expected for an unknown reason. > > However, I tested the Deeplearning component extensively with the leaf > dataset and identified several potential problems that might be causing the > poor accuracy. > > 1. Need to have higher number of epochs (compared to other datasets) to > produce a reasonable accuracy. > > 2. Too many neurons causing overfitting thereby causing poor accuracy. > > 3. Some classes have quite closely related features (Especially the latter > classes are misclassified often) > > I was able to get an accuracy of 86% with Logistic Regression L-BFGS. > Which is quite reasonable. But I'm having trouble reaching that accuracy > with Deeplearning (which should be quite easy). Highest accuracy I reached > so far is 71.xx% > > So I'm still looking for any definite issues causing the poor accuracy. > > Thank you. > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Progress with Deeplearning Component
Hi all, I've been testing the new Deeplearning component with few different datasets (mainly leaf dataset) and the leaf dataset seems to be not working as expected for an unknown reason. However, I tested the Deeplearning component extensively with the leaf dataset and identified several potential problems that might be causing the poor accuracy. 1. Need to have higher number of epochs (compared to other datasets) to produce a reasonable accuracy. 2. Too many neurons causing overfitting thereby causing poor accuracy. 3. Some classes have quite closely related features (Especially the latter classes are misclassified often) I was able to get an accuracy of 86% with Logistic Regression L-BFGS. Which is quite reasonable. But I'm having trouble reaching that accuracy with Deeplearning (which should be quite easy). Highest accuracy I reached so far is 71.xx% So I'm still looking for any definite issues causing the poor accuracy. Thank you. -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Introducing new Activation hyper parameter for Deeplearning
Hi all, I've changed the machine-learner.xml to accommodate the new "Activation Type" parameter. However, I feel it's unwise to let the user type the activation type as a string. A combo-box would be more appropriate. There are about six different activation types (max out, max out w dropout, tanh, ...) But how should I pass these values to parameters.jag to create them on the go. One option is to embed them in parameter.jag (buildInput method) I would like to know your opinion on this. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hi, Yes, I'm alright with the idea of having the review this week. On Mon, Jul 27, 2015 at 11:27 AM, Nirmal Fernando wrote: > Thanks Thushan for the update. > > On Mon, Jul 27, 2015 at 6:01 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I'm doing some tests with several datasets and most of them seemed to be >> working fine. Somehow, I stumbled upon the leaf dataset ( >> https://archive.ics.uci.edu/ml/datasets/Leaf), which does not seem to be >> working well for. However, the dataset works fine with other algorithms >> (e.g. Logistic Regression L-BFGS) Therefore, I suspect this is due to some >> sort of malformed data format. I'm right now looking into that. >> >> Furthermore, I am thinking of starting with the D3 visualization on the >> parameter setting stage. Should we be moving forward with that idea? >> > +1 > >> >> Finally, I would like to remind that, we haven't decided a date for code >> review. Should we do that? >> > > Yes, let's have it this week, if you are ok. > >> >> Thank you >> >> On Wed, Jul 22, 2015 at 11:13 PM, Thushan Ganegedara >> wrote: >> >>> Hi, >>> >>> Apologies about the late reply. >>> >>> Notes of the Demonstration >>> >>> Time duration: approximately 30 mins >>> >>> The demonstration was to demonstrate the implemented deeplearning >>> feature of WSO2-ML. The demo started first explaining the dataset used >>> (i.e. MNIST). The dataset is a CSV file with approximately 3 rows and >>> 784 features. >>> >>> Next the dataset was loaded to WSO2-ml. Here a concern was raised >>> regarding selecting the type of data in the Preprocessing Phase (i.e. >>> Categorical vs Numerical) The suggestion was that there should be a UI >>> feature to change the data type for all the variables at once (very useful >>> for large amounts of features). >>> >>> Next the deeplearning algorithm for MNIST dataset was demonstrated and >>> was able to achieve an appx 95% accuracy. Regarding the deeplearning >>> algorithms, H-2-O doesn't seem to have different deeplearning algorithms at >>> the moment, but a general deep network + classifier (probably autoencoder). >>> So the idea was to ask H-2-O team whether they are planning to implement >>> different networks in the future. >>> >>> Also, it was suggested to add a visualization feature in parameter >>> setting stage to provide a summarized visualization of the network to the >>> user. >>> >>> Furthermore, another suggestion was to test the deep network on real >>> world datasets and see how it performs. For this datasets from Kaggle will >>> be used. >>> >>> >>> About progress. >>> >>> I'm currently testing the algorithm against different datasets. and I'll >>> provide a detailed report on that in the recent future. >>> >>> Thank you >>> >>> >>> >>> On Wed, Jul 22, 2015 at 2:00 PM, Nirmal Fernando >>> wrote: >>> >>>> @Thushan how are you progressing? Could you please send the notes of >>>> our last review? >>>> >>>> On Thu, Jul 16, 2015 at 10:43 AM, CD Athuraliya >>>> wrote: >>>> >>>>> >>>>> >>>>> On Mon, Jul 13, 2015 at 11:12 AM, Thushan Ganegedara >>>> > wrote: >>>>> >>>>>> Hello CD, >>>>>> >>>>>> Yes, it seems to be working fine now. But why does it show the axes >>>>>> in meters? Is this a d3 specific thing? >>>>>> >>>>> >>>>> I think *m* stands for *Milli* here. >>>>> >>>>>> >>>>>> On Mon, Jul 13, 2015 at 3:17 PM, Thushan Ganegedara >>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> Thank you very much for pointing out. I'll get the latest update and >>>>>>> see. >>>>>>> >>>>>>> On Mon, Jul 13, 2015 at 3:03 PM, CD Athuraliya >>>>>>> wrote: >>>>>>> >>>>>>>> Hi Thushan, >>>>>>>> >>>>>>>> That method has been updated. Please get the latest. You might have >>>>>>>> to define your own case depend
Re: [Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hi all, I'm doing some tests with several datasets and most of them seemed to be working fine. Somehow, I stumbled upon the leaf dataset ( https://archive.ics.uci.edu/ml/datasets/Leaf), which does not seem to be working well for. However, the dataset works fine with other algorithms (e.g. Logistic Regression L-BFGS) Therefore, I suspect this is due to some sort of malformed data format. I'm right now looking into that. Furthermore, I am thinking of starting with the D3 visualization on the parameter setting stage. Should we be moving forward with that idea? Finally, I would like to remind that, we haven't decided a date for code review. Should we do that? Thank you On Wed, Jul 22, 2015 at 11:13 PM, Thushan Ganegedara wrote: > Hi, > > Apologies about the late reply. > > Notes of the Demonstration > > Time duration: approximately 30 mins > > The demonstration was to demonstrate the implemented deeplearning feature > of WSO2-ML. The demo started first explaining the dataset used (i.e. > MNIST). The dataset is a CSV file with approximately 3 rows and 784 > features. > > Next the dataset was loaded to WSO2-ml. Here a concern was raised > regarding selecting the type of data in the Preprocessing Phase (i.e. > Categorical vs Numerical) The suggestion was that there should be a UI > feature to change the data type for all the variables at once (very useful > for large amounts of features). > > Next the deeplearning algorithm for MNIST dataset was demonstrated and was > able to achieve an appx 95% accuracy. Regarding the deeplearning > algorithms, H-2-O doesn't seem to have different deeplearning algorithms at > the moment, but a general deep network + classifier (probably autoencoder). > So the idea was to ask H-2-O team whether they are planning to implement > different networks in the future. > > Also, it was suggested to add a visualization feature in parameter setting > stage to provide a summarized visualization of the network to the user. > > Furthermore, another suggestion was to test the deep network on real world > datasets and see how it performs. For this datasets from Kaggle will be > used. > > > About progress. > > I'm currently testing the algorithm against different datasets. and I'll > provide a detailed report on that in the recent future. > > Thank you > > > > On Wed, Jul 22, 2015 at 2:00 PM, Nirmal Fernando wrote: > >> @Thushan how are you progressing? Could you please send the notes of our >> last review? >> >> On Thu, Jul 16, 2015 at 10:43 AM, CD Athuraliya >> wrote: >> >>> >>> >>> On Mon, Jul 13, 2015 at 11:12 AM, Thushan Ganegedara >>> wrote: >>> >>>> Hello CD, >>>> >>>> Yes, it seems to be working fine now. But why does it show the axes in >>>> meters? Is this a d3 specific thing? >>>> >>> >>> I think *m* stands for *Milli* here. >>> >>>> >>>> On Mon, Jul 13, 2015 at 3:17 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> Thank you very much for pointing out. I'll get the latest update and >>>>> see. >>>>> >>>>> On Mon, Jul 13, 2015 at 3:03 PM, CD Athuraliya >>>>> wrote: >>>>> >>>>>> Hi Thushan, >>>>>> >>>>>> That method has been updated. Please get the latest. You might have >>>>>> to define your own case depending on predicted values. >>>>>> >>>>>> CD Athuraliya >>>>>> Sent from my mobile device >>>>>> On Jul 13, 2015 10:24 AM, "Nirmal Fernando" wrote: >>>>>> >>>>>>> Great work Thushan! On the UI issues, @CD could help you. AFAIK >>>>>>> actual keeps the pointer to the actual label and predicted is the >>>>>>> probability and predictedLabel is after rounding it using a threshold. >>>>>>> >>>>>>> On Mon, Jul 13, 2015 at 7:14 AM, Thushan Ganegedara < >>>>>>> thu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I have integrated H-2-O deeplearning to WSO2-ml successfully. >>>>>>>> Following are the stats on 2 tests conducted (screenshots attached). >>>>>>>> >>>>>>>> Iris dataset - 93.62% Accuracy >>>>>>>> MNIST (Small) dataset - 94.94% Accuracy >>>>>
Re: [Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hi, Apologies about the late reply. Notes of the Demonstration Time duration: approximately 30 mins The demonstration was to demonstrate the implemented deeplearning feature of WSO2-ML. The demo started first explaining the dataset used (i.e. MNIST). The dataset is a CSV file with approximately 3 rows and 784 features. Next the dataset was loaded to WSO2-ml. Here a concern was raised regarding selecting the type of data in the Preprocessing Phase (i.e. Categorical vs Numerical) The suggestion was that there should be a UI feature to change the data type for all the variables at once (very useful for large amounts of features). Next the deeplearning algorithm for MNIST dataset was demonstrated and was able to achieve an appx 95% accuracy. Regarding the deeplearning algorithms, H-2-O doesn't seem to have different deeplearning algorithms at the moment, but a general deep network + classifier (probably autoencoder). So the idea was to ask H-2-O team whether they are planning to implement different networks in the future. Also, it was suggested to add a visualization feature in parameter setting stage to provide a summarized visualization of the network to the user. Furthermore, another suggestion was to test the deep network on real world datasets and see how it performs. For this datasets from Kaggle will be used. About progress. I'm currently testing the algorithm against different datasets. and I'll provide a detailed report on that in the recent future. Thank you On Wed, Jul 22, 2015 at 2:00 PM, Nirmal Fernando wrote: > @Thushan how are you progressing? Could you please send the notes of our > last review? > > On Thu, Jul 16, 2015 at 10:43 AM, CD Athuraliya > wrote: > >> >> >> On Mon, Jul 13, 2015 at 11:12 AM, Thushan Ganegedara >> wrote: >> >>> Hello CD, >>> >>> Yes, it seems to be working fine now. But why does it show the axes in >>> meters? Is this a d3 specific thing? >>> >> >> I think *m* stands for *Milli* here. >> >>> >>> On Mon, Jul 13, 2015 at 3:17 PM, Thushan Ganegedara >>> wrote: >>> >>>> Hi all, >>>> >>>> Thank you very much for pointing out. I'll get the latest update and >>>> see. >>>> >>>> On Mon, Jul 13, 2015 at 3:03 PM, CD Athuraliya >>>> wrote: >>>> >>>>> Hi Thushan, >>>>> >>>>> That method has been updated. Please get the latest. You might have to >>>>> define your own case depending on predicted values. >>>>> >>>>> CD Athuraliya >>>>> Sent from my mobile device >>>>> On Jul 13, 2015 10:24 AM, "Nirmal Fernando" wrote: >>>>> >>>>>> Great work Thushan! On the UI issues, @CD could help you. AFAIK >>>>>> actual keeps the pointer to the actual label and predicted is the >>>>>> probability and predictedLabel is after rounding it using a threshold. >>>>>> >>>>>> On Mon, Jul 13, 2015 at 7:14 AM, Thushan Ganegedara >>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I have integrated H-2-O deeplearning to WSO2-ml successfully. >>>>>>> Following are the stats on 2 tests conducted (screenshots attached). >>>>>>> >>>>>>> Iris dataset - 93.62% Accuracy >>>>>>> MNIST (Small) dataset - 94.94% Accuracy >>>>>>> >>>>>>> However, there were few unusual issues that I had to spend lot of >>>>>>> time to identify. >>>>>>> >>>>>>> *FrameSplitter does not work for any value other than 0.5. Any value >>>>>>> other than 0.5, the following error is returned* >>>>>>> (Frame splitter is used to split trainingData to train and valid >>>>>>> sets) >>>>>>> barrier onExCompletion for >>>>>>> hex.deeplearning.DeepLearning$DeepLearningDriver@25e994ae >>>>>>> java.lang.RuntimeException: java.lang.RuntimeException: >>>>>>> java.lang.NullPointerException >>>>>>> at >>>>>>> hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:382) >>>>>>> >>>>>>> *DeepLearningModel.score(double[] vec) method doesn't work. * >>>>>>> The predictions obtained with score(Frame f) and score(double[] v) >>>>>>> is shown below. >>>>>
Re: [Dev] [GSoC-2015] Proposal 11: Recommendation Solution for WSO2 Machine Learner
;>>>>>>>>>>>>> implementing as soon as >>>>>>>>>>>>>>>>>>>>>>>>>>>>> possible. It would be great if you can review the >>>>>>>>>>>>>>>>>>>>>>>>>>>>> proposal and provide >>>>>>>>>>>>>>>>>>>>>>>>>>>>> feedback. Previous mail thread attached at [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <http://wso2.markmail.org/search/%5BDev%5D+%5BGSoC-2015%5D+Proposal+11:+Recommendation+Solution+for+WSO2+Machine+Learner+from:%22Madawa+Soysa%22> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> . >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> [1] >>>>>>>>>>>>>>>>>>>>>>>>>>>>> - >>>>>>>>>>>>>>>>>>>>>>>>>>>>> http://wso2.markmail.org/search/%5BDev%5D+%5BGSoC-2015%5D+Proposal+11:+Recommendation+Solution+for+WSO2+Machine+Learner+from:%22Madawa+Soysa%22 >>>>>>>>>>>>>>>>>>>>>>>>>>>>> <http://wso2.markmail.org/search/%5BDev%5D+%5BGSoC-2015%5D+Proposal+11:+Recommendation+Solution+for+WSO2+Machine+Learner+from:%22Madawa+Soysa%22> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Thanks, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Madawa >>>>>>>>>>>>>>>>>>>>>>>>>>>>> -- >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> *_**Madawa >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Soysa* >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Undergraduate, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Department of Computer Science and Engineering, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> Chairman, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> IEEE Computer Society Student Chapter, >>>>>>>>>>>>>>>>>>>>>>>>>>>>> University of Moratuwa. >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>>>>>> >>>>>>>>>>>>>>>>>>>>>>>>&
Re: [Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hello Mahesan,
Thank you for pointing that out. This was actually before the latest built,
This was happening because of following code snippent in view-model.jag
var actual = testResultDataPointsSample[i].predictedVsActual.actual;
var predicted =
testResultDataPointsSample[i].predictedVsActual.predicted;
var labeledPredicted = labelPredicted(predicted, 0.5);
if(actual == labeledPredicted) {
predictedVsActualPoint[2] = 'Correct';
}
else {
predictedVsActualPoint[2] = 'Incorrect';
}
where it should compare *actual == predicted* for deeplearning. But it is
fixed in the latest commit as CD mentioned.
So it is working properly at the moment.
I've attached a screenshot of new version
On Wed, Jul 15, 2015 at 7:54 PM, Sinnathamby Mahesan
wrote:
> Hi Thushan
> thank you for sending the attachments.
> I am just wondering why I see many red-dots in the graphs:
> For example, for iris data set, oaccroding to the table only 3 were found
> incorrectly predicted
> whereas the scatter diagram shows many reds as well as greens.
> Enlighten me if the way I see is wrong.
> :-)
> Regards
> Mahesan
>
> On 13 July 2015 at 07:14, Thushan Ganegedara wrote:
>
>> Hi all,
>>
>> I have integrated H-2-O deeplearning to WSO2-ml successfully. Following
>> are the stats on 2 tests conducted (screenshots attached).
>>
>> Iris dataset - 93.62% Accuracy
>> MNIST (Small) dataset - 94.94% Accuracy
>>
>> However, there were few unusual issues that I had to spend lot of time to
>> identify.
>>
>> *FrameSplitter does not work for any value other than 0.5. Any value
>> other than 0.5, the following error is returned*
>> (Frame splitter is used to split trainingData to train and valid sets)
>> barrier onExCompletion for
>> hex.deeplearning.DeepLearning$DeepLearningDriver@25e994ae
>> java.lang.RuntimeException: java.lang.RuntimeException:
>> java.lang.NullPointerException
>> at
>> hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:382)
>>
>> *DeepLearningModel.score(double[] vec) method doesn't work. *
>> The predictions obtained with score(Frame f) and score(double[] v) is
>> shown below.
>>
>> *Actual, score(Frame f), score(double[] v)*
>> 0.0, 0.0, 1.0
>> 1.0, 1.0, 2.0
>> 2.0, 2.0, 2.0
>> 2.0, 1.0, 2.0
>> 1.0, 1.0, 2.0
>>
>> As you can see, score(double[] v) is quite poor.
>>
>> After fixing above issues, everything seems to be working fine at the
>> moment.
>>
>> However, the I've a concern regarding the following method in
>> view-model.jag -> function
>> drawPredictedVsActualChart(testResultDataPointsSample)
>>
>> var actual = testResultDataPointsSample[i].predictedVsActual.actual;
>> var predicted =
>> testResultDataPointsSample[i].predictedVsActual.predicted;
>> var labeledPredicted = labelPredicted(predicted, 0.5);
>>
>> if(actual == labeledPredicted) {
>> predictedVsActualPoint[2] = 'Correct';
>> }
>> else {
>> predictedVsActualPoint[2] = 'Incorrect';
>> }
>>
>> why does it compare the *actual and labeledPredicted* where it should be
>> comparing *actual and predicted*?
>>
>> Also, the *Actual vs Predicted graph for MNIST show the axis in "Meters"
>> *(mnist.png) which doesn't make sense. I'm still looking into this.
>>
>> Thank you
>>
>>
>>
>> --
>> Regards,
>>
>> Thushan Ganegedara
>> School of IT
>> University of Sydney, Australia
>>
>> ___
>> Dev mailing list
>> Dev@wso2.org
>> http://wso2.org/cgi-bin/mailman/listinfo/dev
>>
>>
>
>
> --
> ~~
> Sinnathamby Mahesan
>
>
>
> ~~
> ~~
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hello CD,
Yes, it seems to be working fine now. But why does it show the axes in
meters? Is this a d3 specific thing?
On Mon, Jul 13, 2015 at 3:17 PM, Thushan Ganegedara
wrote:
> Hi all,
>
> Thank you very much for pointing out. I'll get the latest update and see.
>
> On Mon, Jul 13, 2015 at 3:03 PM, CD Athuraliya
> wrote:
>
>> Hi Thushan,
>>
>> That method has been updated. Please get the latest. You might have to
>> define your own case depending on predicted values.
>>
>> CD Athuraliya
>> Sent from my mobile device
>> On Jul 13, 2015 10:24 AM, "Nirmal Fernando" wrote:
>>
>>> Great work Thushan! On the UI issues, @CD could help you. AFAIK actual
>>> keeps the pointer to the actual label and predicted is the probability and
>>> predictedLabel is after rounding it using a threshold.
>>>
>>> On Mon, Jul 13, 2015 at 7:14 AM, Thushan Ganegedara
>>> wrote:
>>>
>>>> Hi all,
>>>>
>>>> I have integrated H-2-O deeplearning to WSO2-ml successfully. Following
>>>> are the stats on 2 tests conducted (screenshots attached).
>>>>
>>>> Iris dataset - 93.62% Accuracy
>>>> MNIST (Small) dataset - 94.94% Accuracy
>>>>
>>>> However, there were few unusual issues that I had to spend lot of time
>>>> to identify.
>>>>
>>>> *FrameSplitter does not work for any value other than 0.5. Any value
>>>> other than 0.5, the following error is returned*
>>>> (Frame splitter is used to split trainingData to train and valid sets)
>>>> barrier onExCompletion for
>>>> hex.deeplearning.DeepLearning$DeepLearningDriver@25e994ae
>>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>>> java.lang.NullPointerException
>>>> at
>>>> hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:382)
>>>>
>>>> *DeepLearningModel.score(double[] vec) method doesn't work. *
>>>> The predictions obtained with score(Frame f) and score(double[] v) is
>>>> shown below.
>>>>
>>>> *Actual, score(Frame f), score(double[] v)*
>>>> 0.0, 0.0, 1.0
>>>> 1.0, 1.0, 2.0
>>>> 2.0, 2.0, 2.0
>>>> 2.0, 1.0, 2.0
>>>> 1.0, 1.0, 2.0
>>>>
>>>> As you can see, score(double[] v) is quite poor.
>>>>
>>>> After fixing above issues, everything seems to be working fine at the
>>>> moment.
>>>>
>>>> However, the I've a concern regarding the following method in
>>>> view-model.jag -> function
>>>> drawPredictedVsActualChart(testResultDataPointsSample)
>>>>
>>>> var actual = testResultDataPointsSample[i].predictedVsActual.actual;
>>>> var predicted =
>>>> testResultDataPointsSample[i].predictedVsActual.predicted;
>>>> var labeledPredicted = labelPredicted(predicted, 0.5);
>>>>
>>>> if(actual == labeledPredicted) {
>>>> predictedVsActualPoint[2] = 'Correct';
>>>> }
>>>> else {
>>>> predictedVsActualPoint[2] = 'Incorrect';
>>>> }
>>>>
>>>> why does it compare the *actual and labeledPredicted* where it should
>>>> be comparing *actual and predicted*?
>>>>
>>>> Also, the *Actual vs Predicted graph for MNIST show the axis in
>>>> "Meters" *(mnist.png) which doesn't make sense. I'm still looking into
>>>> this.
>>>>
>>>> Thank you
>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>>
>>>> Thushan Ganegedara
>>>> School of IT
>>>> University of Sydney, Australia
>>>>
>>>
>>>
>>>
>>> --
>>>
>>> Thanks & regards,
>>> Nirmal
>>>
>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>>> Mobile: +94715779733
>>> Blog: http://nirmalfdo.blogspot.com/
>>>
>>>
>>>
>
>
> --
> Regards,
>
> Thushan Ganegedara
> School of IT
> University of Sydney, Australia
>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hi all,
Thank you very much for pointing out. I'll get the latest update and see.
On Mon, Jul 13, 2015 at 3:03 PM, CD Athuraliya wrote:
> Hi Thushan,
>
> That method has been updated. Please get the latest. You might have to
> define your own case depending on predicted values.
>
> CD Athuraliya
> Sent from my mobile device
> On Jul 13, 2015 10:24 AM, "Nirmal Fernando" wrote:
>
>> Great work Thushan! On the UI issues, @CD could help you. AFAIK actual
>> keeps the pointer to the actual label and predicted is the probability and
>> predictedLabel is after rounding it using a threshold.
>>
>> On Mon, Jul 13, 2015 at 7:14 AM, Thushan Ganegedara
>> wrote:
>>
>>> Hi all,
>>>
>>> I have integrated H-2-O deeplearning to WSO2-ml successfully. Following
>>> are the stats on 2 tests conducted (screenshots attached).
>>>
>>> Iris dataset - 93.62% Accuracy
>>> MNIST (Small) dataset - 94.94% Accuracy
>>>
>>> However, there were few unusual issues that I had to spend lot of time
>>> to identify.
>>>
>>> *FrameSplitter does not work for any value other than 0.5. Any value
>>> other than 0.5, the following error is returned*
>>> (Frame splitter is used to split trainingData to train and valid sets)
>>> barrier onExCompletion for
>>> hex.deeplearning.DeepLearning$DeepLearningDriver@25e994ae
>>> java.lang.RuntimeException: java.lang.RuntimeException:
>>> java.lang.NullPointerException
>>> at
>>> hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:382)
>>>
>>> *DeepLearningModel.score(double[] vec) method doesn't work. *
>>> The predictions obtained with score(Frame f) and score(double[] v) is
>>> shown below.
>>>
>>> *Actual, score(Frame f), score(double[] v)*
>>> 0.0, 0.0, 1.0
>>> 1.0, 1.0, 2.0
>>> 2.0, 2.0, 2.0
>>> 2.0, 1.0, 2.0
>>> 1.0, 1.0, 2.0
>>>
>>> As you can see, score(double[] v) is quite poor.
>>>
>>> After fixing above issues, everything seems to be working fine at the
>>> moment.
>>>
>>> However, the I've a concern regarding the following method in
>>> view-model.jag -> function
>>> drawPredictedVsActualChart(testResultDataPointsSample)
>>>
>>> var actual = testResultDataPointsSample[i].predictedVsActual.actual;
>>> var predicted =
>>> testResultDataPointsSample[i].predictedVsActual.predicted;
>>> var labeledPredicted = labelPredicted(predicted, 0.5);
>>>
>>> if(actual == labeledPredicted) {
>>> predictedVsActualPoint[2] = 'Correct';
>>> }
>>> else {
>>> predictedVsActualPoint[2] = 'Incorrect';
>>> }
>>>
>>> why does it compare the *actual and labeledPredicted* where it should
>>> be comparing *actual and predicted*?
>>>
>>> Also, the *Actual vs Predicted graph for MNIST show the axis in
>>> "Meters" *(mnist.png) which doesn't make sense. I'm still looking into
>>> this.
>>>
>>> Thank you
>>>
>>>
>>>
>>> --
>>> Regards,
>>>
>>> Thushan Ganegedara
>>> School of IT
>>> University of Sydney, Australia
>>>
>>
>>
>>
>> --
>>
>> Thanks & regards,
>> Nirmal
>>
>> Associate Technical Lead - Data Technologies Team, WSO2 Inc.
>> Mobile: +94715779733
>> Blog: http://nirmalfdo.blogspot.com/
>>
>>
>>
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Update - Deeplearning Integration to WSO2-ml
Hi all,
I have integrated H-2-O deeplearning to WSO2-ml successfully. Following are
the stats on 2 tests conducted (screenshots attached).
Iris dataset - 93.62% Accuracy
MNIST (Small) dataset - 94.94% Accuracy
However, there were few unusual issues that I had to spend lot of time to
identify.
*FrameSplitter does not work for any value other than 0.5. Any value other
than 0.5, the following error is returned*
(Frame splitter is used to split trainingData to train and valid sets)
barrier onExCompletion for
hex.deeplearning.DeepLearning$DeepLearningDriver@25e994ae
java.lang.RuntimeException: java.lang.RuntimeException:
java.lang.NullPointerException
at
hex.deeplearning.DeepLearning$DeepLearningDriver.trainModel(DeepLearning.java:382)
*DeepLearningModel.score(double[] vec) method doesn't work. *
The predictions obtained with score(Frame f) and score(double[] v) is
shown below.
*Actual, score(Frame f), score(double[] v)*
0.0, 0.0, 1.0
1.0, 1.0, 2.0
2.0, 2.0, 2.0
2.0, 1.0, 2.0
1.0, 1.0, 2.0
As you can see, score(double[] v) is quite poor.
After fixing above issues, everything seems to be working fine at the
moment.
However, the I've a concern regarding the following method in
view-model.jag -> function
drawPredictedVsActualChart(testResultDataPointsSample)
var actual = testResultDataPointsSample[i].predictedVsActual.actual;
var predicted =
testResultDataPointsSample[i].predictedVsActual.predicted;
var labeledPredicted = labelPredicted(predicted, 0.5);
if(actual == labeledPredicted) {
predictedVsActualPoint[2] = 'Correct';
}
else {
predictedVsActualPoint[2] = 'Incorrect';
}
why does it compare the *actual and labeledPredicted* where it should be
comparing *actual and predicted*?
Also, the *Actual vs Predicted graph for MNIST show the axis in
"Meters" *(mnist.png)
which doesn't make sense. I'm still looking into this.
Thank you
--
Regards,
Thushan Ganegedara
School of IT
University of Sydney, Australia
___
Dev mailing list
Dev@wso2.org
http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Error while trying to create a Dataset
Hi,
Never mind!. A clean install did the trick.
On Wed, Jul 8, 2015 at 4:12 PM, Thushan Ganegedara wrote:
> Hi,
>
> I'll got an update and built it. But now getting a new error during the
> startup of WSO2Server
>
> [2015-07-08 16:07:46,617] ERROR
> {org.wso2.carbon.ndatasource.core.internal.DataS
> ourceServiceComponent} - Error in intializing system data sources: Error
> in ini
> tializing system data sources: Error in initializing system data sources
> at 'C:\
>
> Users\Thush\Documents\NetBeansProjects\WSO2-ML\wso2ml-1.0.0-SNAPSHOT\repository\
> conf\datasources\analytics-datasources.xml' - Error in loading data source
> provi
> der: org.wso2.carbon.datasource.reader.hadoop.HDFSDataSourceReader cannot
> be fou
> nd by org.wso2.carbon.ndatasource.core_4.4.0
> org.wso2.carbon.ndatasource.common.DataSourceException: Error in
> initializing system data sources: Error in initializing system data
> sources at 'C:\...\WSO2-ML\wso2ml-1.0.0-SNAPSHOT\repository\conf\datasou
> rces\analytics-datasources.xml' - Error in loading data source provider:
> org.wso
> 2.carbon.datasource.reader.hadoop.HDFSDataSourceReader cannot be found by
> org.ws
> o2.carbon.ndatasource.core_4.4.0
> at
> org.wso2.carbon.ndatasource.core.DataSourceManager.initSystemDataSour
> ces(DataSourceManager.java:168)
> at
> org.wso2.carbon.ndatasource.core.internal.DataSourceServiceComponent.
> initSystemDataSources(DataSourceServiceComponent.java:192)
> at
> org.wso2.carbon.ndatasource.core.internal.DataSourceServiceComponent.
> setSecretCallbackHandlerService(DataSourceServiceComponent.java:178)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.eclipse.equinox.internal.ds.model.ComponentReference.bind(Compone
> ntReference.java:376)
> at
> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.bindRefere
> nce(ServiceComponentProp.java:430)
> at
> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.bind(Servi
> ceComponentProp.java:218)
> at
> org.eclipse.equinox.internal.ds.model.ServiceComponentProp.build(Serv
> iceComponentProp.java:343)
> at
> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponent(Instan
> ceProcess.java:620)
> at
> org.eclipse.equinox.internal.ds.InstanceProcess.buildComponents(Insta
> nceProcess.java:197)
> at
> org.eclipse.equinox.internal.ds.Resolver.getEligible(Resolver.java:34
> 3)
> at
> org.eclipse.equinox.internal.ds.SCRManager.serviceChanged(SCRManager.
> java:222)
> at
> org.eclipse.osgi.internal.serviceregistry.FilteredServiceListener.ser
> viceChanged(FilteredServiceListener.java:107)
> at
> org.eclipse.osgi.framework.internal.core.BundleContextImpl.dispatchEv
> ent(BundleContextImpl.java:861)
> at
> org.eclipse.osgi.framework.eventmgr.EventManager.dispatchEvent(EventM
> anager.java:230)
> at
> org.eclipse.osgi.framework.eventmgr.ListenerQueue.dispatchEventSynchr
> onous(ListenerQueue.java:148)
> at
> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServ
> iceEventPrivileged(ServiceRegistry.java:819)
> at
> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.publishServ
> iceEvent(ServiceRegistry.java:771)
> at
> org.eclipse.osgi.internal.serviceregistry.ServiceRegistrationImpl.reg
> ister(ServiceRegistrationImpl.java:130)
> at
> org.eclipse.osgi.internal.serviceregistry.ServiceRegistry.registerSer
> vice(ServiceRegistry.java:214)
> at
> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerSe
> rvice(BundleContextImpl.java:433)
> at
> org.eclipse.osgi.framework.internal.core.BundleContextImpl.registerSe
> rvice(BundleContextImpl.java:451)
> at
> org.wso2.carbon.securevault.internal.SecretManagerInitializerComponen
> t.activate(SecretManagerInitializerComponent.java:48)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
> java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
> sorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.eclipse.equinox.internal.ds.model.ServiceComponent.activate(Servi
> ceComponent.ja
Re: [Dev] [ML] Error while trying to create a Dataset
ger.jav
a:816)
at
org.eclipse.equinox.internal.ds.SCRManager$QueuedJob.dispatch(SCRMana
ger.java:783)
at
org.eclipse.equinox.internal.ds.WorkThread.run(WorkThread.java:89)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.wso2.carbon.ndatasource.common.DataSourceException: Error in
init
ializing system data sources at
'C:\Users\Thush\Documents\NetBeansProjects\WSO2-
ML\wso2ml-1.0.0-SNAPSHOT\repository\conf\datasources\analytics-datasources.xml'
- Error in loading data source provider:
org.wso2.carbon.datasource.reader.hadoo
p.HDFSDataSourceReader cannot be found by
org.wso2.carbon.ndatasource.core_4.4.0
at
org.wso2.carbon.ndatasource.core.DataSourceManager.initSystemDataSour
ce(DataSourceManager.java:188)
at
org.wso2.carbon.ndatasource.core.DataSourceManager.initSystemDataSour
ces(DataSourceManager.java:164)
... 40 more
Caused by: org.wso2.carbon.ndatasource.common.DataSourceException: Error in
load
ing data source provider:
org.wso2.carbon.datasource.reader.hadoop.HDFSDataSourc
eReader cannot be found by org.wso2.carbon.ndatasource.core_4.4.0
at
org.wso2.carbon.ndatasource.core.DataSourceManager.addDataSourceProvi
ders(DataSourceManager.java:139)
at
org.wso2.carbon.ndatasource.core.DataSourceManager.initSystemDataSour
ce(DataSourceManager.java:180)
... 41 more
Caused by: java.lang.ClassNotFoundException:
org.wso2.carbon.datasource.reader.h
adoop.HDFSDataSourceReader cannot be found by
org.wso2.carbon.ndatasource.core_4
.4.0
at
org.eclipse.osgi.internal.loader.BundleLoader.findClassInternal(Bundl
eLoader.java:501)
at
org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.
java:421)
at
org.eclipse.osgi.internal.loader.BundleLoader.findClass(BundleLoader.
java:412)
at
org.eclipse.osgi.internal.baseadaptor.DefaultClassLoader.loadClass(De
faultClassLoader.java:107)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at
org.wso2.carbon.ndatasource.core.DataSourceManager.addDataSourceProvi
ders(DataSourceManager.java:136)
... 42 more
Thank you
On Wed, Jul 8, 2015 at 1:54 PM, Nirmal Fernando wrote:
> Seems Jenkins is yet to build the latest code. You have to update
> carbon-ml and build it too.
>
> On Wed, Jul 8, 2015 at 9:23 AM, Nirmal Fernando wrote:
>
>> Hi Thushan,
>>
>> Yep, we had this issue yesterday and it is resolved now. Please do an
>> online build of product-ml.
>>
>> On Wed, Jul 8, 2015 at 8:56 AM, Thushan Ganegedara
>> wrote:
>>
>>> Hi all,
>>>
>>> I'm getting the following error while trying to create a dataset in the
>>> WSO2-ml
>>>
>>> I've got the latest carbon pull and product-ml pull. And I've done a
>>> maven clean install
>>>
>>> [2015-07-08 09:59:56,011] ERROR
>>> {org.wso2.carbon.ml.rest.api.DatasetApiV10} - Error occurred while
>>> uploading a [dataset] MLDataset [id=0, name=Iris, tenantId=-1234,
>>> userName=admin, dataSourceType=file, dataTargetType=file, sourcePath=null,
>>> dataType=csv, comments=, version=1.0.2, containsHeader=true] of tenant [id]
>>> -12
>>> 34 and [user] admin . Cause:
>>> org.wso2.carbon.ml.core.exceptions.MLDataProcessing
>>> Exception - Failed to extract the sample points from path:
>>> C:\Users\Thush\Documents\NetBeansProjects\WSO2-ML\wso2ml-1.0.0-SNAPSHOT\tmp\Iris.-1234.1436313595997.
>>> Cause: com.typesafe.config.ConfigException$Missing: No configuration
>>> setting found for key 'akka.version'
>>> org.wso2.carbon.ml.core.exceptions.MLDataProcessingException: Failed to
>>> extract
>>> the sample points from path: C:\Users\...\WSO2-ML\w
>>> so2ml-1.0.0-SNAPSHOT\tmp\Iris.-1234.1436313595997. Cause:
>>> com.typesafe.config.Co
>>> nfigException$Missing: No configuration setting found for key
>>> 'akka.version'
>>> at
>>> org.wso2.carbon.ml.core.impl.FileDatasetProcessor.process(FileDataset
>>> Processor.java:73)
>>> at
>>> org.wso2.carbon.ml.core.impl.MLDatasetProcessor.process(MLDatasetProc
>>> essor.java:165)
>>> at
>>> org.wso2.carbon.ml.rest.api.DatasetApiV10.uploadDataset(DatasetApiV10
>>> .java:100)
>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> at
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.
>>> java:62)
>>> at
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAcces
>>> sorImpl.java:43)
>>>
[Dev] [ML] Error while trying to create a Dataset
tainer.java:47) at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValv e.java:62) at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.inv oke(CarbonStuckThreadDetectionValve.java:159) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java: 950) at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(Ca rbonContextCreatorValve.java:57) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineVal ve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.jav a:421) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp 11Processor.java:1074) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process( AbstractProtocol.java:611) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpo int.java:1739) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoin t.java:1698) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor. java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor .java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskTh read.java:61) at java.lang.Thread.run(Thread.java:745) -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Serialization issue with H-2-O
Hi, Sounds good. I'll do so. On Tue, Jul 7, 2015 at 9:49 PM, Nirmal Fernando wrote: > Yes, let's make a note of it and proceed till we get some help from H2O. > > On Tue, Jul 7, 2015 at 5:13 PM, Thushan Ganegedara > wrote: > >> Hello, >> >> It seems it's quite hard to debug the issue with custom serialization of >> DLModel and there's no reply from developers either. >> >> One possibility is to handle the persistence and retrieval for the >> DLModel in the MLModelHandler itself, until this gets sorted out. >> >> Could you share your opinion on this? >> >> On Tue, Jul 7, 2015 at 12:01 PM, Thushan Ganegedara >> wrote: >> >>> Hi all, >>> >>> I've introduced a MLModel for Deeplearning and currently looking into >>> serialization of that model. It seems serialization of H-2-O >>> DeeplearningModel (with custom serializers) is quite tricky. >>> >>> I have been able to successfully serialize the model by using an >>> AutoBuffer (H-2-O) and ByteStream >>> >>> However in the deserialization process, it gives an error (described in >>> https://groups.google.com/forum/#!topic/h2ostream/JRjdj2f7CXw) >>> >>> Therefore it is still impossible to deserialize the model. I've done >>> extensive debuggin on H-2-O tracking all the steps during the >>> serialization. But it seems to occur from a class created during start up >>> time by something called a "Weaver". I still trying to figure out how to >>> debug them. >>> >>> Thank you >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Serialization issue with H-2-O
Hello, It seems it's quite hard to debug the issue with custom serialization of DLModel and there's no reply from developers either. One possibility is to handle the persistence and retrieval for the DLModel in the MLModelHandler itself, until this gets sorted out. Could you share your opinion on this? On Tue, Jul 7, 2015 at 12:01 PM, Thushan Ganegedara wrote: > Hi all, > > I've introduced a MLModel for Deeplearning and currently looking into > serialization of that model. It seems serialization of H-2-O > DeeplearningModel (with custom serializers) is quite tricky. > > I have been able to successfully serialize the model by using an > AutoBuffer (H-2-O) and ByteStream > > However in the deserialization process, it gives an error (described in > https://groups.google.com/forum/#!topic/h2ostream/JRjdj2f7CXw) > > Therefore it is still impossible to deserialize the model. I've done > extensive debuggin on H-2-O tracking all the steps during the > serialization. But it seems to occur from a class created during start up > time by something called a "Weaver". I still trying to figure out how to > debug them. > > Thank you > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Serialization issue with H-2-O
Hi all, I've introduced a MLModel for Deeplearning and currently looking into serialization of that model. It seems serialization of H-2-O DeeplearningModel (with custom serializers) is quite tricky. I have been able to successfully serialize the model by using an AutoBuffer (H-2-O) and ByteStream However in the deserialization process, it gives an error (described in https://groups.google.com/forum/#!topic/h2ostream/JRjdj2f7CXw) Therefore it is still impossible to deserialize the model. I've done extensive debuggin on H-2-O tracking all the steps during the serialization. But it seems to occur from a class created during start up time by something called a "Weaver". I still trying to figure out how to debug them. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Passing data to H-2-O
Hello,
I'm afraid that won't work. Because we will have to go deep into H-2-O and
change their classes to use standard serialization. As I mentioned earlier,
they are using a custom serializer for all serialization.
So even if we implement a standard H-2-O serializer, it won't work for the
basic classes in H-2-O
On Wed, Jul 1, 2015 at 2:31 PM, Nirmal Fernando wrote:
> Hi Thushan,
>
> Instead, can we introduce a H2O Model object with standard serialization ?
>
> On Wed, Jul 1, 2015 at 9:57 AM, Thushan Ganegedara
> wrote:
>
>> Hi,
>>
>> I got the response from the H-2-O team, it seems that they are not
>> following the standard serialization procedure. (They are using an
>> internally generated serialization code).
>>
>> Specifically, I've to save all the keys generated by the model (not the
>> model itself)
>>
>> Therefore, I was thinking of including the keys in the MLModel rather
>> than than the model itself. However, I'm not sure whether that would work
>>
>> And then save the keys seperately using a specific H-2-O serializer in
>> the persist and retrieve methods in the MLModelHandler.
>>
>>
>>
>>
>> On Wed, Jul 1, 2015 at 10:35 AM, Thushan Ganegedara
>> wrote:
>>
>>> Hi all,
>>>
>>> I was able to get H-2-O running from WSO2-ML UI. It can take a given
>>> file and train a deeplearning algorithm and produce the model.
>>>
>>> However, I get the following error after the training process. The error
>>> is due to the inability to deserialize the DeepLearningModel (H-2-O class)
>>> Though the serialization happens, deserialization produce the following
>>> error.
>>>
>>> I have already posted this on their google group and awaiting a response.
>>>
>>> [2015-07-01 09:48:14,490] ERROR
>>> {org.wso2.carbon.ml.core.impl.MLModelHandler} -
>>> Failed to build the model [id] 52
>>> org.wso2.carbon.ml.core.exceptions.MLModelBuilderException: An error
>>> occurred wh
>>> ile building supervised machine learning model: An error occurred while
>>> building
>>> stacked autoencoders model: Job aborted due to stage failure: Task 0 in
>>> stage 1
>>> .0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID
>>> 1, local
>>> host): java.io.InvalidClassException: hex.deeplearning.DeepLearning; no
>>> valid co
>>> nstructor
>>> at
>>> java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(Obje
>>> ctStreamClass.java:150)
>>> at
>>> java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:768
>>> )
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
>>> 775)
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
>>> 93)
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
>>> 801)
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
>>> 93)
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
>>> 801)
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
>>> 93)
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
>>> 801)
>>> at
>>> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
>>> at
>>> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
>>> 93)
>>> at
>>> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>>>
>>> at
>>> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
>>> 801)
>>> at
>>&
Re: [Dev] [ML] Passing data to H-2-O
Hi,
I got the response from the H-2-O team, it seems that they are not
following the standard serialization procedure. (They are using an
internally generated serialization code).
Specifically, I've to save all the keys generated by the model (not the
model itself)
Therefore, I was thinking of including the keys in the MLModel rather than
than the model itself. However, I'm not sure whether that would work
And then save the keys seperately using a specific H-2-O serializer in the
persist and retrieve methods in the MLModelHandler.
On Wed, Jul 1, 2015 at 10:35 AM, Thushan Ganegedara
wrote:
> Hi all,
>
> I was able to get H-2-O running from WSO2-ML UI. It can take a given file
> and train a deeplearning algorithm and produce the model.
>
> However, I get the following error after the training process. The error
> is due to the inability to deserialize the DeepLearningModel (H-2-O class)
> Though the serialization happens, deserialization produce the following
> error.
>
> I have already posted this on their google group and awaiting a response.
>
> [2015-07-01 09:48:14,490] ERROR
> {org.wso2.carbon.ml.core.impl.MLModelHandler} -
> Failed to build the model [id] 52
> org.wso2.carbon.ml.core.exceptions.MLModelBuilderException: An error
> occurred wh
> ile building supervised machine learning model: An error occurred while
> building
> stacked autoencoders model: Job aborted due to stage failure: Task 0 in
> stage 1
> .0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1,
> local
> host): java.io.InvalidClassException: hex.deeplearning.DeepLearning; no
> valid co
> nstructor
> at
> java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(Obje
> ctStreamClass.java:150)
> at
> java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:768
> )
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 775)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
> 93)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
> 93)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
> 93)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
> 93)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
> 93)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at
> java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
> 93)
> at
> java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
>
> at
> java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
> 801)
> at
> java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
> at
> org.apache.spark.serializer.JavaDeserializationStream.readObject(Java
> Serializer.scala:68)
> at
> org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSe
> rializer.scala:94)
> at
> org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
> at org.apache.spark.scheduler.Task.run(Task.scala:64)
> at
> org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
&
Re: [Dev] [ML] Passing data to H-2-O
Hi all,
I was able to get H-2-O running from WSO2-ML UI. It can take a given file
and train a deeplearning algorithm and produce the model.
However, I get the following error after the training process. The error is
due to the inability to deserialize the DeepLearningModel (H-2-O class)
Though the serialization happens, deserialization produce the following
error.
I have already posted this on their google group and awaiting a response.
[2015-07-01 09:48:14,490] ERROR
{org.wso2.carbon.ml.core.impl.MLModelHandler} -
Failed to build the model [id] 52
org.wso2.carbon.ml.core.exceptions.MLModelBuilderException: An error
occurred wh
ile building supervised machine learning model: An error occurred while
building
stacked autoencoders model: Job aborted due to stage failure: Task 0 in
stage 1
.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1,
local
host): java.io.InvalidClassException: hex.deeplearning.DeepLearning; no
valid co
nstructor
at
java.io.ObjectStreamClass$ExceptionInfo.newInvalidClassException(Obje
ctStreamClass.java:150)
at
java.io.ObjectStreamClass.checkDeserialize(ObjectStreamClass.java:768
)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
775)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
93)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
93)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
93)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
93)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
93)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at
java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:19
93)
at
java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1918)
at
java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1
801)
at
java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351)
at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371)
at
org.apache.spark.serializer.JavaDeserializationStream.readObject(Java
Serializer.scala:68)
at
org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSe
rializer.scala:94)
at
org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)
at org.apache.spark.scheduler.Task.run(Task.scala:64)
at
org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:203)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
at java.lang.Thread.run(Thread.java:745)
Driver stacktrace:
at
org.wso2.carbon.ml.core.spark.algorithms.DeeplearningModel.buildModel
(DeeplearningModel.java:94)
at
org.wso2.carbon.ml.core.impl.MLModelHandler$ModelBuilder.run(MLModelH
andler.java:588)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:617)
at java.lang.Thread.run(Thread.java:745)
On Tue, Jun 30, 2015 at 2:34 PM, Thushan Ganegedara
wrote:
> Thank you very much
>
> It works now.
>
> On Tue, Jun 30, 2015 at 11:35 AM, Nirmal Fernando wrote:
>
>> Hi Thushan,
>>
>> Here you go;
>> https://github.com/wso2/carbon-ml/commit/959c89e2fbfe04ec4c49290fd68ac08ae8718e6a
>>
>> context.getFacts().getDatasetURL()
>>
>> On Tue, Jun 30, 2015 at 5:12 AM, Thushan Ganegedara
>> wrote:
>>
>>> H
Re: [Dev] [ML] Passing data to H-2-O
Thank you very much It works now. On Tue, Jun 30, 2015 at 11:35 AM, Nirmal Fernando wrote: > Hi Thushan, > > Here you go; > https://github.com/wso2/carbon-ml/commit/959c89e2fbfe04ec4c49290fd68ac08ae8718e6a > > context.getFacts().getDatasetURL() > > On Tue, Jun 30, 2015 at 5:12 AM, Thushan Ganegedara > wrote: > >> Hello all, >> >> It seems H-2-O takes data as a file (.csv format) But in wso2-ml data is >> passed as JavaRDD to train method. >> >> So clearly there's going to be an inconsistency between deeplearning and >> other algorithms. >> >> I already posted the question in the h-2-o google group. They have said >> it's theoritically possible to pass data to h-2-o in the code itself. >> However, they haven't come out with a specific solution there. So I'm >> waiting on that. >> >> Is there a way to directly access the file within the classifier (or >> maybe pass the file to the classifier), if that option becomes the last >> resort for this task? >> >> Thank you >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Passing data to H-2-O
Hello all, It seems H-2-O takes data as a file (.csv format) But in wso2-ml data is passed as JavaRDD to train method. So clearly there's going to be an inconsistency between deeplearning and other algorithms. I already posted the question in the h-2-o google group. They have said it's theoritically possible to pass data to h-2-o in the code itself. However, they haven't come out with a specific solution there. So I'm waiting on that. Is there a way to directly access the file within the classifier (or maybe pass the file to the classifier), if that option becomes the last resort for this task? Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi, Yes, sure we can have a call. What time do you want to do it? On Thu, Jun 25, 2015 at 11:16 PM, Nirmal Fernando wrote: > Hi Thushan, > > Let's have a call to discuss this, I think you can implement the deep > learning support without much (if not any) change. > > On Thu, Jun 25, 2015 at 11:54 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I'm looking into integrating deep networks algorithm (Java). From what I >> understood, algorithms are called the following way. >> >> *SupervisedModel* class builds the correct model according to provided >> parameters using buildmodel(...) >> >> These buildmodel(...) methods first, a *ModelSummary *object is >> created and then a *MLModel* object >> >> However, beyond that is done using *apache spark* and spark-specific >> objects (e.g. NaiveBayesClassifier, LogisticRegression, ...) But in my >> case, I cannot rely on these objects. >> >> I would like to know what to do regarding this issue. >> >> One thing I can do is that, *produce a ModelSummaryObject *directly from >> the Deep Autoencoder Algorithm or a *custom SAEModel object*. >> >> Could you please comment on this? >> >> >> >> On Thu, Jun 25, 2015 at 3:11 PM, Thushan Ganegedara >> wrote: >> >>> Hello CD, >>> >>> Thank you very much for the explanation. It makes sense now >>> >>> >>> On Thu, Jun 25, 2015 at 3:02 PM, CD Athuraliya >>> wrote: >>> >>>> Hi Thushan, >>>> >>>> What we are getting from API has the alphabetically order (enforced by >>>> query response and by the way we persist hyper-parameters in DB). But we >>>> need to enforce some logical order for hyper-parameters in our UI. This is >>>> the reason to change the order. >>>> >>>> Regards, >>>> CD >>>> >>>> On Thu, Jun 25, 2015 at 10:25 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hello CD, >>>>> >>>>> Thank you very much for the clarification. >>>>> >>>>> On Thu, Jun 25, 2015 at 2:50 PM, CD Athuraliya >>>>> wrote: >>>>> >>>>>> Hi Thushan, >>>>>> >>>>>> For the moment we are not retrieving algorithms in algorithm >>>>>> selection step (algorithm.jag) from API. So you will have to add your new >>>>>> algorithm to relevant .jag file manually (by updating markup). But for >>>>>> hyper-parameters it should work by adding them to machine-learner.xml >>>>>> as Nirmal has mentioned. We will be fixing algorithm.jag soon to retrieve >>>>>> algorithms from API, but for now please follow these steps. >>>>>> >>>>>> Regards, >>>>>> CD >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 25, 2015 at 10:11 AM, Nirmal Fernando >>>>>> wrote: >>>>>> >>>>>>> Ok. Few pointers below; >>>>>>> >>>>>>> 1. We have all the default set of hyper parameters for each >>>>>>> algorithm in machine-learner.xml file. >>>>>>> https://github.com/wso2/carbon-ml/blob/master/features/ml/org.wso2.carbon.ml.core.feature/src/main/resources/conf/machine-learner.xml#L55 >>>>>>> >>>>>>> 2. If you like to add a new algorithm, you should first add it in >>>>>>> machine-learner.xml file. >>>>>>> >>>>>>> 3. Then, once you select your new algorithm via UI, you'll get the >>>>>>> default set of hyper parameters. >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 25, 2015 at 9:58 AM, Thushan Ganegedara < >>>>>>> thu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm trying to add the hyper-parameters related to deep-networks. >>>>>>>> (Deep Networks needs several new additional parameters) >>>>>>>> So once the ajax call made, it should return correct >>>>>>>> hyper-parameters for the deep network algorithm specified by user. >>>>>>>> >>>>>>>> On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando >>>>>>>> wrote: >>>
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi, Sounds good. I'll wait on this On Thu, Jun 25, 2015 at 4:29 PM, Nirmal Fernando wrote: > Hi Thushan, > > Give us today, we'll get back to you. Plan is to decouple Spark from model > generation implementation. > > On Thu, Jun 25, 2015 at 11:54 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I'm looking into integrating deep networks algorithm (Java). From what I >> understood, algorithms are called the following way. >> >> *SupervisedModel* class builds the correct model according to provided >> parameters using buildmodel(...) >> >> These buildmodel(...) methods first, a *ModelSummary *object is >> created and then a *MLModel* object >> >> However, beyond that is done using *apache spark* and spark-specific >> objects (e.g. NaiveBayesClassifier, LogisticRegression, ...) But in my >> case, I cannot rely on these objects. >> >> I would like to know what to do regarding this issue. >> >> One thing I can do is that, *produce a ModelSummaryObject *directly from >> the Deep Autoencoder Algorithm or a *custom SAEModel object*. >> >> Could you please comment on this? >> >> >> >> On Thu, Jun 25, 2015 at 3:11 PM, Thushan Ganegedara >> wrote: >> >>> Hello CD, >>> >>> Thank you very much for the explanation. It makes sense now >>> >>> >>> On Thu, Jun 25, 2015 at 3:02 PM, CD Athuraliya >>> wrote: >>> >>>> Hi Thushan, >>>> >>>> What we are getting from API has the alphabetically order (enforced by >>>> query response and by the way we persist hyper-parameters in DB). But we >>>> need to enforce some logical order for hyper-parameters in our UI. This is >>>> the reason to change the order. >>>> >>>> Regards, >>>> CD >>>> >>>> On Thu, Jun 25, 2015 at 10:25 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hello CD, >>>>> >>>>> Thank you very much for the clarification. >>>>> >>>>> On Thu, Jun 25, 2015 at 2:50 PM, CD Athuraliya >>>>> wrote: >>>>> >>>>>> Hi Thushan, >>>>>> >>>>>> For the moment we are not retrieving algorithms in algorithm >>>>>> selection step (algorithm.jag) from API. So you will have to add your new >>>>>> algorithm to relevant .jag file manually (by updating markup). But for >>>>>> hyper-parameters it should work by adding them to machine-learner.xml >>>>>> as Nirmal has mentioned. We will be fixing algorithm.jag soon to retrieve >>>>>> algorithms from API, but for now please follow these steps. >>>>>> >>>>>> Regards, >>>>>> CD >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jun 25, 2015 at 10:11 AM, Nirmal Fernando >>>>>> wrote: >>>>>> >>>>>>> Ok. Few pointers below; >>>>>>> >>>>>>> 1. We have all the default set of hyper parameters for each >>>>>>> algorithm in machine-learner.xml file. >>>>>>> https://github.com/wso2/carbon-ml/blob/master/features/ml/org.wso2.carbon.ml.core.feature/src/main/resources/conf/machine-learner.xml#L55 >>>>>>> >>>>>>> 2. If you like to add a new algorithm, you should first add it in >>>>>>> machine-learner.xml file. >>>>>>> >>>>>>> 3. Then, once you select your new algorithm via UI, you'll get the >>>>>>> default set of hyper parameters. >>>>>>> >>>>>>> >>>>>>> On Thu, Jun 25, 2015 at 9:58 AM, Thushan Ganegedara < >>>>>>> thu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> I'm trying to add the hyper-parameters related to deep-networks. >>>>>>>> (Deep Networks needs several new additional parameters) >>>>>>>> So once the ajax call made, it should return correct >>>>>>>> hyper-parameters for the deep network algorithm specified by user. >>>>>>>> >>>>>>>> On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando >>>>>>>> wrote: >>>>>>>> >>>>>
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi all, I'm looking into integrating deep networks algorithm (Java). From what I understood, algorithms are called the following way. *SupervisedModel* class builds the correct model according to provided parameters using buildmodel(...) These buildmodel(...) methods first, a *ModelSummary *object is created and then a *MLModel* object However, beyond that is done using *apache spark* and spark-specific objects (e.g. NaiveBayesClassifier, LogisticRegression, ...) But in my case, I cannot rely on these objects. I would like to know what to do regarding this issue. One thing I can do is that, *produce a ModelSummaryObject *directly from the Deep Autoencoder Algorithm or a *custom SAEModel object*. Could you please comment on this? On Thu, Jun 25, 2015 at 3:11 PM, Thushan Ganegedara wrote: > Hello CD, > > Thank you very much for the explanation. It makes sense now > > > On Thu, Jun 25, 2015 at 3:02 PM, CD Athuraliya > wrote: > >> Hi Thushan, >> >> What we are getting from API has the alphabetically order (enforced by >> query response and by the way we persist hyper-parameters in DB). But we >> need to enforce some logical order for hyper-parameters in our UI. This is >> the reason to change the order. >> >> Regards, >> CD >> >> On Thu, Jun 25, 2015 at 10:25 AM, Thushan Ganegedara >> wrote: >> >>> Hello CD, >>> >>> Thank you very much for the clarification. >>> >>> On Thu, Jun 25, 2015 at 2:50 PM, CD Athuraliya >>> wrote: >>> >>>> Hi Thushan, >>>> >>>> For the moment we are not retrieving algorithms in algorithm selection >>>> step (algorithm.jag) from API. So you will have to add your new algorithm >>>> to relevant .jag file manually (by updating markup). But for >>>> hyper-parameters it should work by adding them to machine-learner.xml >>>> as Nirmal has mentioned. We will be fixing algorithm.jag soon to retrieve >>>> algorithms from API, but for now please follow these steps. >>>> >>>> Regards, >>>> CD >>>> >>>> >>>> >>>> On Thu, Jun 25, 2015 at 10:11 AM, Nirmal Fernando >>>> wrote: >>>> >>>>> Ok. Few pointers below; >>>>> >>>>> 1. We have all the default set of hyper parameters for each algorithm >>>>> in machine-learner.xml file. >>>>> https://github.com/wso2/carbon-ml/blob/master/features/ml/org.wso2.carbon.ml.core.feature/src/main/resources/conf/machine-learner.xml#L55 >>>>> >>>>> 2. If you like to add a new algorithm, you should first add it in >>>>> machine-learner.xml file. >>>>> >>>>> 3. Then, once you select your new algorithm via UI, you'll get the >>>>> default set of hyper parameters. >>>>> >>>>> >>>>> On Thu, Jun 25, 2015 at 9:58 AM, Thushan Ganegedara >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> I'm trying to add the hyper-parameters related to deep-networks. >>>>>> (Deep Networks needs several new additional parameters) >>>>>> So once the ajax call made, it should return correct hyper-parameters >>>>>> for the deep network algorithm specified by user. >>>>>> >>>>>> On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando >>>>>> wrote: >>>>>> >>>>>>> Hi Thushan, >>>>>>> >>>>>>> Nope, we do not have one at hand as of now, but we will need to do >>>>>>> one when time permits. >>>>>>> >>>>>>> What are you trying to do? >>>>>>> >>>>>>> On Thu, Jun 25, 2015 at 9:45 AM, Thushan Ganegedara < >>>>>>> thu...@gmail.com> wrote: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> I started looking into Jaggery UI with the intention of changing >>>>>>>> the interface to accommodate Deep Networks. >>>>>>>> >>>>>>>> One of the initial tasks is to get the correct hyper parameters for >>>>>>>> the Deep Network user select. I found out that this is returned with, >>>>>>>> s*erverUrl + "/api/analyses/" + analysisId + >>>>>>>> "/hyperParameters?algorithmName=" + algorithmName, l
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hello CD, Thank you very much for the explanation. It makes sense now On Thu, Jun 25, 2015 at 3:02 PM, CD Athuraliya wrote: > Hi Thushan, > > What we are getting from API has the alphabetically order (enforced by > query response and by the way we persist hyper-parameters in DB). But we > need to enforce some logical order for hyper-parameters in our UI. This is > the reason to change the order. > > Regards, > CD > > On Thu, Jun 25, 2015 at 10:25 AM, Thushan Ganegedara > wrote: > >> Hello CD, >> >> Thank you very much for the clarification. >> >> On Thu, Jun 25, 2015 at 2:50 PM, CD Athuraliya >> wrote: >> >>> Hi Thushan, >>> >>> For the moment we are not retrieving algorithms in algorithm selection >>> step (algorithm.jag) from API. So you will have to add your new algorithm >>> to relevant .jag file manually (by updating markup). But for >>> hyper-parameters it should work by adding them to machine-learner.xml >>> as Nirmal has mentioned. We will be fixing algorithm.jag soon to retrieve >>> algorithms from API, but for now please follow these steps. >>> >>> Regards, >>> CD >>> >>> >>> >>> On Thu, Jun 25, 2015 at 10:11 AM, Nirmal Fernando >>> wrote: >>> >>>> Ok. Few pointers below; >>>> >>>> 1. We have all the default set of hyper parameters for each algorithm >>>> in machine-learner.xml file. >>>> https://github.com/wso2/carbon-ml/blob/master/features/ml/org.wso2.carbon.ml.core.feature/src/main/resources/conf/machine-learner.xml#L55 >>>> >>>> 2. If you like to add a new algorithm, you should first add it in >>>> machine-learner.xml file. >>>> >>>> 3. Then, once you select your new algorithm via UI, you'll get the >>>> default set of hyper parameters. >>>> >>>> >>>> On Thu, Jun 25, 2015 at 9:58 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm trying to add the hyper-parameters related to deep-networks. (Deep >>>>> Networks needs several new additional parameters) >>>>> So once the ajax call made, it should return correct hyper-parameters >>>>> for the deep network algorithm specified by user. >>>>> >>>>> On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando >>>>> wrote: >>>>> >>>>>> Hi Thushan, >>>>>> >>>>>> Nope, we do not have one at hand as of now, but we will need to do >>>>>> one when time permits. >>>>>> >>>>>> What are you trying to do? >>>>>> >>>>>> On Thu, Jun 25, 2015 at 9:45 AM, Thushan Ganegedara >>>>> > wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I started looking into Jaggery UI with the intention of changing the >>>>>>> interface to accommodate Deep Networks. >>>>>>> >>>>>>> One of the initial tasks is to get the correct hyper parameters for >>>>>>> the Deep Network user select. I found out that this is returned with, >>>>>>> s*erverUrl + "/api/analyses/" + analysisId + >>>>>>> "/hyperParameters?algorithmName=" + algorithmName, located in >>>>>>> parameters.jag* >>>>>>> >>>>>>> I started looking into the retrievel process of hyperparameters and >>>>>>> it seems this is using the api produced by *carbon-ml* to retrieve >>>>>>> hyperparameters. >>>>>>> >>>>>>> I noticed that, *AnalysisAPIv10.java* and *MLAnalysisHandler.jav*a >>>>>>> are the high-level classes which handles this request. >>>>>>> >>>>>>> However, I'm having trouble delving into detailed request handling. >>>>>>> >>>>>>> Is there any documentation which might help? I couldn't find any dev >>>>>>> level documentation in the official WSO2-ml documentation. >>>>>>> >>>>>>> Thank you >>>>>>> >>>>>>> -- >>>>>>> Regards, >>>>>>> >>>>>>> Thushan Ganegedara >>>>>>> School of IT >>>>&g
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hello CD, Thank you very much for the clarification. On Thu, Jun 25, 2015 at 2:50 PM, CD Athuraliya wrote: > Hi Thushan, > > For the moment we are not retrieving algorithms in algorithm selection > step (algorithm.jag) from API. So you will have to add your new algorithm > to relevant .jag file manually (by updating markup). But for > hyper-parameters it should work by adding them to machine-learner.xml > as Nirmal has mentioned. We will be fixing algorithm.jag soon to retrieve > algorithms from API, but for now please follow these steps. > > Regards, > CD > > > > On Thu, Jun 25, 2015 at 10:11 AM, Nirmal Fernando wrote: > >> Ok. Few pointers below; >> >> 1. We have all the default set of hyper parameters for each algorithm in >> machine-learner.xml file. >> https://github.com/wso2/carbon-ml/blob/master/features/ml/org.wso2.carbon.ml.core.feature/src/main/resources/conf/machine-learner.xml#L55 >> >> 2. If you like to add a new algorithm, you should first add it in >> machine-learner.xml file. >> >> 3. Then, once you select your new algorithm via UI, you'll get the >> default set of hyper parameters. >> >> >> On Thu, Jun 25, 2015 at 9:58 AM, Thushan Ganegedara >> wrote: >> >>> Hi, >>> >>> I'm trying to add the hyper-parameters related to deep-networks. (Deep >>> Networks needs several new additional parameters) >>> So once the ajax call made, it should return correct hyper-parameters >>> for the deep network algorithm specified by user. >>> >>> On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando >>> wrote: >>> >>>> Hi Thushan, >>>> >>>> Nope, we do not have one at hand as of now, but we will need to do one >>>> when time permits. >>>> >>>> What are you trying to do? >>>> >>>> On Thu, Jun 25, 2015 at 9:45 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Hi all, >>>>> >>>>> I started looking into Jaggery UI with the intention of changing the >>>>> interface to accommodate Deep Networks. >>>>> >>>>> One of the initial tasks is to get the correct hyper parameters for >>>>> the Deep Network user select. I found out that this is returned with, >>>>> s*erverUrl + "/api/analyses/" + analysisId + >>>>> "/hyperParameters?algorithmName=" + algorithmName, located in >>>>> parameters.jag* >>>>> >>>>> I started looking into the retrievel process of hyperparameters and it >>>>> seems this is using the api produced by *carbon-ml* to retrieve >>>>> hyperparameters. >>>>> >>>>> I noticed that, *AnalysisAPIv10.java* and *MLAnalysisHandler.jav*a >>>>> are the high-level classes which handles this request. >>>>> >>>>> However, I'm having trouble delving into detailed request handling. >>>>> >>>>> Is there any documentation which might help? I couldn't find any dev >>>>> level documentation in the official WSO2-ml documentation. >>>>> >>>>> Thank you >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Thushan Ganegedara >>>>> School of IT >>>>> University of Sydney, Australia >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <http://cdathuraliya.tumblr.com/> > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi, Thank you very much for the clarification I also noticed that, in the parameters.jag, after retrieve the parameters, they are added to the variable array while changing the order Ex: hyperParametersArray[0] = sanitize(hyperParametersResponse[2].value); hyperParametersArray[1] = sanitize(hyperParametersResponse[1].value); hyperParametersArray[2] = sanitize(hyperParametersResponse[0].value); Is there a reason for doing this? On Thu, Jun 25, 2015 at 2:41 PM, Nirmal Fernando wrote: > Ok. Few pointers below; > > 1. We have all the default set of hyper parameters for each algorithm in > machine-learner.xml file. > https://github.com/wso2/carbon-ml/blob/master/features/ml/org.wso2.carbon.ml.core.feature/src/main/resources/conf/machine-learner.xml#L55 > > 2. If you like to add a new algorithm, you should first add it in > machine-learner.xml file. > > 3. Then, once you select your new algorithm via UI, you'll get the default > set of hyper parameters. > > > On Thu, Jun 25, 2015 at 9:58 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> I'm trying to add the hyper-parameters related to deep-networks. (Deep >> Networks needs several new additional parameters) >> So once the ajax call made, it should return correct hyper-parameters for >> the deep network algorithm specified by user. >> >> On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando wrote: >> >>> Hi Thushan, >>> >>> Nope, we do not have one at hand as of now, but we will need to do one >>> when time permits. >>> >>> What are you trying to do? >>> >>> On Thu, Jun 25, 2015 at 9:45 AM, Thushan Ganegedara >>> wrote: >>> >>>> Hi all, >>>> >>>> I started looking into Jaggery UI with the intention of changing the >>>> interface to accommodate Deep Networks. >>>> >>>> One of the initial tasks is to get the correct hyper parameters for the >>>> Deep Network user select. I found out that this is returned with, >>>> s*erverUrl + "/api/analyses/" + analysisId + >>>> "/hyperParameters?algorithmName=" + algorithmName, located in >>>> parameters.jag* >>>> >>>> I started looking into the retrievel process of hyperparameters and it >>>> seems this is using the api produced by *carbon-ml* to retrieve >>>> hyperparameters. >>>> >>>> I noticed that, *AnalysisAPIv10.java* and *MLAnalysisHandler.jav*a are >>>> the high-level classes which handles this request. >>>> >>>> However, I'm having trouble delving into detailed request handling. >>>> >>>> Is there any documentation which might help? I couldn't find any dev >>>> level documentation in the official WSO2-ml documentation. >>>> >>>> Thank you >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi, I think I figured it out. It seems I should add them to machine-learner.xml. MLConfiguration.java retrieves them from there. Is that how it works? On Thu, Jun 25, 2015 at 2:28 PM, Thushan Ganegedara wrote: > Hi, > > I'm trying to add the hyper-parameters related to deep-networks. (Deep > Networks needs several new additional parameters) > So once the ajax call made, it should return correct hyper-parameters for > the deep network algorithm specified by user. > > On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando wrote: > >> Hi Thushan, >> >> Nope, we do not have one at hand as of now, but we will need to do one >> when time permits. >> >> What are you trying to do? >> >> On Thu, Jun 25, 2015 at 9:45 AM, Thushan Ganegedara >> wrote: >> >>> Hi all, >>> >>> I started looking into Jaggery UI with the intention of changing the >>> interface to accommodate Deep Networks. >>> >>> One of the initial tasks is to get the correct hyper parameters for the >>> Deep Network user select. I found out that this is returned with, >>> s*erverUrl + "/api/analyses/" + analysisId + >>> "/hyperParameters?algorithmName=" + algorithmName, located in >>> parameters.jag* >>> >>> I started looking into the retrievel process of hyperparameters and it >>> seems this is using the api produced by *carbon-ml* to retrieve >>> hyperparameters. >>> >>> I noticed that, *AnalysisAPIv10.java* and *MLAnalysisHandler.jav*a are >>> the high-level classes which handles this request. >>> >>> However, I'm having trouble delving into detailed request handling. >>> >>> Is there any documentation which might help? I couldn't find any dev >>> level documentation in the official WSO2-ml documentation. >>> >>> Thank you >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi, I'm trying to add the hyper-parameters related to deep-networks. (Deep Networks needs several new additional parameters) So once the ajax call made, it should return correct hyper-parameters for the deep network algorithm specified by user. On Thu, Jun 25, 2015 at 2:25 PM, Nirmal Fernando wrote: > Hi Thushan, > > Nope, we do not have one at hand as of now, but we will need to do one > when time permits. > > What are you trying to do? > > On Thu, Jun 25, 2015 at 9:45 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I started looking into Jaggery UI with the intention of changing the >> interface to accommodate Deep Networks. >> >> One of the initial tasks is to get the correct hyper parameters for the >> Deep Network user select. I found out that this is returned with, >> s*erverUrl + "/api/analyses/" + analysisId + >> "/hyperParameters?algorithmName=" + algorithmName, located in >> parameters.jag* >> >> I started looking into the retrievel process of hyperparameters and it >> seems this is using the api produced by *carbon-ml* to retrieve >> hyperparameters. >> >> I noticed that, *AnalysisAPIv10.java* and *MLAnalysisHandler.jav*a are >> the high-level classes which handles this request. >> >> However, I'm having trouble delving into detailed request handling. >> >> Is there any documentation which might help? I couldn't find any dev >> level documentation in the official WSO2-ml documentation. >> >> Thank you >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Changing the return JSON of WSO2-ML API calls
Hi all, I started looking into Jaggery UI with the intention of changing the interface to accommodate Deep Networks. One of the initial tasks is to get the correct hyper parameters for the Deep Network user select. I found out that this is returned with, s*erverUrl + "/api/analyses/" + analysisId + "/hyperParameters?algorithmName=" + algorithmName, located in parameters.jag* I started looking into the retrievel process of hyperparameters and it seems this is using the api produced by *carbon-ml* to retrieve hyperparameters. I noticed that, *AnalysisAPIv10.java* and *MLAnalysisHandler.jav*a are the high-level classes which handles this request. However, I'm having trouble delving into detailed request handling. Is there any documentation which might help? I couldn't find any dev level documentation in the official WSO2-ml documentation. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Error - Testing a model's prediction with a Data File
Never mind! I found where the variables are However, even after changing the variable /tmp to C:\tmp the problem still exist (C:\tmp is a valid folder) On Tue, Jun 23, 2015 at 1:37 PM, Thushan Ganegedara wrote: > Hi, > > You're on Windows right? > Yes, I am > > > > /tmp > Can you tell me where to find these variables? > > And are you using master branch? > I'm currently using a forked version of the master branch of product-ml > > Thank you > > On Tue, Jun 23, 2015 at 12:37 PM, Nirmal Fernando wrote: > >> Hi Thushan, >> >> You're on Windows right? We have a usability issue which we'll be fixing >> soon for our Windows users. For now, you have to change following; >> >> >> >> /tmp >> >> to a valid Windows file system folder. Let us know how it goes. And are >> you using master branch? If not please use master, as we have done few >> fixes for Windows usage. >> >> On Tue, Jun 23, 2015 at 5:55 AM, Thushan Ganegedara >> wrote: >> >>> Hi all, >>> >>> I trained a Logistic Regression model on IRIS dataset and I tried >>> classifying a test file (a portion of the original file in csv format) >>> >>> But when I try to do that, I get this error. >>> Error occurred while predicting from model [id] 2 of tenant [id] -1234 >>> and [user] admin. Cause: >>> org.wso2.carbon.ml.core.exceptions.MLModelHandlerException - Failed to >>> retrieve the model [id] 2 >>> >>> The error in the console >>> at java.lang.Thread.run(Thread.java:745) >>> Caused by: java.net.URISyntaxException: Illegal character in path at >>> index 4: /t >>> mp\TestAnalysis2.Model.2015-06-23_10-16-46 >>> at java.net.URI$Parser.fail(URI.java:2848) >>> at java.net.URI$Parser.checkChars(URI.java:3021) >>> at java.net.URI$Parser.parseHierarchical(URI.java:3105) >>> at java.net.URI$Parser.parse(URI.java:3063) >>> at java.net.URI.(URI.java:588) >>> at >>> org.wso2.carbon.ml.core.impl.MLModelHandler.retrieveModel(MLModelHand >>> ler.java:386) >>> >>> >>> This is the URL I have >>> >>> http://10.16.64.212:9763/ml/site/predict/predict.jag?analysisId=2&datasetId=1&modelId=2&projectName=TestProject&analysisName=TestAnalysis2&modelName=TestAnalysis2.Model.2015-06-23_10-16-46&fromCompare=false >>> >>> Any idea why this is happening? >>> >>> Thank you >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Error - Testing a model's prediction with a Data File
Hi, You're on Windows right? Yes, I am /tmp Can you tell me where to find these variables? And are you using master branch? I'm currently using a forked version of the master branch of product-ml Thank you On Tue, Jun 23, 2015 at 12:37 PM, Nirmal Fernando wrote: > Hi Thushan, > > You're on Windows right? We have a usability issue which we'll be fixing > soon for our Windows users. For now, you have to change following; > > > > /tmp > > to a valid Windows file system folder. Let us know how it goes. And are > you using master branch? If not please use master, as we have done few > fixes for Windows usage. > > On Tue, Jun 23, 2015 at 5:55 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> I trained a Logistic Regression model on IRIS dataset and I tried >> classifying a test file (a portion of the original file in csv format) >> >> But when I try to do that, I get this error. >> Error occurred while predicting from model [id] 2 of tenant [id] -1234 >> and [user] admin. Cause: >> org.wso2.carbon.ml.core.exceptions.MLModelHandlerException - Failed to >> retrieve the model [id] 2 >> >> The error in the console >> at java.lang.Thread.run(Thread.java:745) >> Caused by: java.net.URISyntaxException: Illegal character in path at >> index 4: /t >> mp\TestAnalysis2.Model.2015-06-23_10-16-46 >> at java.net.URI$Parser.fail(URI.java:2848) >> at java.net.URI$Parser.checkChars(URI.java:3021) >> at java.net.URI$Parser.parseHierarchical(URI.java:3105) >> at java.net.URI$Parser.parse(URI.java:3063) >> at java.net.URI.(URI.java:588) >> at >> org.wso2.carbon.ml.core.impl.MLModelHandler.retrieveModel(MLModelHand >> ler.java:386) >> >> >> This is the URL I have >> >> http://10.16.64.212:9763/ml/site/predict/predict.jag?analysisId=2&datasetId=1&modelId=2&projectName=TestProject&analysisName=TestAnalysis2&modelName=TestAnalysis2.Model.2015-06-23_10-16-46&fromCompare=false >> >> Any idea why this is happening? >> >> Thank you >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Error - Testing a model's prediction with a Data File
Hi all, I trained a Logistic Regression model on IRIS dataset and I tried classifying a test file (a portion of the original file in csv format) But when I try to do that, I get this error. Error occurred while predicting from model [id] 2 of tenant [id] -1234 and [user] admin. Cause: org.wso2.carbon.ml.core.exceptions.MLModelHandlerException - Failed to retrieve the model [id] 2 The error in the console at java.lang.Thread.run(Thread.java:745) Caused by: java.net.URISyntaxException: Illegal character in path at index 4: /t mp\TestAnalysis2.Model.2015-06-23_10-16-46 at java.net.URI$Parser.fail(URI.java:2848) at java.net.URI$Parser.checkChars(URI.java:3021) at java.net.URI$Parser.parseHierarchical(URI.java:3105) at java.net.URI$Parser.parse(URI.java:3063) at java.net.URI.(URI.java:588) at org.wso2.carbon.ml.core.impl.MLModelHandler.retrieveModel(MLModelHand ler.java:386) This is the URL I have http://10.16.64.212:9763/ml/site/predict/predict.jag?analysisId=2&datasetId=1&modelId=2&projectName=TestProject&analysisName=TestAnalysis2&modelName=TestAnalysis2.Model.2015-06-23_10-16-46&fromCompare=false Any idea why this is happening? Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Progress with H-2-O library
Hi all, The previous week, I've been looking into H-2-O library and how to integrate it with java. Unfortunately, H-2-O is quite well established for python and R, but not for Java. That is, there aren't resources explaining how to use H-2-O in Java other than the Javadoc itself. Their google group suggests one way to use H2O in java is to call H2O via the rest API. However, in that case again, there aren't any examples explaining how to properly use the REST API. But there are few resources such as https://github.com/h2oai/h2o-2/wiki/REST-API One options mentioned is to run an example in python and refer the commands.log to see the actual http requests made to h-2-o and figure out how REST API works. (This is what I'm doing right now) ( https://groups.google.com/forum/#!topic/h2ostream/9EYXHYXqBxo) Also I posted a question in the google group, asking whether any examples in Java is available. Thanks -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Proposed UIs for Deep Network Integration - WSO2-ML
Hi, Ok, then I will continue with the test accuracy in mind. On Fri, Jun 19, 2015 at 2:01 PM, Nirmal Fernando wrote: > Hi Thushan, > > Yes, so currently we take a training data fraction as an input from the > user (check the wizard) which splits the dataset to training and test. And > for other algorithms too, Spark generates an accuracy measurement using the > test data predictions vs actual. > > test *accuracy* sounds good as a measurement. > > On Fri, Jun 19, 2015 at 9:19 AM, Thushan Ganegedara > wrote: > >> Hi all, >> >> Thank you very much for the feedback >> >> One small thing, is there a some sort of accuracy measurement that we can >> show for deep networks? >> >> Yes, there is. Usually the accuracy of the deep network is shown with a >> validation dataset and a test set (i.e. validation error and test error). >> In other words, after training the network, we run an independent test set >> and see how accurate the algorithm is (# of correct results/ # of total >> results) >> >> It seems the test accuracy would fit for the model comparison >> >> On Fri, Jun 19, 2015 at 1:01 PM, Nirmal Fernando wrote: >> >>> Hi Thushan, >>> >>> Looks good for me too. One small thing, is there a some sort of accuracy >>> measurement that we can show for deep networks? This is required for the >>> model comparison page; >>> https://docs.wso2.com/display/ML100/Model+Comparison >>> >>> On Fri, Jun 19, 2015 at 7:27 AM, Thushan Ganegedara >>> wrote: >>> >>>> Hi, >>>> >>>> as he configure, can we visualize the deep network ( use some d3 graph >>>> library). >>>> I don't think I understood. Are you asking if we could show the user a >>>> diagram of the network he has specified (like the attached image)? >>>> >>> >>>> Also, after he has trained, can we let him visualize the output of >>>> intermediate layers ( this is a advanced feature, so optional). >>>> Intermediate levels are quite tricky to visualize. Two major issues >>>> with visualizing intermediate layers are, >>>> 1. Intermediate layers are not linear transformations of the input so >>>> the visualized filters of intermediate layers doesn't make much sense (I >>>> tried it, what I saw was completely random pixels) >>>> >>>> 2. It is possible to overcome the above issue with a technique called >>>> activation maximization (we are solving the optimization problem z = >>>> sigmoid(x.W) by keeping W constant and changing x to get the maximum z). >>>> However, this is highly non-convex. So it easily get stuck in local maxima. >>>> Also, this is to costly for large dimensional data (it will work for MNIST >>>> though). >>>> >>>> Thank you >>>> >>>> On Fri, Jun 19, 2015 at 11:50 AM, Srinath Perera >>>> wrote: >>>> >>>>> Hi Tushan, >>>>> >>>>> OK high level. However, as he configure, can we visualize the deep >>>>> network ( use some d3 graph library). >>>>> >>>>> Also, after he has trained, can we let him visualize the output of >>>>> intermediate layers ( this is a advanced feature, so optional). >>>>> >>>>> --Srinath >>>>> >>>>> On Fri, Jun 19, 2015 at 6:56 AM, Thushan Ganegedara >>>>> wrote: >>>>> >>>>>> Dear all, >>>>>> >>>>>> Please find the proposed UI changes for the Deep Network Integration >>>>>> attached herewith. >>>>>> >>>>>> Feedback would be highly appreciated. >>>>>> >>>>>> -- >>>>>> Regards, >>>>>> >>>>>> Thushan Ganegedara >>>>>> School of IT >>>>>> University of Sydney, Australia >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> >>>>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>>>> Site: http://people.apache.org/~hemapani/ >>>>> Photos: http://www.flickr.com/photos/hemapani/ >>>>> Phone: 0772360902 >>>>> >>>> >>>> >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Proposed UIs for Deep Network Integration - WSO2-ML
Hi all, Thank you very much for the feedback One small thing, is there a some sort of accuracy measurement that we can show for deep networks? Yes, there is. Usually the accuracy of the deep network is shown with a validation dataset and a test set (i.e. validation error and test error). In other words, after training the network, we run an independent test set and see how accurate the algorithm is (# of correct results/ # of total results) It seems the test accuracy would fit for the model comparison On Fri, Jun 19, 2015 at 1:01 PM, Nirmal Fernando wrote: > Hi Thushan, > > Looks good for me too. One small thing, is there a some sort of accuracy > measurement that we can show for deep networks? This is required for the > model comparison page; > https://docs.wso2.com/display/ML100/Model+Comparison > > On Fri, Jun 19, 2015 at 7:27 AM, Thushan Ganegedara > wrote: > >> Hi, >> >> as he configure, can we visualize the deep network ( use some d3 graph >> library). >> I don't think I understood. Are you asking if we could show the user a >> diagram of the network he has specified (like the attached image)? >> > >> Also, after he has trained, can we let him visualize the output of >> intermediate layers ( this is a advanced feature, so optional). >> Intermediate levels are quite tricky to visualize. Two major issues with >> visualizing intermediate layers are, >> 1. Intermediate layers are not linear transformations of the input so the >> visualized filters of intermediate layers doesn't make much sense (I tried >> it, what I saw was completely random pixels) >> >> 2. It is possible to overcome the above issue with a technique called >> activation maximization (we are solving the optimization problem z = >> sigmoid(x.W) by keeping W constant and changing x to get the maximum z). >> However, this is highly non-convex. So it easily get stuck in local maxima. >> Also, this is to costly for large dimensional data (it will work for MNIST >> though). >> >> Thank you >> >> On Fri, Jun 19, 2015 at 11:50 AM, Srinath Perera >> wrote: >> >>> Hi Tushan, >>> >>> OK high level. However, as he configure, can we visualize the deep >>> network ( use some d3 graph library). >>> >>> Also, after he has trained, can we let him visualize the output of >>> intermediate layers ( this is a advanced feature, so optional). >>> >>> --Srinath >>> >>> On Fri, Jun 19, 2015 at 6:56 AM, Thushan Ganegedara >>> wrote: >>> >>>> Dear all, >>>> >>>> Please find the proposed UI changes for the Deep Network Integration >>>> attached herewith. >>>> >>>> Feedback would be highly appreciated. >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >>> Site: http://people.apache.org/~hemapani/ >>> Photos: http://www.flickr.com/photos/hemapani/ >>> Phone: 0772360902 >>> >> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Progress so far with DL4J
Hi, Yes, that sounds good. I will start working on UI aspects while waiting for them to release fixes. If they release fixes while working on the UI it should be fine. Otherwise, I will start implementing the autoencoder. On Thu, Jun 18, 2015 at 12:04 PM, Srinath Perera wrote: > Let's work on other parts of the project, for example we can look at UI to > configure Deep learning while waiting for them to do it. However, fix takes > time, we have to write auto encoder. > > On Thu, Jun 18, 2015 at 7:26 AM, Thushan Ganegedara > wrote: > >> Dear all, >> >> I posted about poor accuracy on deeplearning4j google group and they >> replied saying they are aware of this issue ( >> https://groups.google.com/forum/#!topic/deeplearning4j/gRQH8rA9SQI). >> They are currently working on it, to get the tuned examples out soon. >> >> However, from the appearance of their response, deeplearning4j has few >> bugs/issues with gradient calculation after they moved to nd4j. >> >> I can see two options right now, >> >> 1. Wait till the bugs are fixed and tuned examples are out >> 2. Write autoencoder myself, adhering to the coding style they have >> followed. So the example can be easily integrated with deeplearning4j after >> a stable version is out. >> >> >> Would highly appreciate your feedback. >> >> On Wed, Jun 17, 2015 at 10:27 AM, Thushan Ganegedara >> wrote: >> >>> Hi, >>> >>> I couldn't find any reports on high accuracy on any set of data. The >>> popular datasets seem to be Iris dataset and MNIST dataset. However, their >>> example page on git hub shows several results for different techniques and >>> datasets (https://github.com/deeplearning4j/dl4j-0.0.3.3-examples). And >>> as it can be seen, the accuracies mentioned are quite low. (However, the >>> developers claim that this is because they have run the examples with less >>> number of nodes and less iterations) >>> >>> >>> >>> On Tue, Jun 16, 2015 at 6:03 PM, Srinath Perera >>> wrote: >>> >>>> Are there any other datasets where dl4j suppose to do well? As long as >>>> it does better with *some* datasets, we can go ahead with those? >>>> >>>> On Tue, Jun 16, 2015 at 9:13 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Yes, there are few use list (Git hub and google group). I will inquire >>>>> about this in user lists. >>>>> >>>>> Thank you >>>>> >>>>> >>>>> On Tue, Jun 16, 2015 at 12:34 PM, Nirmal Fernando >>>>> wrote: >>>>> >>>>>> Thanks Thushan for the update. >>>>>> >>>>>> In addition to you digging into the code, can you also inquire on the >>>>>> poor performance from the DL4J user list (if there's one exist)? >>>>>> >>>>>> On Tue, Jun 16, 2015 at 5:27 AM, Thushan Ganegedara >>>>> > wrote: >>>>>> >>>>>>> Dear all, >>>>>>> >>>>>>> Please find the update regarding DL4J testing >>>>>>> >>>>>>> *Poor Accuracy* >>>>>>> I have been testing DL4J extensively with *MNIST and Iris* datasets >>>>>>> (Small and Full). However, I was unable to get a reasonable accuracy >>>>>>> with >>>>>>> DL4J for the aforementioned datasets. The F1-score was around 0.02, >>>>>>> which >>>>>>> is very low. >>>>>>> >>>>>>> I tried with different settings mainly for the following attributes >>>>>>> >>>>>>> Weight initialization >>>>>>> Gradient Descent >>>>>>> Iterations >>>>>>> Type of units: Autoencoder/RBM >>>>>>> >>>>>>> >>>>>>> But none of the settings gave a reasonable accuracy. Furthermore, >>>>>>> the predicted values for the test data usually *belong to 1 or 2 >>>>>>> classes *(e.g. when trained on MNIST dataset, the program predict 0 >>>>>>> and 1 only, though there are 10 possible classes) >>>>>>> >>>>>>> Also there are many reports of *poor accuracy of DL4J.* The best >>>>>>> accuracy I could find reported was around 0.5 F1 score for MNIST, which >>>&g
Re: [Dev] [ML] Progress so far with DL4J
Dear all, I posted about poor accuracy on deeplearning4j google group and they replied saying they are aware of this issue ( https://groups.google.com/forum/#!topic/deeplearning4j/gRQH8rA9SQI). They are currently working on it, to get the tuned examples out soon. However, from the appearance of their response, deeplearning4j has few bugs/issues with gradient calculation after they moved to nd4j. I can see two options right now, 1. Wait till the bugs are fixed and tuned examples are out 2. Write autoencoder myself, adhering to the coding style they have followed. So the example can be easily integrated with deeplearning4j after a stable version is out. Would highly appreciate your feedback. On Wed, Jun 17, 2015 at 10:27 AM, Thushan Ganegedara wrote: > Hi, > > I couldn't find any reports on high accuracy on any set of data. The > popular datasets seem to be Iris dataset and MNIST dataset. However, their > example page on git hub shows several results for different techniques and > datasets (https://github.com/deeplearning4j/dl4j-0.0.3.3-examples). And > as it can be seen, the accuracies mentioned are quite low. (However, the > developers claim that this is because they have run the examples with less > number of nodes and less iterations) > > > > On Tue, Jun 16, 2015 at 6:03 PM, Srinath Perera wrote: > >> Are there any other datasets where dl4j suppose to do well? As long as it >> does better with *some* datasets, we can go ahead with those? >> >> On Tue, Jun 16, 2015 at 9:13 AM, Thushan Ganegedara >> wrote: >> >>> Yes, there are few use list (Git hub and google group). I will inquire >>> about this in user lists. >>> >>> Thank you >>> >>> >>> On Tue, Jun 16, 2015 at 12:34 PM, Nirmal Fernando >>> wrote: >>> >>>> Thanks Thushan for the update. >>>> >>>> In addition to you digging into the code, can you also inquire on the >>>> poor performance from the DL4J user list (if there's one exist)? >>>> >>>> On Tue, Jun 16, 2015 at 5:27 AM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Dear all, >>>>> >>>>> Please find the update regarding DL4J testing >>>>> >>>>> *Poor Accuracy* >>>>> I have been testing DL4J extensively with *MNIST and Iris* datasets >>>>> (Small and Full). However, I was unable to get a reasonable accuracy with >>>>> DL4J for the aforementioned datasets. The F1-score was around 0.02, which >>>>> is very low. >>>>> >>>>> I tried with different settings mainly for the following attributes >>>>> >>>>> Weight initialization >>>>> Gradient Descent >>>>> Iterations >>>>> Type of units: Autoencoder/RBM >>>>> >>>>> >>>>> But none of the settings gave a reasonable accuracy. Furthermore, the >>>>> predicted values for the test data usually *belong to 1 or 2 classes >>>>> *(e.g. >>>>> when trained on MNIST dataset, the program predict 0 and 1 only, though >>>>> there are 10 possible classes) >>>>> >>>>> Also there are many reports of *poor accuracy of DL4J.* The best >>>>> accuracy I could find reported was around 0.5 F1 score for MNIST, which is >>>>> still very low. (e.g. MNIST can easily reach 0.9+ accuracy for even a >>>>> basic deep network) >>>>> >>>>> I'm currently trying to delve in to the code for DL4J and figure out >>>>> how the learning is done. I'm assuming there are some faults in the >>>>> learning process which causes the algorithm to learn poorly. >>>>> >>>>> Thank you >>>>> >>>>> -- >>>>> Regards, >>>>> >>>>> Thushan Ganegedara >>>>> School of IT >>>>> University of Sydney, Australia >>>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Thanks & regards, >>>> Nirmal >>>> >>>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>>> Mobile: +94715779733 >>>> Blog: http://nirmalfdo.blogspot.com/ >>>> >>>> >>>> >>> >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Blog: http://srinathsview.blogspot.com twitter:@srinath_perera >> Site: http://people.apache.org/~hemapani/ >> Photos: http://www.flickr.com/photos/hemapani/ >> Phone: 0772360902 >> > > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Progress so far with DL4J
Hi, I couldn't find any reports on high accuracy on any set of data. The popular datasets seem to be Iris dataset and MNIST dataset. However, their example page on git hub shows several results for different techniques and datasets (https://github.com/deeplearning4j/dl4j-0.0.3.3-examples). And as it can be seen, the accuracies mentioned are quite low. (However, the developers claim that this is because they have run the examples with less number of nodes and less iterations) On Tue, Jun 16, 2015 at 6:03 PM, Srinath Perera wrote: > Are there any other datasets where dl4j suppose to do well? As long as it > does better with *some* datasets, we can go ahead with those? > > On Tue, Jun 16, 2015 at 9:13 AM, Thushan Ganegedara > wrote: > >> Yes, there are few use list (Git hub and google group). I will inquire >> about this in user lists. >> >> Thank you >> >> >> On Tue, Jun 16, 2015 at 12:34 PM, Nirmal Fernando >> wrote: >> >>> Thanks Thushan for the update. >>> >>> In addition to you digging into the code, can you also inquire on the >>> poor performance from the DL4J user list (if there's one exist)? >>> >>> On Tue, Jun 16, 2015 at 5:27 AM, Thushan Ganegedara >>> wrote: >>> >>>> Dear all, >>>> >>>> Please find the update regarding DL4J testing >>>> >>>> *Poor Accuracy* >>>> I have been testing DL4J extensively with *MNIST and Iris* datasets >>>> (Small and Full). However, I was unable to get a reasonable accuracy with >>>> DL4J for the aforementioned datasets. The F1-score was around 0.02, which >>>> is very low. >>>> >>>> I tried with different settings mainly for the following attributes >>>> >>>> Weight initialization >>>> Gradient Descent >>>> Iterations >>>> Type of units: Autoencoder/RBM >>>> >>>> >>>> But none of the settings gave a reasonable accuracy. Furthermore, the >>>> predicted values for the test data usually *belong to 1 or 2 classes *(e.g. >>>> when trained on MNIST dataset, the program predict 0 and 1 only, though >>>> there are 10 possible classes) >>>> >>>> Also there are many reports of *poor accuracy of DL4J.* The best >>>> accuracy I could find reported was around 0.5 F1 score for MNIST, which is >>>> still very low. (e.g. MNIST can easily reach 0.9+ accuracy for even a >>>> basic deep network) >>>> >>>> I'm currently trying to delve in to the code for DL4J and figure out >>>> how the learning is done. I'm assuming there are some faults in the >>>> learning process which causes the algorithm to learn poorly. >>>> >>>> Thank you >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Blog: http://srinathsview.blogspot.com twitter:@srinath_perera > Site: http://people.apache.org/~hemapani/ > Photos: http://www.flickr.com/photos/hemapani/ > Phone: 0772360902 > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Progress so far with DL4J
Hi, Thank you for your feedback. As for the moment I haven't tried CNN. However, we do not need CNNs for the MNIST dataset. With a 3-layer SAE (Stacked AE) we can easily reach an accuracy of 98%+ (Without any special regularization). As I've tested this in python with a hand-written SAE Also, CNN creates quite an ovehead with many sub-steps (convolution, subsampling, pooling) as far as I know about it. CNNs do quite a good job for high-resolution images, but in this case it will create a substantial overhead. However, I will try it and see. Thank you for your suggestion On Tue, Jun 16, 2015 at 4:09 PM, CD Athuraliya wrote: > Hi Thushan, > > Assuming your are running examples, did you try CNN for MNIST data? What > is the accuracy? > > You can use DL4J Gitter chat room [1] for quick responses from devs. > > [1] https://gitter.im/deeplearning4j/deeplearning4j > > Regards, > CD > > On Tue, Jun 16, 2015 at 9:13 AM, Thushan Ganegedara > wrote: > >> Yes, there are few use list (Git hub and google group). I will inquire >> about this in user lists. >> >> Thank you >> >> >> On Tue, Jun 16, 2015 at 12:34 PM, Nirmal Fernando >> wrote: >> >>> Thanks Thushan for the update. >>> >>> In addition to you digging into the code, can you also inquire on the >>> poor performance from the DL4J user list (if there's one exist)? >>> >>> On Tue, Jun 16, 2015 at 5:27 AM, Thushan Ganegedara >>> wrote: >>> >>>> Dear all, >>>> >>>> Please find the update regarding DL4J testing >>>> >>>> *Poor Accuracy* >>>> I have been testing DL4J extensively with *MNIST and Iris* datasets >>>> (Small and Full). However, I was unable to get a reasonable accuracy with >>>> DL4J for the aforementioned datasets. The F1-score was around 0.02, which >>>> is very low. >>>> >>>> I tried with different settings mainly for the following attributes >>>> >>>> Weight initialization >>>> Gradient Descent >>>> Iterations >>>> Type of units: Autoencoder/RBM >>>> >>>> >>>> But none of the settings gave a reasonable accuracy. Furthermore, the >>>> predicted values for the test data usually *belong to 1 or 2 classes *(e.g. >>>> when trained on MNIST dataset, the program predict 0 and 1 only, though >>>> there are 10 possible classes) >>>> >>>> Also there are many reports of *poor accuracy of DL4J.* The best >>>> accuracy I could find reported was around 0.5 F1 score for MNIST, which is >>>> still very low. (e.g. MNIST can easily reach 0.9+ accuracy for even a >>>> basic deep network) >>>> >>>> I'm currently trying to delve in to the code for DL4J and figure out >>>> how the learning is done. I'm assuming there are some faults in the >>>> learning process which causes the algorithm to learn poorly. >>>> >>>> Thank you >>>> >>>> -- >>>> Regards, >>>> >>>> Thushan Ganegedara >>>> School of IT >>>> University of Sydney, Australia >>>> >>> >>> >>> >>> -- >>> >>> Thanks & regards, >>> Nirmal >>> >>> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >>> Mobile: +94715779733 >>> Blog: http://nirmalfdo.blogspot.com/ >>> >>> >>> >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > *CD Athuraliya* > Software Engineer > WSO2, Inc. > lean . enterprise . middleware > Mobile: +94 716288847 <94716288847> > LinkedIn <http://lk.linkedin.com/in/cdathuraliya> | Twitter > <https://twitter.com/cdathuraliya> | Blog > <http://cdathuraliya.tumblr.com/> > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [ML] Progress so far with DL4J
Yes, there are few use list (Git hub and google group). I will inquire about this in user lists. Thank you On Tue, Jun 16, 2015 at 12:34 PM, Nirmal Fernando wrote: > Thanks Thushan for the update. > > In addition to you digging into the code, can you also inquire on the poor > performance from the DL4J user list (if there's one exist)? > > On Tue, Jun 16, 2015 at 5:27 AM, Thushan Ganegedara > wrote: > >> Dear all, >> >> Please find the update regarding DL4J testing >> >> *Poor Accuracy* >> I have been testing DL4J extensively with *MNIST and Iris* datasets >> (Small and Full). However, I was unable to get a reasonable accuracy with >> DL4J for the aforementioned datasets. The F1-score was around 0.02, which >> is very low. >> >> I tried with different settings mainly for the following attributes >> >> Weight initialization >> Gradient Descent >> Iterations >> Type of units: Autoencoder/RBM >> >> >> But none of the settings gave a reasonable accuracy. Furthermore, the >> predicted values for the test data usually *belong to 1 or 2 classes *(e.g. >> when trained on MNIST dataset, the program predict 0 and 1 only, though >> there are 10 possible classes) >> >> Also there are many reports of *poor accuracy of DL4J.* The best >> accuracy I could find reported was around 0.5 F1 score for MNIST, which is >> still very low. (e.g. MNIST can easily reach 0.9+ accuracy for even a >> basic deep network) >> >> I'm currently trying to delve in to the code for DL4J and figure out how >> the learning is done. I'm assuming there are some faults in the learning >> process which causes the algorithm to learn poorly. >> >> Thank you >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] Progress so far with DL4J
Dear all, Please find the update regarding DL4J testing *Poor Accuracy* I have been testing DL4J extensively with *MNIST and Iris* datasets (Small and Full). However, I was unable to get a reasonable accuracy with DL4J for the aforementioned datasets. The F1-score was around 0.02, which is very low. I tried with different settings mainly for the following attributes Weight initialization Gradient Descent Iterations Type of units: Autoencoder/RBM But none of the settings gave a reasonable accuracy. Furthermore, the predicted values for the test data usually *belong to 1 or 2 classes *(e.g. when trained on MNIST dataset, the program predict 0 and 1 only, though there are 10 possible classes) Also there are many reports of *poor accuracy of DL4J.* The best accuracy I could find reported was around 0.5 F1 score for MNIST, which is still very low. (e.g. MNIST can easily reach 0.9+ accuracy for even a basic deep network) I'm currently trying to delve in to the code for DL4J and figure out how the learning is done. I'm assuming there are some faults in the learning process which causes the algorithm to learn poorly. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [ML] First Meeting - Deep Learning for WSO2ML
Dear all, Please find the minutes of the 1st meeting on integrating DL for WSO2ML There are 2 potential java libraries that can be used for deep learning (Recommended in DeepLearning.net) - DL4J (http://deeplearning4j.org/) - https://github.com/ivan-vasilev/neuralnetworks However, it was decided to stay with *DL4J* as it has shown to be the most promising in terms of the requirements to be satisfied. Furthermore, the initial focus will be on implementing a *Deep Autoencoder (DA)* (A variation of Deep Networks). Deep Autoencoder is essentially a stack of Multi-Layer Perceptrons with greedy layer-wise pre-training and fine-tuning by backpropagation. Deep Autoencoder was chosen based on following reasons - Simplicity (Compared to other networks such as Convolutional Networks, Stacked Restricted Boltzmann Machines) - High Accuracy (DAs has have shown promising results for various datasets. MNIST - 99.0%+ accuracy) - Easy to generalize. DA can be easily generalized for text, images, etc. After the implementation of DA, the possibility of introducing different techniques *(Denoising, Drop-out, Sparcity) *will be investigated. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2015] Issues with DeepLearning4J
Hello, Yes, I also think sticking to a previous release might be the best option here, as all the latest releases seems to be unstable in many aspects. For example some DL4J dependencies are still very primitive (e.g. Canova has only 1 release - 0.0.0.1) On Fri, Mar 27, 2015 at 3:49 PM, Nirmal Fernando wrote: > Hi Thushan, > > Great work on getting this to work! IMO we should stick to a previous > release till we get a stable new release. We can't keep a locally built one > due to the maintenance overhead. > > On Fri, Mar 27, 2015 at 10:07 AM, Thushan Ganegedara > wrote: > >> Hello, >> >> I was able to get the DeepLearning4J example working ( >> https://github.com/deeplearning4j/dl4j-0.0.3.3-examples) after some >> extensive building and dependency updates. I haven't still run anything as >> I need to bring them to produce proper outputs, but I was able to compile >> WSO2-ML without any issues while including the deeplearning4j MNIST example. >> >> Just for the record I had to follow the following procedure in order to >> get this working >> >> Download *Canova *from (https://github.com/deeplearning4j/Canova.git) >> and build the project and install dependency via mvn install:install-file >> However, in order to compile successfully, I had to skip the Tests and >> Javadoc check because otherwise I cannot compile it (figured out this after >> a discussion with one of the contributors) >> >> Download *nd4j *from (https://github.com/deeplearning4j/nd4j) and build >> it and install it via mvn install:install-file. Because the any of the >> releases do not have, do not contain the (nd4j.linalg.dataset.api.iterator) >> >> Download *DeepLearning4J *from ( >> https://github.com/deeplearning4j/deeplearning4j) and build it and >> install it via mvn install:install-file. Because the any of the releases do >> not have org.deepleaning4j.nn.conf.override package >> >> After the process I have following versions >> >>- Canova - 0.0.0.2-SNAPSHOT >>- nd4j - 0.0.3.5.5.3-SNAPSHOT >>- dl4j - 0.0.3.3.3.alpha1-SNAPSHOT >> >> >> In these conditions it compiles without any issues. >> >> So I guess we have 2 major options >> >>- continue with the locally built dependencies or >>- switch to a previous release which supports this example ( >>http://deeplearning4j.org/mnist-tutorial.html) >> >> >> Thank you >> >> >> On Fri, Mar 27, 2015 at 7:03 AM, Thushan Ganegedara >> wrote: >> >>> We can try to maybe use a previous release corresponding to the example >>> at http://deeplearning4j.org/mnist-tutorial.html (may be <= 0.0.3.2) >>> However, as the explain this in there github issue >>> https://github.com/deeplearning4j/deeplearning4j/issues/173 using it >>> reduces the generality of the deepLearning. But I think it can get the work >>> done. >>> >>> >>> >>> On Thu, Mar 26, 2015 at 11:04 PM, Nirmal Fernando >>> wrote: >>> >>>> Can't we use a released version and samples corresponds to a released >>>> version? >>>> >>>> On Thu, Mar 26, 2015 at 5:29 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Dear Devs, >>>>> >>>>> For the past few days, I've been try to add deeplearning4j dependency >>>>> with maven and was able to do it successfully. >>>>> >>>>> However, the example provided in >>>>> http://deeplearning4j.org/mnist-tutorial.html uses a class called *DBN >>>>> *which is no longer available in the latest deeplearning4j releases. >>>>> >>>>> Furthermore, they have other examples available at >>>>> https://github.com/deeplearning4j/dl4j-0.0.3.3-examples which again >>>>> uses a class (i.e. org.deeplearning4j.nn.conf.override. >>>>> *ClassifierOverride*) which is not available in any of the releases, >>>>> but only the source. I asked about this from the contributors and they >>>>> told >>>>> me to build from the source. >>>>> >>>>> But when I tried that, I was unable to get it to compile, as Canova >>>>> (0.0.0.2) cannot find one of it's dependencies ( >>>>> *openjfx-78-backport-native*) and it's not mentioned in Canova's >>>>> pom.xml Therefore I tried including it in the pom.xml and compiling it. >>>>> But >>>>> it's still giving
Re: [Dev] [GSOC2015] Issues with DeepLearning4J
Hello, I was able to get the DeepLearning4J example working ( https://github.com/deeplearning4j/dl4j-0.0.3.3-examples) after some extensive building and dependency updates. I haven't still run anything as I need to bring them to produce proper outputs, but I was able to compile WSO2-ML without any issues while including the deeplearning4j MNIST example. Just for the record I had to follow the following procedure in order to get this working Download *Canova *from (https://github.com/deeplearning4j/Canova.git) and build the project and install dependency via mvn install:install-file However, in order to compile successfully, I had to skip the Tests and Javadoc check because otherwise I cannot compile it (figured out this after a discussion with one of the contributors) Download *nd4j *from (https://github.com/deeplearning4j/nd4j) and build it and install it via mvn install:install-file. Because the any of the releases do not have, do not contain the (nd4j.linalg.dataset.api.iterator) Download *DeepLearning4J *from ( https://github.com/deeplearning4j/deeplearning4j) and build it and install it via mvn install:install-file. Because the any of the releases do not have org.deepleaning4j.nn.conf.override package After the process I have following versions - Canova - 0.0.0.2-SNAPSHOT - nd4j - 0.0.3.5.5.3-SNAPSHOT - dl4j - 0.0.3.3.3.alpha1-SNAPSHOT In these conditions it compiles without any issues. So I guess we have 2 major options - continue with the locally built dependencies or - switch to a previous release which supports this example ( http://deeplearning4j.org/mnist-tutorial.html) Thank you On Fri, Mar 27, 2015 at 7:03 AM, Thushan Ganegedara wrote: > We can try to maybe use a previous release corresponding to the example at > http://deeplearning4j.org/mnist-tutorial.html (may be <= 0.0.3.2) > However, as the explain this in there github issue > https://github.com/deeplearning4j/deeplearning4j/issues/173 using it > reduces the generality of the deepLearning. But I think it can get the work > done. > > > > On Thu, Mar 26, 2015 at 11:04 PM, Nirmal Fernando wrote: > >> Can't we use a released version and samples corresponds to a released >> version? >> >> On Thu, Mar 26, 2015 at 5:29 PM, Thushan Ganegedara >> wrote: >> >>> Dear Devs, >>> >>> For the past few days, I've been try to add deeplearning4j dependency >>> with maven and was able to do it successfully. >>> >>> However, the example provided in >>> http://deeplearning4j.org/mnist-tutorial.html uses a class called *DBN >>> *which >>> is no longer available in the latest deeplearning4j releases. >>> >>> Furthermore, they have other examples available at >>> https://github.com/deeplearning4j/dl4j-0.0.3.3-examples which again >>> uses a class (i.e. org.deeplearning4j.nn.conf.override. >>> *ClassifierOverride*) which is not available in any of the releases, >>> but only the source. I asked about this from the contributors and they told >>> me to build from the source. >>> >>> But when I tried that, I was unable to get it to compile, as Canova >>> (0.0.0.2) cannot find one of it's dependencies ( >>> *openjfx-78-backport-native*) and it's not mentioned in Canova's >>> pom.xml Therefore I tried including it in the pom.xml and compiling it. But >>> it's still giving me some problems. >>> >>> So by looking at all these information, deeplearning4j seems a little >>> bit unstable at the moment. >>> >>> I would highly appreciate your thoughts about this and if there are any >>> alternative suggestions. >>> >>> Thank you >>> >>> >>> -- >>> Regards, >>> >>> Thushan Ganegedara >>> School of IT >>> University of Sydney, Australia >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Senior Software Engineer- Platform Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] [GSOC2015] Issues with DeepLearning4J
We can try to maybe use a previous release corresponding to the example at http://deeplearning4j.org/mnist-tutorial.html (may be <= 0.0.3.2) However, as the explain this in there github issue https://github.com/deeplearning4j/deeplearning4j/issues/173 using it reduces the generality of the deepLearning. But I think it can get the work done. On Thu, Mar 26, 2015 at 11:04 PM, Nirmal Fernando wrote: > Can't we use a released version and samples corresponds to a released > version? > > On Thu, Mar 26, 2015 at 5:29 PM, Thushan Ganegedara > wrote: > >> Dear Devs, >> >> For the past few days, I've been try to add deeplearning4j dependency >> with maven and was able to do it successfully. >> >> However, the example provided in >> http://deeplearning4j.org/mnist-tutorial.html uses a class called *DBN *which >> is no longer available in the latest deeplearning4j releases. >> >> Furthermore, they have other examples available at >> https://github.com/deeplearning4j/dl4j-0.0.3.3-examples which again uses >> a class (i.e. org.deeplearning4j.nn.conf.override.*ClassifierOverride*) >> which is not available in any of the releases, but only the source. I asked >> about this from the contributors and they told me to build from the source. >> >> But when I tried that, I was unable to get it to compile, as Canova >> (0.0.0.2) cannot find one of it's dependencies ( >> *openjfx-78-backport-native*) and it's not mentioned in Canova's pom.xml >> Therefore I tried including it in the pom.xml and compiling it. But it's >> still giving me some problems. >> >> So by looking at all these information, deeplearning4j seems a little >> bit unstable at the moment. >> >> I would highly appreciate your thoughts about this and if there are any >> alternative suggestions. >> >> Thank you >> >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> > > > > -- > > Thanks & regards, > Nirmal > > Senior Software Engineer- Platform Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] [GSOC2015] Issues with DeepLearning4J
Dear Devs, For the past few days, I've been try to add deeplearning4j dependency with maven and was able to do it successfully. However, the example provided in http://deeplearning4j.org/mnist-tutorial.html uses a class called *DBN *which is no longer available in the latest deeplearning4j releases. Furthermore, they have other examples available at https://github.com/deeplearning4j/dl4j-0.0.3.3-examples which again uses a class (i.e. org.deeplearning4j.nn.conf.override.*ClassifierOverride*) which is not available in any of the releases, but only the source. I asked about this from the contributors and they told me to build from the source. But when I tried that, I was unable to get it to compile, as Canova (0.0.0.2) cannot find one of it's dependencies (*openjfx-78-backport-native*) and it's not mentioned in Canova's pom.xml Therefore I tried including it in the pom.xml and compiling it. But it's still giving me some problems. So by looking at all these information, deeplearning4j seems a little bit unstable at the moment. I would highly appreciate your thoughts about this and if there are any alternative suggestions. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Interested in Deep Learning Integration for Machine Learner - GSOC 2015
Dear Devs, On the DeepLearning4J website it says, "*ND4J requires Java 7*" But on the WSO2-ml docs it says "*To build the product from the source distribution, you must use JDK 1.6 instead of JDK 1.7*" It seems that this could cause some incompatibilities during building. Is there any workaround to overcome this incompatibility? On Tue, Mar 24, 2015 at 4:35 PM, Nirmal Fernando wrote: > Thushan, > > Proposal looks good, please go ahead and start creating the proposal in > Melange. > > On Mon, Mar 23, 2015 at 7:50 PM, Nirmal Fernando wrote: > >> Please try out the M1 release for a better understanding on the back-end: >> https://github.com/wso2/product-ml/releases/tag/1.0.0-m1 >> >> On Mon, Mar 23, 2015 at 4:59 PM, Nirmal Fernando wrote: >> >>> Thanks Thushan. We'll review it and let you know. >>> >>> Btw, you tried latest ML code right? >>> >>> On Mon, Mar 23, 2015 at 4:56 PM, Thushan Ganegedara >>> wrote: >>> >>>> Dear Mentors, >>>> >>>> I have installed and run the WSO2 Machine Learner successfully. For the >>>> past few days I've been looking in details at the WSO2-ML and >>>> DeepLearning4J. >>>> >>>> I've prepared a *draft proposal* with my knowledge of deep learning >>>> and how deep learning can be beneficial in a business perspective. Please >>>> find it attached herewith. >>>> >>>> I would highly appreciate if you could provide some feedback regarding >>>> the proposal. >>>> >>>> Thank you >>>> >>>> >>>> On Tue, Mar 17, 2015 at 4:36 PM, Thushan Ganegedara >>>> wrote: >>>> >>>>> Dear Nirmal, >>>>> >>>>> Thank you very much for sharing the previous e-mail trails. I will >>>>> look into the resources mentioned in the emails. However, I have been >>>>> watching lectures, and following tutorials on http://deeplearning.net/ >>>>> for a while, which acts as the primary hub for deep learning knowledge >>>>> sharing. >>>>> >>>>> My next step will be to play around with DeepLearning4J as I've been >>>>> implementing deep learning in Python (Theano) before this. I think I'll >>>>> soon be able to figure out what type of functionality provided by >>>>> DeepLearning4J. >>>>> >>>>> That will help me to come up with a strong proposal that leverages >>>>> functionality of DeepLearning4J to the maximum. >>>>> >>>>> >>>>> Thank you very much for pointing me to the other resources. >>>>> >>>>> On Tue, Mar 17, 2015 at 4:01 PM, Nirmal Fernando >>>>> wrote: >>>>> >>>>>> Thanks for your interest on this project, Thushan. >>>>>> >>>>>> We already had a discussion with another student (unfortunately, it >>>>>> appeared he's not eligible to participate in GSoC) at http://wso2 >>>>>> .markmail.org/search/list:org.wso2.dev#query:list%3Aorg.wso2 >>>>>> .dev%20from%3A%22Nirmal%20Fernando%22+page:1+mid:3brcf5pex7zdcitm+state:results >>>>>> >>>>>> Also, we had another discussion here: >>>>>> http://mail.wso2.org/mailarchive/dev/2015-March/thread.html#44758 >>>>>> where we shared some resources that you could look at. >>>>>> >>>>>> On WSO2 Machine Learner, I'll share a presentation created by Srinath. >>>>>> >>>>>> On Tue, Mar 17, 2015 at 10:23 AM, Thushan Ganegedara < >>>>>> thu...@gmail.com> wrote: >>>>>> >>>>>>> Dear Developers, >>>>>>> >>>>>>> I am Thushan Ganegedara, a PhD student at University of Sydney. I >>>>>>> graduated from University of Moratuwa, Sri Lanka in 2014. >>>>>>> >>>>>>> I became quite interested in the idea of Deep Learning for WSO2 >>>>>>> Machine Learner for GSOC 2015. Neural networks is one my major >>>>>>> strengths as >>>>>>> I have been working with Neural Networks (specifically >>>>>>> Self-Organizing-Maps >>>>>>> and Back-propagation) (Java) for almost 2 years so far. >>>>>>> Furthermore, I am currently pursuing my PhD in representation
Re: [Dev] HTTP Status 405 - HTTP method GET is not supported by this URL [Machin Learner]
Dear Devs, The INSTALL.TXT file present at the* product-ml\modules\distribution\src *location says that, 1. The directory containing this INSTALL.txt file will be the root and it will be treated as Carbon home directory and let it be CARBON_HOME. But if I try to set this directory as CARBON_HOME, the console gives an error saying CARBON_HOME variable is incorrect (because the above location doesn't have carbon related files there) Then I downloaded Carbon (4.3.0) and set up the env variable so everything works perfectly. But still if I try to access *https://localhost:9443/mlUI <https://localhost:9443/mlUI>*, I'm presented with HTTP Error 405 Could you kindly point me to a more comprehensive manual which elucidates the steps to get WSO2ML working, if there's any? Thank you On Thu, Mar 19, 2015 at 2:41 PM, Thushan Ganegedara wrote: > Dear Devs, > > I have forked WSO2 Machine Learner with the intention of getting more > insight into the product. I've followed the instructions in the Read Me and > I was able to successfully start the WSO2Server. > > However, afterwards when I entered the credentials on the managing > console, it returns the following error > HTTP Status 405 - HTTP method GET is not supported by this URL > I was able to find other instances where the same error has occurred, > > https://wso2.org/jira/browse/CARBON-8032?jql=text%20~%20%22HTTP%20Status%20405%20GET%22 > > http://wso2-oxygen-tank.10903.n7.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2&query=HTTP+Status+405+-+HTTP+method+GET+is+not+supported+by+this+URL&n=2 > > But these issue trackers/forums doesn't offer a clear fix. I tried > suffixing '/' as mentioned in one post, but that didn't help me. > > I'm using Carbon 4.3.0 version > > I would highly appreciate if you could provide some insight to this. > > Thank you > > -- > Regards, > > Thushan Ganegedara > School of IT > University of Sydney, Australia > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] HTTP Status 405 - HTTP method GET is not supported by this URL [Machin Learner]
Dear Devs, I have forked WSO2 Machine Learner with the intention of getting more insight into the product. I've followed the instructions in the Read Me and I was able to successfully start the WSO2Server. However, afterwards when I entered the credentials on the managing console, it returns the following error HTTP Status 405 - HTTP method GET is not supported by this URL I was able to find other instances where the same error has occurred, https://wso2.org/jira/browse/CARBON-8032?jql=text%20~%20%22HTTP%20Status%20405%20GET%22 http://wso2-oxygen-tank.10903.n7.nabble.com/template/NamlServlet.jtp?macro=search_page&node=2&query=HTTP+Status+405+-+HTTP+method+GET+is+not+supported+by+this+URL&n=2 But these issue trackers/forums doesn't offer a clear fix. I tried suffixing '/' as mentioned in one post, but that didn't help me. I'm using Carbon 4.3.0 version I would highly appreciate if you could provide some insight to this. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
Re: [Dev] Interested in Deep Learning Integration for Machine Learner - GSOC 2015
Dear Nirmal, Thank you very much for sharing the previous e-mail trails. I will look into the resources mentioned in the emails. However, I have been watching lectures, and following tutorials on http://deeplearning.net/ for a while, which acts as the primary hub for deep learning knowledge sharing. My next step will be to play around with DeepLearning4J as I've been implementing deep learning in Python (Theano) before this. I think I'll soon be able to figure out what type of functionality provided by DeepLearning4J. That will help me to come up with a strong proposal that leverages functionality of DeepLearning4J to the maximum. Thank you very much for pointing me to the other resources. On Tue, Mar 17, 2015 at 4:01 PM, Nirmal Fernando wrote: > Thanks for your interest on this project, Thushan. > > We already had a discussion with another student (unfortunately, it > appeared he's not eligible to participate in GSoC) at http://wso2 > .markmail.org/search/list:org.wso2.dev#query:list%3Aorg.wso2 > .dev%20from%3A%22Nirmal%20Fernando%22+page:1+mid:3brcf5pex7zdcitm+state:results > > Also, we had another discussion here: > http://mail.wso2.org/mailarchive/dev/2015-March/thread.html#44758 > where we shared some resources that you could look at. > > On WSO2 Machine Learner, I'll share a presentation created by Srinath. > > On Tue, Mar 17, 2015 at 10:23 AM, Thushan Ganegedara > wrote: > >> Dear Developers, >> >> I am Thushan Ganegedara, a PhD student at University of Sydney. I >> graduated from University of Moratuwa, Sri Lanka in 2014. >> >> I became quite interested in the idea of Deep Learning for WSO2 Machine >> Learner for GSOC 2015. Neural networks is one my major strengths as I have >> been working with Neural Networks (specifically Self-Organizing-Maps and >> Back-propagation) (Java) for almost 2 years so far. >> Furthermore, I am currently pursuing my PhD in representation learning >> (i.e. Deep Learning) >> >> I have already published a paper in IJCNN 2014 >> http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6889650&tag=1 >> and expecting to publish another in IJCNN 2015 >> >> I would appreciate greatly if you could let me know any >> wiki/documentation (other than GIT Readme document and WSO2 Oxygen Tank) >> where I can familiarize myself with the Machine Learner more. >> >> >> Looking forward to hearing from you. Thank you >> >> -- >> Regards, >> >> Thushan Ganegedara >> School of IT >> University of Sydney, Australia >> >> ___ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> > > > -- > > Thanks & regards, > Nirmal > > Senior Software Engineer- Platform Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
[Dev] Interested in Deep Learning Integration for Machine Learner - GSOC 2015
Dear Developers, I am Thushan Ganegedara, a PhD student at University of Sydney. I graduated from University of Moratuwa, Sri Lanka in 2014. I became quite interested in the idea of Deep Learning for WSO2 Machine Learner for GSOC 2015. Neural networks is one my major strengths as I have been working with Neural Networks (specifically Self-Organizing-Maps and Back-propagation) (Java) for almost 2 years so far. Furthermore, I am currently pursuing my PhD in representation learning (i.e. Deep Learning) I have already published a paper in IJCNN 2014 http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=6889650&tag=1 and expecting to publish another in IJCNN 2015 I would appreciate greatly if you could let me know any wiki/documentation (other than GIT Readme document and WSO2 Oxygen Tank) where I can familiarize myself with the Machine Learner more. Looking forward to hearing from you. Thank you -- Regards, Thushan Ganegedara School of IT University of Sydney, Australia ___ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
