Hi Bernard, The ConfigEntryTable[1] is used to store the Tracing details of an EI flow, if the flow is really long this can easily exceed the "6000" character limit set in the coloumn definition. If the coloum length is not enough you can increase it by increasing the
> field.length="configData:6000" in the siddhi app residing in <EI_HOME>/wso2/analytics/wso2/worker/deployment/siddhi-files directory. Please note for this to be valid, the table must be deleted and created again by deploying the siddhi app. Please note, tracing is not recommened to be enabled y default in production environment as it will lead to high amount of data being persisted which can lead to performance drop. Hope this solves the problem! [1] https://github.com/wso2/analytics-solutions/blob/master/features/ei-analytics-feature/org.wso2.analytics.solutions.ei.analytics.feature/src/main/resources/siddhi-files/EI_Analytics_StatApp.siddhi#L51 Best Regards, *Niveathika Rajendran,* *Senior Software Engineer.* *Mobile : +94 077 903 7536* On Wed, Nov 28, 2018 at 9:13 PM Bernard Paris <bernard.pa...@uclouvain.be> wrote: > Hi Niveathika > > I checked my carbon.xml and found missing block related to > <CryptoService> > I corrected it and now the client can connect to analytics, great ! > > Anyway, the message about any "CryptoService.Secret" i missing is still > logged at server startup. > The property "CryptoService.Secret" is never referenced in any of the > config files and the documentation I found do not tell a word about it. > > But ok, I can live with this. > The connexion between the agent and analytics server is now ok, I can see > datas report in the analytics dashboard console. > Anyway there are some errors in the worker logs related to data collects: > it seems some SQL request are wrong: > > [2018-11-28 16:30:07,921] ERROR > {org.wso2.siddhi.core.stream.input.source.InputEventHandler} - Error on > 'EI_Analytics_StatApp'. Cannot execute update/insert operation (update) on > table 'ConfigEntryTable' with SQL query INSERT INTO ConfigEntryTable > (metaTenantId , hashcode , entryName , configData , eventTimestamp ) > VALUES (?, ?, ?, ?, ?) . Error in applying transport property mapping for > 'wso2event' source at 'ConfigEntryEventStream' stream. > org.wso2.extension.siddhi.store.rdbms.exception.RDBMSTableException: Error > on 'EI_Analytics_StatApp'. Cannot execute update/insert operation (update) > on table 'ConfigEntryTable' with SQL query INSERT INTO ConfigEntryTable > (metaTenantId , hashcode , entryName , configData , eventTimestamp ) > VALUES (?, ?, ?, ?, ?) . > at > org.wso2.extension.siddhi.store.rdbms.RDBMSEventTable.batchProcessInsert(RDBMSEventTable.java:733) > at > org.wso2.extension.siddhi.store.rdbms.RDBMSEventTable.updateOrAdd(RDBMSEventTable.java:622) > at > org.wso2.siddhi.core.table.record.AbstractRecordTable.updateOrAdd(AbstractRecordTable.java:323) > at > org.wso2.siddhi.core.table.Table.updateOrAddEvents(Table.java:288) > at > org.wso2.siddhi.core.query.output.callback.UpdateOrInsertTableCallback.send(UpdateOrInsertTableCallback.java:78) > > > Thanks, > Bernard > > > > Le 27 nov. 2018 à 17:35, Niveathika Rajendran <niveath...@wso2.com> a > écrit : > > Hi Bernard, > > Yes, it could be. > > After analysing the workflow, I have concluded the following, > 1. Encryption failing - Leads to an empty password > 2. The login credentials send to SP will be as follows, "admin:" i.e, > username: admin and password empty > 3. SP is refusing connection as it is a NOT valid credential > > Since the encryption is failing, please check on the mentioned configs in > "Configuring Keystores[1]" > > [1] > https://docs.wso2.com/display/ADMIN44x/Configuring+Keystores+in+WSO2+Products > > Best Regards, > * Niveathika Rajendran,* > *Senior Software Engineer.* > *Mobile : +94 077 903 7536* > > > > > > On Tue, Nov 27, 2018 at 9:30 PM Bernard Paris <bernard.pa...@uclouvain.be> > wrote: > >> >> Starting EI-640 server I see this >> >> [2018-11-27 16:56:33,612] [EI-Core] INFO - >> DefaultCryptoProviderComponent 'CryptoService.Secret' property has not been >> set. 'org.wso2.carbon.crypto.provider.SymmetricKeyInternalCryptoProvider' >> won't be registered as an internal crypto provider. Please set the secret >> if the provider needs to be registered. >> >> What is about ? Could it be related with the issue ? >> >> >> Bernard >> >> >> Le 27 nov. 2018 à 11:57, Bernard Paris <bernard.pa...@uclouvain.be> a >> écrit : >> >> Hi, >> >> I have found a data-agent-config.xml file under >> {EI_HOME}/conf/data-bridge/ where lines about SSL and >> client-trustore.jks were commented by default. >> >> I uncommented these lines and restart the server from which datas are to >> be sent to remote analytics but no way, that does absolutely no changes to >> my issue: >> Caused by: ThriftAuthenticationException(message:wrong userName or >> password) >> >> Is it necessarily to define client-trustore in this file ? (what is the >> one used if not) >> >> Can someone explain how the password >> in MessageFlowStatisticsPublisher.xml >> and MessageFlowConfigurationPublisher.xml are being cyphered by the agent >> then uncyphered on server side ? >> >> Thanks, >> Bernard >> >> >> Le 26 nov. 2018 à 09:46, Bernard Paris <bernard.pa...@uclouvain.be> a >> écrit : >> >> Hello Niveathika >> >> >> Le 19 nov. 2018 à 14:53, Niveathika Rajendran <niveath...@wso2.com> a >> écrit : >> >> Hi Bernard, >> >> In your earlier email, you mentioned that the config changes as below on >> your remote machine. >> >> <property encrypted="false" name="password"/> >> >> Is the issue still continuing? >> >> >> >> yes >> >> >> >> Could you check configurations under the <Security> tag in >> <EI_HOME>/conf/carbon.xml? I was able to reproduce this *only* with a >> faulty configuration[1]. >> >> >> >> my keystore is named wso2carbon.jks, alias and password same as >> originals; >> I imported my own certificate in this keystore under alias wso2carbon and >> set the passwd to wso2carbon >> >> >> keytool -v -list -keystore wso2carbon.jks -alias wso2carbon >> >> Alias name: wso2carbon >> Creation date: Nov 15, 2018 >> Entry type: PrivateKeyEntry >> Certificate chain length: 3 >> Certificate[1]: >> Owner: CN=wso2ei.sgsi.ucl.ac.be, OU=SGSI, O=Université catholique de >> Louvain, L=Ottignies-Louvain-la-Neuve, C=BE >> Issuer: CN=TERENA SSL CA 3, O=TERENA, L=Amsterdam, ST=Noord-Holland, C=NL >> Serial number: 516e3c8586ebb5e0aab45fdb05ae071 >> >> >> I'm sure the passwd has been changed correctly because: >> >> $ keytool -v -keypasswd -alias wso2carbon -keystore wso2carbon.jks >> Enter keystore password: >> New key password for <wso2carbon>: >> Passwords must differ >> New key password for <wso2carbon>: >> >> The same keystore jks file is used by all EI servers (EI servers are load >> balanced); and the certificate have been imported in the >> client-trustore.jks of analytics. >> >> >> >> What can explain that remote servers cannot login into Analytics as >>> localhost does ? Are there any grants to set somewhere in Analytics for >>> remote access ? >> >> Regarding the above question, In SP we do not have grants for remote >> access. The certificate keys and the hostname will be verified. Please note >> for hostname verification, it must be enabled explicitly by adding the >> below configuration in the >> <EI_HOME>/wso2/analytics/conf/worker/deployment.yaml file. >> >>> wso2.carbon: >>> >>> hostnameVerificationEnabled: true >>>> >>> >> >> >> I did this but that not resolve the problem. >> >> >> Regards from Belgium, >> >> Bernard >> >> >> >> >> >> >> [1] >> https://stackoverflow.com/questions/50642058/how-to-solve-thrift-authentication-exception-wrong-username-or-password-in-wso?answertab=votes#tab-top >> >> >> Best Regards, >> * Niveathika Rajendran,* >> *Senior Software Engineer.* >> *Mobile : +94 077 903 7536* >> >> >> >> >> >> On Mon, Nov 19, 2018 at 4:03 PM Bernard Paris <bernard.pa...@uclouvain.be> >> wrote: >> >>> Hello, >>> >>> after 3 weeks of investigations, no way about this issue: remote >>> EI-6.4.0 servers definitively cannot send their datas to the >>> Analytics-worker based WSO2 SP bundled in EI640. >>> My local EI does it without any problem. >>> >>> >>> >>> Fot now I use analytics worker out of the box (all local DBs), with >>> default 'admin' user config. >>> For remote servers I get in sender logs a strange error message about >>> :wrong userName or password althought I'm using same >>> MessageFlowConfigurationPublisher.xml and >>> MessageFlowStatisticsPublisher.xml with default admin/admin credentials as >>> my local EI. >>> >>> [ >>> DataBridge-ConnectionService-tcp://wso2ei-mgmt.sgsi.ucl.ac.be:7612-pool-4-thread-1] >>> ERROR >>> {org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker} - >>> Error while trying to connect to the endpoint. Cannot borrow client for >>> ssl://wso2ei-mgmt.sgsi.ucl.ac.be:7712. >>> org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: >>> Cannot borrow client for ssl://wso2ei-mgmt.sgsi.ucl.ac.be:7712. >>> at >>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:134) >>> at >>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:59) >>> at >>> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) >>> at java.util.concurrent.FutureTask.run(FutureTask.java:266) >>> at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) >>> at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) >>> at java.lang.Thread.run(Thread.java:748) >>> Caused by: >>> org.wso2.carbon.databridge.agent.exception.DataEndpointLoginException: >>> Error while trying to login to the data receiver. >>> at >>> org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:49) >>> at >>> org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:128) >>> ... 6 more >>> Caused by: ThriftAuthenticationException(*message:wrong userName or >>> password)* >>> at >>> org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result$connect_resultStandardScheme.read(ThriftSecureEventTransmissionService.java:2020) >>> at >>> org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result$connect_resultStandardScheme.read(ThriftSecureEventTransmissionService.java:1998) >>> at >>> org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$connect_result.read(ThriftSecureEventTransmissionService.java:1940) >>> at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78) >>> at >>> org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.recv_connect(ThriftSecureEventTransmissionService.java:110) >>> at >>> org.wso2.carbon.databridge.commons.thrift.service.secure.ThriftSecureEventTransmissionService$Client.connect(ThriftSecureEventTransmissionService.java:96) >>> at >>> org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftDataEndpoint.login(ThriftDataEndpoint.java:47) >>> ... 7 more >>> >>> >>> In analytics logs I see the ssl connexion succeeds but the login fails >>> despite default admin/admin has not been changed in analytics-worker: >>> >>> [2018-11-14 14:21:51,735] INFO >>> {org.wso2.carbon.databridge.core.DataBridge} - user admin connected >>> [2018-11-14 14:21:51,749] ERROR >>> {org.wso2.carbon.databridge.core.internal.authentication.CarbonAuthenticationHandler} >>> - Authentication failed for username 'admin'. Error : >>> 'Invalid_Credentials'. Error Description : 'The login credential used for >>> login are invalid, username : 'admin'.' >>> [2018-11-14 14:21:51,750] ERROR >>> {org.wso2.carbon.databridge.core.internal.authentication.Authenticator} - >>> wrong userName or password >>> >>> >>> Config: >>> <eventPublisher name="MessageFlowStatisticsPublisher" >>> statistics="disable" trace="disable" xmlns=" >>> http://wso2.org/carbon/eventpublisher";> >>> <from streamName="org.wso2.esb.analytics.stream.FlowEntry" >>> version="1.0.0"/> >>> <mapping customMapping="disable" type="wso2event"/> >>> <to eventAdapterType="wso2event"> >>> <property name="username">admin</property> >>> <property name="protocol">thrift</property> >>> <property name="publishingMode">non-blocking</property> >>> <property name="publishTimeout">0</property> >>> <property name="receiverURL">tcp://10.1.3.12:7612</property> >>> <property encrypted="false" name="password">admin</property> >>> </to> >>> </eventPublisher> >>> >>> >>> What can explain that remote servers cannot login into Analytics as >>> localhost does ? Are there any grants to set somewhere in Analytics for >>> remote access ? >>> >>> >>> Thanks >>> Bernard >>> >>> >>> >>> >>> >>> Le 7 nov. 2018 à 10:32, Niveathika Rajendran <niveath...@wso2.com> a >>> écrit : >>> >>> Hi Bernard, >>> >>> The new EI Analytics profile is based on Carbon5, which is designed to >>> run with minimum configurations. As mentioned in the documentation, these >>> are the default configurations applicable. >>> >>> You can override it by adding the 'auth.configs' element in the >>> deployment.yaml file found under {EI_HOME}//wso2/analytics/conf/worker/ >>> directory. The following configs will add another user to the system namely >>> admin2/admin >>> >>> auth.configs: >>>> type: 'local' >>>> userManager: >>>> adminRole: admin >>>> userStore: >>>> users: >>>> - >>>> user: >>>> username: admin >>>> password: YWRtaW4= >>>> roles: 1 >>>> >>> - >>> >>> user: >>>> username: admin2 >>>> password: YWRtaW4= >>> >>> roles: 1 >>> >>> roles: >>>> - >>>> role: >>>> id: 1 >>>> displayName: admin >>>> >>> >>> Best Regards, >>> * Niveathika Rajendran,* >>> *Senior Software Engineer.* >>> */Mobile : +94 077 903 7536* >>> >>> >>> >>> >>> >>> On Wed, Nov 7, 2018 at 2:46 PM Bernard Paris <bernard.pa...@uclouvain.be> >>> wrote: >>> >>>> Hello Niveathika >>>> >>>> in the analytics distribution inside EI_640, I can't find any >>>> file-based user store you are talking about. >>>> Neither I can't find any auth.configs to configure in deployment.yaml >>>> as described at https://docs.ws >>>> o2.com/display/SP430/User+Management+via+the+IdP+Client+Interface >>>> >>>> To answer your questions here is my config just before deployment >>>> <property name="receiverURL">tcp://10.1.3.12:7612</property> >>>> <property encrypted="false" name="password">admin</property> >>>> >>>> >>>> Thanks for nay help, regards from Belgium, >>>> Bernard >>>> >>>> >>>> Le 6 nov. 2018 à 14:04, Niveathika Rajendran <niveath...@wso2.com> a >>>> écrit : >>>> >>>> Hi Bernard, >>>> >>>> Latest EI-Analytics profile is based on WSO2 Stream Processor for which >>>> the configurations are different from the previous versions. >>>> >>>> EI-Analytics 6.4.0 by default uses a file-based user store(via Local >>>> IdP Client). As you have mentioned, by default admin/admin is the user >>>> credentials used. with admin/admin credentials EI instance should have been >>>> able to publish to Analytics profile w/o any issue. >>>> >>>> Could you give follwing information to reproduce this issue? >>>> 1. Is the TCP protocol used for publishing or SSL protocol? (i.e >>>> <property name="receiverURL">tcp://localhost:7612</property> ) >>>> 2. Is the password encrypted? (i.e <property encrypted="false" >>>> name="password">admin</property>) >>>> >>>> In the meantime, please refer to "User Management via IdP client >>>> Interface" for more information on setting up the user store. >>>> >>>> Best Regards, >>>> Niveathika Rajendran, >>>> Senior Software Engineer. >>>> Mobile : +94 077 903 7536 >>>> >>>> >>>> >>>> >>>> >>>> On Tue, Nov 6, 2018 at 3:59 PM Bernard Paris < >>>> bernard.pa...@uclouvain.be> wrote: >>>> Hi Ramindu, >>>> >>>> thanks for this. >>>> >>>> Effectively I saw that everything has been changed in new analytics >>>> distribution, and I can't cope with it ;-( >>>> By default I don't see any users stuffs, like the use user-mgt.xml >>>> file. So I suppose the "admin" user is just store in the DB, nothing else. >>>> >>>> I didn't change the default password for it, but remote ESB instances >>>> are unable to send data to analytics because of >>>> ERROR >>>> {org.wso2.carbon.databridge.core.internal.authentication.CarbonAuthenticationHandler} >>>> - Authentication failed for username 'admin'. Error : >>>> 'Invalid_Credentials'. Error Description : 'The login credential used for >>>> login are invalid, username : 'admin'.' >>>> ERROR >>>> {org.wso2.carbon.databridge.core.internal.authentication.Authenticator} - >>>> wrong userName or password >>>> >>>> Local instance (I mean IE640 running on same localhost) can. >>>> >>>> I just edited this files to put admin/admin as user/passwd >>>> <EI_HOME>/repository/deployment/server/eventpublishers/MessageFlowConfigurationPublisher.xml >>>> & MessageFlowStatisticsPublisher.xml >>>> >>>> >>>> >>>> Thanks for your help, >>>> Bernard >>>> >>>> >>>> Le 1 nov. 2018 à 14:34, Ramindu De Silva <ramin...@wso2.com> a écrit : >>>> >>>> Hi Bernard, >>>> >>>> EI analytics 6.3.0 uses WSO2 DAS runtime. EI analytics 6.4.0 uses WSO2 >>>> SP runtime. WSO2 SP is a total rewrite of the WSO2 DAS, which has a feature >>>> enhancements as well. >>>> >>>> On Tue, Oct 23, 2018 at 1:50 PM Bernard Paris < >>>> bernard.pa...@uclouvain.be> wrote: >>>> Hi devs, >>>> >>>> we were using these only 4 DBs to make analytics aggregating datas from >>>> our ESB until version 6.3.0 ; >>>> ANALYTICS_CARBON_DB >>>> ANALYTICS_METRICS_DB >>>> ANALYTICS_EVENT_STORE_DB >>>> ANALYTICS_PROCESSED_DATA_STORE_DB >>>> We have our from the scratch data aggregation mechanism instead of >>>> using apache spark which was in DAS. And that the the explanation for not >>>> having configurations for ANALYTICS_EVENT_STORE_DB and >>>> ANALYTICS_PROCESSED_DATA_STORE_DB's . >>>> We still have the ANALYTICS_CARBON_DB and ANALYTICS_METRICS_DB. >>>> Please look into Monitoring Stream Processor in-order to configure the >>>> metrics for WSO2 SP. >>>> >>>> These were postgres databases. >>>> Now I see there are more then 10 databases preconfigured in the default >>>> analytics 6.4.0 config (conf/dashboard/deployment.yaml and >>>> conf/worker/deployment.yaml). >>>> Well, … this is suggesting me lot of questions. >>>> >>>> First of all, is it still recommended (like it is for ESB&DSS >>>> databases) to do *not* use local H2 databases in production environment ? >>>> This question comes because the 6.4.0 analytics seems to me to be used >>>> as it is, out of the box, lot of DBs and no documentation for a any >>>> configuration as it was for previous versions. >>>> Yes. We still recommend NOT to use the embedded H2 databases. >>>> >>>> If we need to create external DBs for all the stuffs, what exactly are >>>> each DB for ? >>>> Please refer Configuring Datasources >>>> >>>> Is there any migration tool and/or documentation about migrating from >>>> analytics 6.3.0 to 6.4.0 ? ( >>>> https://docs.wso2.com/display/EI640/Upgrading+from+WSO2+EI+6.3.0 does't >>>> talk >>>> about that) >>>> >>>> Any matching between former 4 DBs and the 6.4.0 new ones ? >>>> ANALYTICS_CARBON_DB - In-order to use/ migrate this database the >>>> previous analytics, please answer the following questions >>>> Seems, you are not using a usr-mgt database, Do you have add additional >>>> users? If not its not necessary to migrate the carbon db. You just can move >>>> on with a new db >>>> ANALYTICS_METRICS_DB - Metrics values stored in the DB depends on the >>>> node that we run. But IMO since this is a new version of the product, there >>>> is no use of migrating the older metrics data into the new one. >>>> ANALYTICS_EVENT_STORE_DB and ANALYTICS_PROCESSED_DATA_STORE_DB - This >>>> is replaced by aggregation tables. And aggregation will be done via Siddhi >>>> >>>> Is there a way to keep (transfer into 6.4.0) datas we collected with >>>> previous analytics version ? >>>> We are currently looking at several methods in-order to migrate the >>>> ANALYTICS_EVENT_STORE_DB and ANALYTICS_PROCESSED_DATA_STORE_DB data and we >>>> will update you on that regard. >>>> >>>> Thanks, >>>> Bernard >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> Best Regards, >>>> Ramindu. >>>> -- >>>> Ramindu De Silva >>>> Senior Software Engineer >>>> WSO2 Inc.: http://wso2.com >>>> lean.enterprise.middleware >>>> >>>> email: ramin...@wso2.com >>>> mob: +94 719678895 >>>> >>>> >>>> _______________________________________________ >>>> Dev mailing list >>>> Dev@wso2.org >>>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>>> >>>> >>>> >>> _______________________________________________ >>> Dev mailing list >>> Dev@wso2.org >>> http://wso2.org/cgi-bin/mailman/listinfo/dev >>> >> <PastedGraphic-4.png><PastedGraphic-4.png> >> >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> >> >> _______________________________________________ >> Dev mailing list >> Dev@wso2.org >> http://wso2.org/cgi-bin/mailman/listinfo/dev >> > >
_______________________________________________ Dev mailing list Dev@wso2.org http://wso2.org/cgi-bin/mailman/listinfo/dev
