Hi Sébastien, Here's the command line:
lsc -f /local/adm/ssd/cwlssd/lsc-2.0.2/etc -s all where /etc contains lsc.xml, logback.xml, sql-map-config.xml, sql-map-config.d, etc... I'm attaching the lsc.xml file as well. Thank you for your help. cheers, Ven ________________________________________ From: Sébastien Bahloul [[email protected]] Sent: Saturday, August 17, 2013 1:25 AM To: Mahadevan, Venkat Cc: lsc-users Subject: Re: [lsc-users] Fatal LSC error when syncing large database to LDAP Hi Venkat, Can you provide the complete option and command line and also your configuration file (lsc.xml) ? You shouldn't reach such limit. Regards, Le 17 août 2013 09:49, "Mahadevan, Venkat" <[email protected]<mailto:[email protected]>> a écrit : Hello everyone, I am testing LSC 2.0.2 under Redhat Enterprise Linux 5.7 and JDK 1.7 and also JDK 1.6 (both from Oracle, 64-bit). I am syncing an Oracle database with about 600,000 entries to LDAP. What happens is that the job always terminates with an out of memory exception as follows: Exception in thread "main" java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method) at java.lang.Thread.start(Thread.java:691) at java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:943) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1336) at org.lsc.SynchronizeThreadPoolExecutor.runTask(SynchronizeThreadPoolExecutor.java:44) at org.lsc.AbstractSynchronize.synchronize2Ldap(AbstractSynchronize.java:333) at org.lsc.SimpleSynchronize.launchTask(SimpleSynchronize.java:220) at org.lsc.SimpleSynchronize.launch(SimpleSynchronize.java:157) at org.lsc.Launcher.run(Launcher.java:224) at org.lsc.Launcher.launch(Launcher.java:159) at org.lsc.Launcher.main(Launcher.java:142) I attempted to monitor the lsc process while it was running and noticed the following: watch -n 5 -d "ps -eL <pid> | wc -l" The number of threads being spawned increased all the way from 108 to around 30,000 before the lsc process crashed and threw the exception above. Increasing the memory to the JVM via -Xmx had no helpful effect. Is there some way to tune the maximum number of threads that LSC uses or is this a known issue with large synchronizations? Thanks in advance for any advice. Kind regards, Ven _______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected]<mailto:[email protected]> http://lists.lsc-project.org/listinfo/lsc-users
<?xml version="1.0" ?> <lsc xmlns="http://lsc-project.org/XSD/lsc-core-2.0.xsd"; revision="0"> <connections> <databaseConnection> <name>cwldb-source-conn</name> <url>jdbc:oracle:thin:@cwldb:1521:cwldb</url> <username>test</username> <password>test</password> <driver>oracle.jdbc.OracleDriver</driver> </databaseConnection> <ldapConnection> <name>ldap-dst-conn</name> <url>ldap://ldap:636/dc=devl,dc=id,dc=ubc,dc=ca</url> <username>cn=Directory Manager</username> <password>test</password> <authentication>SIMPLE</authentication> <referral>IGNORE</referral> <derefAliases>NEVER</derefAliases> <version>VERSION_3</version> <pageSize>-1</pageSize> <factory>com.sun.jndi.ldap.LdapCtxFactory</factory> <tlsActivated>true</tlsActivated> </ldapConnection> </connections> <tasks> <task> <name>MySyncTask</name> <bean>org.lsc.beans.SimpleBean</bean> <databaseSourceService> <name>cwldb-source-service</name> <connection reference="cwldb-source-conn" /> <requestNameForList>getInetOrgPersonList</requestNameForList> <requestNameForObject>getInetOrgPerson</requestNameForObject> <requestNameForClean>getInetOrgPersonClean</requestNameForClean> </databaseSourceService> <ldapDestinationService> <name>ldap-dst-service</name> <connection reference="ldap-dst-conn" /> <baseDn>ou=CWL,dc=devl,dc=id,dc=ubc,dc=ca</baseDn> <pivotAttributes> <string>ubceducwlloginkey</string> </pivotAttributes> <fetchedAttributes> <string>sn</string> <string>uid</string> <string>objectClass</string> <string>cn</string> <string>ubcEduCwlPUID</string> <string>givenName</string> <string>ubcEduCwlLoginKey</string> <string>gidNumber</string> <string>uidNumber</string> <string>homeDirectory</string> <string>employeeNumber</string> <string>loginShell</string> <string>userPassword</string> <string>nsAccountLock</string> <string>ubcEduStudentNumber</string> <string>displayName</string> </fetchedAttributes> <getAllFilter>(objectClass=inetorgperson)</getAllFilter> <getOneFilter>(&(objectClass=inetorgperson)(ubceducwlloginkey={ubceducwlloginkey}))</getOneFilter> </ldapDestinationService> <propertiesBasedSyncOptions> <mainIdentifier>"uid=" + srcBean.getDatasetFirstValueById("uid") + ",ou=CWL,dc=devl,dc=id,dc=ubc,dc=ca"</mainIdentifier> <defaultDelimiter>;</defaultDelimiter> <defaultPolicy>FORCE</defaultPolicy> <conditions> <create>true</create> <update>true</update> <delete>true</delete> <changeId>true</changeId> </conditions> <dataset> <name>objectClass</name> <policy>KEEP</policy> <createValues> <string>"posixAccount"</string> <string>"eduPerson"</string> <string>"inetLocalMailRecipient"</string> <string>"krbPrincipalAux"</string> <string>"krbTicketPolicyAux"</string> <string>"ubcEdu"</string> <string>"inetUser"</string> <string>"inetOrgPerson"</string> <string>"organizationalPerson"</string> <string>"person"</string> <string>"top"</string> </createValues> <delimiter>,</delimiter> </dataset> <dataset> <name>cn</name> <policy>FORCE</policy> <forceValues> <string> var first_name = srcBean.getDatasetFirstValueById("givenName"); var last_name = srcBean.getDatasetFirstValueById("sn") if ( first_name == "" ) { first_name = "UNDEFINED"; } if (last_name == "") { last_name = "UNDEFINED"; } var cn_name = first_name + " " + last_name; cn_name </string> </forceValues> <createValues></createValues> </dataset> <dataset> <name>userPassword</name> <policy>FORCE</policy> <forceValues> <string> <![CDATA[js: importPackage(java.lang); importPackage(org.apache.commons.codec.binary); var userPassword = srcBean.getDatasetFirstValueById("userPassword"); var s = userPassword.split(" "); var salt = s[0]; var pass = s[1]; var octetsSalt = Base64.decodeBase64(salt.getBytes()); var saltEnc = Base64.encodeBase64(octetsSalt); var octetsPass = Base64.decodeBase64(pass.getBytes()); var passEnc = Base64.encodeBase64(octetsPass); // var concatbytes = java.lang.reflect.Array.newInstance(java.lang.Character.TYPE, octetsPass.length + octetsSalt.length); var concatbytes = []; concatbytes = octetsPass.concat(octetsSalt); //var occon = java.lang.reflect.Array.newInstance(java.lang.Character.TYPE, concatbytes.length); var occon = Base64.encodeBase64(concatbytes); var finaloutput = "{SMD5}" + new java.lang.String(occon); finaloutput ]]> </string> </forceValues> <createValues></createValues> </dataset> <dataset> <name>gidNumber</name> <policy>KEEP</policy> <forceValues></forceValues> <createValues> <string>"999"</string> </createValues> </dataset> <dataset> <name>uidNumber</name> <policy>KEEP</policy> <forceValues></forceValues> <createValues> <string>"999"</string> </createValues> </dataset> <dataset> <name>homeDirectory</name> <policy>FORCE</policy> <forceValues><string>"/home/" + srcBean.getDatasetFirstValueById("uid")</string></forceValues> <createValues> </createValues> </dataset> <dataset> <name>loginShell</name> <policy>KEEP</policy> <forceValues></forceValues> <createValues> <string>"/bin/bash"</string> </createValues> </dataset> <dataset> <name>nsAccountLock</name> <policy>FORCE</policy> <forceValues> <string> <![CDATA[js: var login_status = srcBean.getDatasetFirstValueById("nsAccountLock"); var bool_status = ""; if (login_status.toLowerCase() == "enabled") { bool_status = "FALSE"; } else { bool_status = "TRUE"; } bool_status ]]> </string> </forceValues> <createValues></createValues> </dataset> <dataset> <name>sn</name> <policy>FORCE</policy> <forceValues> <string> <![CDATA[js: var last_name = srcBean.getDatasetFirstValueById("sn"); if (last_name == "") { last_name = "UNDEFINED"; } last_name ]]> </string> </forceValues> <createValues></createValues> </dataset> </propertiesBasedSyncOptions> </task> </tasks> </lsc>
_______________________________________________________________ Ldap Synchronization Connector (LSC) - http://lsc-project.org lsc-users mailing list [email protected] http://lists.lsc-project.org/listinfo/lsc-users
