Hello Andreas, Thanks for your concern. Yes I will do that, We have our own Jira for the project; and I will email the issues and a summary of the project as soon as we get over this hump. I am happy that we have not had to rule out Synapse totally and a hybrid approach may even be better; not sure ?
I do have a list of Synapse Jira items, growing, that I will add, but I apologise for running out of time at the moment. When we get our current project up and running, I have allocated a week to work on Synapse stuff for you guys. In short and in relation to SFTP only the issues are: 1) we cannot have passwords in clear text in synapse scripts or anywhere. Due to the strict compliance we require; we can be seen as breaking US law doing this; one is the HIPPA Act. There are other compliance and accreditation standards we must comply too; that have very strict security policies. We review projects here so that we don't go to gaol (Jail in US) :-) 2) Ability to pass to VFS, parameters in variables (properties); so being able to download a file based on say a property value containing a name constructed, for example using a script or javainclude date.now(). 3) I do not know if comments below are true but checking the 30 mb file limit; It seems this is a problem with Commons VFS SFTP and not Jsch. I am not sure what combination is used in Synapse VFS mediator. I know this has been addressed and I have tested as OK for VFS files. 4) upgrade to latest Jsch jar. 5) This one is a contentious issue: Roll back a VFS MOVE or DELETE, when the subsequent web service fails. This is like wrapping the SFTP get and WEB service put in a transaction. The issue we have now is that if a web service fails, the SFTP actions have been done as if successful. So the move has to be "compensated" by moving them back to the normal directory. Not sure if WS-Atomic transactions works in Synapse for SFTP; but that's the idea ? Not sure if transactions have been considered in general for Synapse ? I am not sure that this can be done using a failure sequence as just the move, without the download, on the SFTP server, is not possible ? Of these only 1 to 3 impact us now, maybe 4. Re 2: we are not happy that some partners do not allow us to move or delete files, but that's their policy and so what happens is that the files grow on their server and we currently use Date Modified to download latest. They also include the date of file in the file name. So we can use the Jsch lsat property getMTime() or build a file name regex using todays date to download new files. We currently have our system running in Australia and in the UK, we expanded to US and have found that SFTP is the most used B2B mechanism over there. We were Amazed. Basically in Australia Web Services are standard and no one uses SFTP, its dead. So this was the major attraction of Synapse to us. It seems to me the base Use case for SFTP is to download a named file in text. Next is a regex. The next specialisation is to allow that name to be constructed programmatically. Next is to allow a regex to be constructed programmatically. More specialised use cases are the ones addressed by VFS mediator now where you use Move and Delete to get only new files only. For us it is rare to get this permission on other people servers. We do have one partner that is smart enough to do the move automatically, on their side, when we get a file. Other partners either allow us to delete or move files. So we have every combination. Changing how VFS works now does not seem to be as easy, I guess as: - VFS Mediator may not be designed to be updated on the fly. I assume it would be only initialised on start up. The rqmt above requires the file regex to be changed each day, for the new day, or to be totally dynamic. Java regex, does not do date now; but it can be expanded to do this. I have some code for this. - It is not clear how you bootstrap this; as the property has to be set before a VFS proxy starts; hence why I avoided this solution now. This was my first experiment to try and reset the property for the regex after the first time the proxy executed Using: <definitions xmlns="http://ws.apache.org/ns/synapse";> <proxy name="StockQuoteProxy" transports="vfs"> <parameter name="transport.vfs.FileURI">file:///c:/test/in</parameter> <parameter name="transport.vfs.ContentType">text/plain</parameter> <parameter name="transport.vfs.FileNamePattern">.*\.edi</parameter> <parameter name="transport.PollInterval">15</parameter> <inSequence> <log level="full"/> <property name="transport.vfs.FileNamePattern" value=".*\.txt"/> </inSequence> </target> </proxy> </definitions> But this doesn't change the Pattern; only the ".edi" files get done :-) Thanks all the best Kim -----Original Message----- From: Andreas Veithen [mailto:[email protected]] Sent: Wednesday, 8 April 2009 7:13 AM To: [email protected] Subject: Re: Can VFS SFTP be used to download todays files only Kim, Maybe at this point it would be useful if you could give a quick overview of the open issues that you still face. Andreas On Tue, Apr 7, 2009 at 08:55, Kim Horn <[email protected]> wrote: > Created a work around to this by writing an independent service using > just Jsch & Quartz doing an sftp every hour. > From Java we can pass in any expression we like and actually using Jsch > can find the Latest Modified files from the system date. This also > solves the SFTP passwords in clear text issue as now I can encrypt them > in Java. So this service runs totally independent of Synapse. As a bonus > this service can unzip SFTped zipped files we get from some locations > and split large files as Jsch can tell us how big they are before > SFTPing. A prior issue was submitting large files to some web services > that required a new type of iterator in Synapse to be developed. > > So now Synapse just uses VFS to poles the file system, the service > downloads files too, to submit manageable size text files to the > destination Web Services. > > From the other Forums this solution also will resolve a possible issue > with Commons VFS SFTP having problems with large files. Not sure if > these issues are real but dates are fairly recent. > > Start Quotes: > > "This doesn't work with larger files. The VFS seems to have an issue > with the getInputStream(), where it loads the whole file into memory > before returning a memory stream. Eventually the JVM will run out of > memory. In any case, it was a nice start--I'm going to try using Jsch to > work around this problem. > # posted by Blogger Wes : 11:34 PM, November 24, 2008 > > Yes. The code is not working with larger files. It's OK up to 30mb > files. If it exceeds more than 30mb it's causing problems(memory > issues). Any idea how to get rid of this problem. > # posted by Anonymous Anonymous : 3:17 PM, December 16, 2008. > > End Quotes. > > > I have used a more recent release of Jsch than the Jar used in Synapse; > that requires a later version of Java, so would not be possible to do > this as a Synapse class mediator at the moment, although this would be a > better solution. > > Thanks > Kim > > > > -----Original Message----- > From: Kim Horn [mailto:[email protected]] > Sent: Tuesday, 7 April 2009 7:19 AM > To: [email protected] > Subject: RE: Can VFS SFTP be used to download todays files only > > Hello Asankha, > > yes we did get some small simple scripts into production, to help our > manual process, in parralel with evaluation, but had to pull them out as > they did not run with the latest trunk required to solve another issue; > the VFS memory not being released. The fix to this resulted in the "out" > sequences producing exceptions with VFS. > > Thanks > Kim > > > > -----Original Message----- > From: Asankha Perera on behalf of Asankha C. Perera > Sent: Mon 06/04/2009 20:20 > To: [email protected] > Subject: Re: Can VFS SFTP be used to download todays files only > > Hi Kim >> 1) How do you specify a regex to download files with todays date that > will >> run in synapse unattended for weeks ? >> Yes I can hard code an expression to match 31 March 2009 and after but > that >> is no good today, is it. >> Java has the concept of Date NOW; this is what I mean by today. >> >> You say this is "Quite easy" please show me how, as I cannot see how > this >> can be done ? If it is easy then my problem is solved. Great, but this > is my >> question ? >> >> 2) If an SFTP server has 2000 old files permanently on it now that > will not >> be removed; thats grows every day. How to I download just the new > files ? >> E.G Those placed on there with todays date in the name. I have no > permission >> on the partners SFTP server to MOVE files or DELETE them, they just > stay >> there. It is not a good assumption that files can be MOVED or DELETED > on >> another organisations SFTP server. Very rarely in real B2B do you get > this >> luxury. >> > The real world cases for which the transport has been originally written > > is to poll for a file or files at some specific location that matches > some pattern, process them, and either 1) delete or 2) move them since > the transport does not have state to remember what has been polled > already. > > The commercial Aqualogic service bus follows this pattern too [1], while > > both Mule and Camel additionally supports the specification of date info > > in the file pattern. I couldn't readily find much about Fuse or JBoss > etc. > > The solution I gave will not work, if you cannot move or delete > processed files on the target sftp server, to prevent re-processing them > > later. However, supporting of date information in the pattern would be > fairly easy to implement. >> 3) You suggest SYnapse has a concept of new files ? How does this work > ? >> > This is if you are creating a new file.. e.g. saving a message received > via http/s, JMS, email etc as a file - maybe to an sftp location like in > [2] >> 4) Are you saying that when I start Synapse with an SFTP mediator it >> remembers every file downloaded ever in the past; so that when we > re-start >> Synapse (after maintenence) it remembers the old files. If this is > true this >> is great. I am interested to know where this record of files is kept. > Where >> is the file ? >> >> 5) Otherwise I am missing some very interesting behaviour of the > Synapse VFS >> Mediator that does not seem to be provided by the samples or the >> documentation or any prior Forum discussions. >> > No, the transport is implemented as whats called an "Axis2 transport" .. > > its actually out of the Synapse core, and not directly dependent or > linked. Implementing a record of processed files - even if possible > would not be worth it - I think the use of Date in the regex is the most > > simple way to handle this >> I am quite happy to raise a Jira but there is no need if Synapse can > do >> this, easily, as you say ? >> > Well.. the point of a JIRA is that all good ideas are captured by those > who propose them, so that new developers or the current developers - > when they have free time, can look at real world user suggestions for > improvement and implement them for future versions >> I do not understand this comment about being in a hurry; we are > evaluating >> Synapse to solve a commercial problem and unfortunately we have > encountered >> many issues that we are trying to resolve. > I interpreted your multiple emails in the past about dropping or almost > dropping the use of Synapse to mean that you are in a hurry to get this > implemented in Synapse, and thats why I even suggested a pure Java > implementation. Of course if you are still in evaluation, the best is to > > try to select another few ESBs (open source and/or commercial) and try > to implement the same, and consider the total cost and the complexity of > > the final solution. In this case, consider the effort to implement whats > > missing in Synapse into the total cost of the solution. > > However, from some of your emails, I see production requirements being > talked about, and thats why sometime back I suggested getting commercial > > support - in which case, someone else would have taken care of all of > these issues - implementing what was missing as required. To be honest, > you could have got a complete solution for your real world problem, with > > commercial support and a guarantee, probably for much less than the > number of hours you've already spent and would spend * a reasonable > hourly rate for the cost of your time, which again is money. > >> Unfortunately when I think all the hurdles are solved; then we stumble > over new ones. >> Yes unfortunately real world projects have definite requirements and > budgets and deadlines. >> I do not see the point of open source software if it cannot solve > commercial >> problems; unless Synapse is just an Academic Research project ? I am > sure >> WS02 are using Synapse to solve real world commercial problems and not > just >> for research. I am sure they make money from using the code. >> > Of course, many real world users use Synapse - but some are shy to use > the public forums, and risk exposing confidential information, security > and passwords and sensitive data etc. I usually privately inform users > whenever this happens.. since logs could reveal a lot of information > which users sometimes are not aware of; and I remember I informed you > too when you posted a Base64 encrypted password of a live server. > > Synapse is not an academic project at all.. it never was. Multiple large > > health care providers in the US, the US Army, Governments in Asia and > Europe, mobile resource management companies, one of the largest auto > makers (who is still doing good in this economy), and SOA appliance > vendor that embeds Synapse, European / Asian banks and a large media > company across the US and Europe all use it. These are some of the users > > I know about.. and WSO2 would have more users too, that I am not aware > of > > Here are some "public" comments by a few users in the past: > "Thanks for the quick fix. Synapse continues to rock." > <http://markmail.org/message/qbk4wvncjvtvat2s> > "Synapse continues to impress. Looking forward to 1.1 features upon its > release" <http://markmail.org/message/zwyrzleaop25bqad> > "Thanks everyone for taking time to look into this -- you guys are great > > (know wonder Synapse is such a great product!)" > <http://markmail.org/message/5lldrfesfb2wjv44> > "I have been using the excellent Apache Synapse lightweight Enterprise > Service Bus over the last year or so, including on a live, production > project." > <http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-wit > hout.html> > > "We are using Apache Synapse version 1.1 to develop webservices for our > mission critical "Surface Deployment and Distribution command" of U.S > Army" <http://markmail.org/message/b67h6ggfh7vspt2u> > >>"I do not see the point of open source software if it cannot solve > commercial problems;" > > I took this line out of the previous block, so that I can address it > separately.. As far as I am aware, Open source software is not about > building free software by volunteers to commercial companies who makes > money out of it. For users its more about having lifetime access to the > source code of the products, and knowing that licensing is free when you > > are ready to drop paying for support, so that you do not lock into > proprietary vendor products, or with vendors who may not survive long > term. When the code is available freely, you can compile and support it > yourself, or get someone else to support it for you. In Apache, its > about a community of people who are interested in building something.. > In Apache we even consider 'Community over code', and enjoy working with > > diverse people and learning from them for us to improve. But Apache does > > not pay us! we are free to sell support or consulting to make sure we > also survive :-) > > Open source developers who volunteer their time, do not owe anything to > the users - besides making sure the source code is available in full. > Anyone who users the software is free to fix any issues, and if they > like to contribute them back to the community. > >> Unfortunately not all Synapse users are expert enough to contribute > yet; so >> once again I mention the Catch 22 of your comments. It takes time for > new >> people to get up to speed so they can contribute. >> >> So I spent about 4 hours researching this today before posting; and I >> believe this is a contribution. We have real world requirements that > would improve Synapse; this is a >> contribution. >> >> Asking questions that do not seem to be covered by documentation; so > that >> others can benefit is a contribution. Thats all I can offer at the > moment. >> > I am fully aware that all first time users would not be using Synapse > like they were pro's. From my past experience at WSO2, and at present, > many users ask for help on a POC and/or training to get them started. > They then develop an initial prototype project themselves and/or move > into production with development support or professional services. > Thereafter they use dev support to ask questions and learn more advanced > > stuff. > > BTW, if I couldn't swim, I am aware there are many sites on the Internet > > that hosts loads of documentation, and even YouTube videos on how to > swim all styles - free like open source software :-) . Now I may > evaluate these techniques to see if they work in a pool of 5' depth.. > but not in the open sea, not without knowing someone is there to help me > > if I get in trouble. I would rather pay a professional trainer to train > me to swim, and then try it out more stuff when I am comfortable.. and > of course blog about it later so that others would learn something from > me too. When I can swim and know I can stay afloat, the YouTube videos > would allow me to learn more cool stuff.. and maybe I could invent > something new and contribute it back so that others could follow it too. > > I will not be like Adinnapubbaka in the story Mattakundali [3] and try > to learn and do what I cannot. >> Unfortunately we have made an extensive investment in Synapse, in > time, and >> at this point I am trying not to have to drop the use of this product > and >> trying my hardest to demonstrate that this product can solve just one > single >> real world problem for us. Am I in a hurry, yes, we have passed the > deadline. >> > So does this mean that you are trying to install Synapse into > production?.. I would not invest much time or effort in an evaluation, > but consider more options in parallel - maybe with some dev support to > help me, to reduce my overall risk. > > cheers > asankha > > [1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html > [2] http://synapse.apache.org/Synapse_Samples.html#Sample254 > [3] > http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+ > (Flat-Earring)%22 > > -- > Asankha C. Perera > AdroitLogic, http://adroitlogic.org > > http://esbmagic.blogspot.com > > > > > >
