Hi Kim
1) How do you specify a regex to download files with todays date that will
run in synapse unattended for weeks ?
Yes I can hard code an expression to match 31 March 2009 and after but that
is no good today, is it.
Java has the concept of Date NOW; this is what I mean by today.
You say this is "Quite easy" please show me how, as I cannot see how this
can be done ? If it is easy then my problem is solved. Great, but this is my
question ?
2) If an SFTP server has 2000 old files permanently on it now that will not
be removed; thats grows every day. How to I download just the new files ?
E.G Those placed on there with todays date in the name. I have no permission
on the partners SFTP server to MOVE files or DELETE them, they just stay
there. It is not a good assumption that files can be MOVED or DELETED on
another organisations SFTP server. Very rarely in real B2B do you get this
luxury.
The real world cases for which the transport has been originally written
is to poll for a file or files at some specific location that matches
some pattern, process them, and either 1) delete or 2) move them since
the transport does not have state to remember what has been polled already.
The commercial Aqualogic service bus follows this pattern too [1], while
both Mule and Camel additionally supports the specification of date info
in the file pattern. I couldn't readily find much about Fuse or JBoss etc.
The solution I gave will not work, if you cannot move or delete
processed files on the target sftp server, to prevent re-processing them
later. However, supporting of date information in the pattern would be
fairly easy to implement.
3) You suggest SYnapse has a concept of new files ? How does this work ?
This is if you are creating a new file.. e.g. saving a message received
via http/s, JMS, email etc as a file - maybe to an sftp location like in [2]
4) Are you saying that when I start Synapse with an SFTP mediator it
remembers every file downloaded ever in the past; so that when we re-start
Synapse (after maintenence) it remembers the old files. If this is true this
is great. I am interested to know where this record of files is kept. Where
is the file ?
5) Otherwise I am missing some very interesting behaviour of the Synapse VFS
Mediator that does not seem to be provided by the samples or the
documentation or any prior Forum discussions.
No, the transport is implemented as whats called an "Axis2 transport" ..
its actually out of the Synapse core, and not directly dependent or
linked. Implementing a record of processed files - even if possible
would not be worth it - I think the use of Date in the regex is the most
simple way to handle this
I am quite happy to raise a Jira but there is no need if Synapse can do
this, easily, as you say ?
Well.. the point of a JIRA is that all good ideas are captured by those
who propose them, so that new developers or the current developers -
when they have free time, can look at real world user suggestions for
improvement and implement them for future versions
I do not understand this comment about being in a hurry; we are evaluating
Synapse to solve a commercial problem and unfortunately we have encountered
many issues that we are trying to resolve.
I interpreted your multiple emails in the past about dropping or almost
dropping the use of Synapse to mean that you are in a hurry to get this
implemented in Synapse, and thats why I even suggested a pure Java
implementation. Of course if you are still in evaluation, the best is to
try to select another few ESBs (open source and/or commercial) and try
to implement the same, and consider the total cost and the complexity of
the final solution. In this case, consider the effort to implement whats
missing in Synapse into the total cost of the solution.
However, from some of your emails, I see production requirements being
talked about, and thats why sometime back I suggested getting commercial
support - in which case, someone else would have taken care of all of
these issues - implementing what was missing as required. To be honest,
you could have got a complete solution for your real world problem, with
commercial support and a guarantee, probably for much less than the
number of hours you've already spent and would spend * a reasonable
hourly rate for the cost of your time, which again is money.
Unfortunately when I think all the hurdles are solved; then we stumble over new
ones.
Yes unfortunately real world projects have definite requirements and budgets
and deadlines.
I do not see the point of open source software if it cannot solve commercial
problems; unless Synapse is just an Academic Research project ? I am sure
WS02 are using Synapse to solve real world commercial problems and not just
for research. I am sure they make money from using the code.
Of course, many real world users use Synapse - but some are shy to use
the public forums, and risk exposing confidential information, security
and passwords and sensitive data etc. I usually privately inform users
whenever this happens.. since logs could reveal a lot of information
which users sometimes are not aware of; and I remember I informed you
too when you posted a Base64 encrypted password of a live server.
Synapse is not an academic project at all.. it never was. Multiple large
health care providers in the US, the US Army, Governments in Asia and
Europe, mobile resource management companies, one of the largest auto
makers (who is still doing good in this economy), and SOA appliance
vendor that embeds Synapse, European / Asian banks and a large media
company across the US and Europe all use it. These are some of the users
I know about.. and WSO2 would have more users too, that I am not aware of
Here are some "public" comments by a few users in the past:
"Thanks for the quick fix. Synapse continues to rock."
<http://markmail.org/message/qbk4wvncjvtvat2s>
"Synapse continues to impress. Looking forward to 1.1 features upon its
release" <http://markmail.org/message/zwyrzleaop25bqad>
"Thanks everyone for taking time to look into this -- you guys are great
(know wonder Synapse is such a great product!)"
<http://markmail.org/message/5lldrfesfb2wjv44>
"I have been using the excellent Apache Synapse lightweight Enterprise
Service Bus over the last year or so, including on a live, production
project."
<http://thejavamonkey.blogspot.com/2008/09/apache-synapse-how-to-run-without.html>
"We are using Apache Synapse version 1.1 to develop webservices for our
mission critical "Surface Deployment and Distribution command" of U.S
Army" <http://markmail.org/message/b67h6ggfh7vspt2u>
"I do not see the point of open source software if it cannot solve commercial
problems;"
I took this line out of the previous block, so that I can address it
separately.. As far as I am aware, Open source software is not about
building free software by volunteers to commercial companies who makes
money out of it. For users its more about having lifetime access to the
source code of the products, and knowing that licensing is free when you
are ready to drop paying for support, so that you do not lock into
proprietary vendor products, or with vendors who may not survive long
term. When the code is available freely, you can compile and support it
yourself, or get someone else to support it for you. In Apache, its
about a community of people who are interested in building something..
In Apache we even consider 'Community over code', and enjoy working with
diverse people and learning from them for us to improve. But Apache does
not pay us! we are free to sell support or consulting to make sure we
also survive :-)
Open source developers who volunteer their time, do not owe anything to
the users - besides making sure the source code is available in full.
Anyone who users the software is free to fix any issues, and if they
like to contribute them back to the community.
Unfortunately not all Synapse users are expert enough to contribute yet; so
once again I mention the Catch 22 of your comments. It takes time for new
people to get up to speed so they can contribute.
So I spent about 4 hours researching this today before posting; and I
believe this is a contribution. We have real world requirements that would
improve Synapse; this is a
contribution.
Asking questions that do not seem to be covered by documentation; so that
others can benefit is a contribution. Thats all I can offer at the moment.
I am fully aware that all first time users would not be using Synapse
like they were pro's. From my past experience at WSO2, and at present,
many users ask for help on a POC and/or training to get them started.
They then develop an initial prototype project themselves and/or move
into production with development support or professional services.
Thereafter they use dev support to ask questions and learn more advanced
stuff.
BTW, if I couldn't swim, I am aware there are many sites on the Internet
that hosts loads of documentation, and even YouTube videos on how to
swim all styles - free like open source software :-) . Now I may
evaluate these techniques to see if they work in a pool of 5' depth..
but not in the open sea, not without knowing someone is there to help me
if I get in trouble. I would rather pay a professional trainer to train
me to swim, and then try it out more stuff when I am comfortable.. and
of course blog about it later so that others would learn something from
me too. When I can swim and know I can stay afloat, the YouTube videos
would allow me to learn more cool stuff.. and maybe I could invent
something new and contribute it back so that others could follow it too.
I will not be like Adinnapubbaka in the story Mattakundali [3] and try
to learn and do what I cannot.
Unfortunately we have made an extensive investment in Synapse, in time, and
at this point I am trying not to have to drop the use of this product and
trying my hardest to demonstrate that this product can solve just one single
real world problem for us. Am I in a hurry, yes, we have passed the deadline.
So does this mean that you are trying to install Synapse into
production?.. I would not invest much time or effort in an evaluation,
but consider more options in parallel - maybe with some dev support to
help me, to reduce my overall risk.
cheers
asankha
[1] http://edocs.bea.com/alsb/docs261/sftptransport/intro.html
[2] http://synapse.apache.org/Synapse_Samples.html#Sample254
[3]
http://books.google.lk/books?id=sTclW3OWyw8C&pg=PA37&dq=%22mattakundali+(Flat-Earring)%22
--
Asankha C. Perera
AdroitLogic, http://adroitlogic.org
http://esbmagic.blogspot.com
