Re: File Transfer conundrum

[Paul Gilmartin]

On Mon, 14 Jan 2008 12:20:37 -0500, Shmuel Metz (Seymour J.) wrote:
>
>What happens if you use struc?
>
IIRC, "STRU R" fails with a z/OS client and a non-IBM server, which
IIRC was the OP's required configuration.

Kobayashi Alternative?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Shmuel Metz (Seymour J.)]

In <[EMAIL PROTECTED]>, on 01/09/2008
   at 10:13 PM, Bruce Baxter <[EMAIL PROTECTED]> said:

>As stated by another poster, there doesn't seem to be any way to 
>get it to honor this and strip it off on the final FTP.

You don't want FTP to strip it off, you want FTP to transmit the data
intact. The problem seems to be that FTP is building its own RDW's instead
of using the data transmitted to it.

What happens if you use struc?
 
-- 
 Shmuel (Seymour J.) Metz, SysProg and JOAT
 ISO position; see  
We don't care. We don't have to care, we're Congress.
(S877: The Shut up and Eat Your spam act of 2003)

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Tony Harminc]

On Fri, 11 Jan 2008 09:45:10 -0500, Anne & Lynn Wheeler <[EMAIL PROTECTED]> 
wrote:

>http://www.swift.com/
>
>"swift-2" providing internet capability and opening up for b-to-b;

There is a good introduction to how SWIFT works in Ross Anderson's book
Security Engineering. The book is now available online, and SWIFT is in
section 9.3.1 of http://www.cl.cam.ac.uk/~rja14/Papers/SE-09.pdf .

I highly recommend this book, and there's a new edition coming out in a
couple of months.

Tony H.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Gary Green]

Interesting.

Thanks.

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Anne & Lynn Wheeler
Sent: Friday, January 11, 2008 9:45 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

The following message is a courtesy copy of an article that has been posted
to bit.listserv.ibm-main as well.

[EMAIL PROTECTED] (Gary Green) writes:
> I lost track of who posted the original inquiry, so take this for what
> it's worth.
>
> If the requirement is in the financial industry, could the
> communications between the two/various systems use S.W.I.F.T. (Society
> for Worldwide Interbank Financial Telecommunications)?  It's been some
> time since I wrote anything for SWIFT, but it was extremely secure and
> most financial institutions should be linked in.
>
> When I did some work, it was used in the securities market, primarily
> for payments, foreign exchange, securities, etc...  However, there
> were rumblings that the SWIFT organization was thinking about opening
> up the network for other financial "transactions"; which I took to
> mean data exchange...

home page
http://www.swift.com/

"swift-2" providing internet capability and opening up for b-to-b;

we had been brought in to consult with small client/server startup that
wanted to do payment transactions on their server; they also had this
technology they called SSL they wanted to use ... and result is sometimes
now called e-commerce. part of the effort was something called payment
gateway (transition between internet and acquiring networks)
http://www.garlic.com/~lynn/subnetwork.html#gateway

we then were involved in x9a10 financial standard working group (in the
mid-90s had been given the requirement to preserve the integrity of the
financial infrastructure for all retail payments) that resulted in the x9.59
financial standard
http://www.garlic.com/~lynn/x959.html#x959

some years ago we were also asked to provide some input to the swift-2 (what
it was called at the time) specification.

Connecting to the secure IP network (SIPN)
http://www.swift.com/index.cfm?item_id=2304

from above:

SWIFTNet messaging services are provided via SWIFT's secure IP network
(SIPN), a highly secure and reliable network. Full redundancy, advanced
recovery mechanisms and first class operations and customer support services
ensure continuous network availability for SWIFTNet services.

... snip ..

SWIFTNet Interfaces Qualification
http://www.swift.com/index.cfm?item_id=2451

other refs:

Securities Markets Infrastructures
http://www.swift.com/index.cfm?item_id=2437

Banking Markets infrastructures
http://www.swift.com/index.cfm?item_id=57981

from above:

Additionally, SWIFT is now complementing its position in the wholesale, high
value clearing market by extending its portfolio of SWIFTNet messaging
solutions to the low-value payments and ACH market.

... snip ...

--
For IBM-MAIN subscribe / signoff / archive access instructions, send email
to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO Search the
archives at http://bama.ua.edu/archives/ibm-main.html


--
No virus found in this incoming message.
Checked by AVG Free Edition.
Version: 7.5.516 / Virus Database: 269.19.0/1218 - Release Date: 1/10/2008
1:32 PM


  <http://e-mail-servers.com/98799038b867f835c5577e917ab264d6worker.jpg> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Anne & Lynn Wheeler]

The following message is a courtesy copy of an article
that has been posted to bit.listserv.ibm-main as well.

[EMAIL PROTECTED] (Gary Green) writes:
> I lost track of who posted the original inquiry, so take this for what it's
> worth.
>
> If the requirement is in the financial industry, could the communications
> between the two/various systems use S.W.I.F.T. (Society for Worldwide
> Interbank Financial Telecommunications)?  It's been some time since I wrote
> anything for SWIFT, but it was extremely secure and most financial
> institutions should be linked in.
>
> When I did some work, it was used in the securities market, primarily for
> payments, foreign exchange, securities, etc...  However, there were
> rumblings that the SWIFT organization was thinking about opening up the
> network for other financial "transactions"; which I took to mean data
> exchange...

home page
http://www.swift.com/

"swift-2" providing internet capability and opening up for b-to-b;

we had been brought in to consult with small client/server startup that
wanted to do payment transactions on their server; they also had this
technology they called SSL they wanted to use ... and result is
sometimes now called e-commerce. part of the effort was something called
payment gateway (transition between internet and acquiring networks)
http://www.garlic.com/~lynn/subnetwork.html#gateway

we then were involved in x9a10 financial standard working group (in the
mid-90s had been given the requirement to preserve the integrity of the
financial infrastructure for all retail payments) that resulted
in the x9.59 financial standard
http://www.garlic.com/~lynn/x959.html#x959

some years ago we were also asked to provide some input to the swift-2
(what it was called at the time) specification.

Connecting to the secure IP network (SIPN)
http://www.swift.com/index.cfm?item_id=2304

from above:

SWIFTNet messaging services are provided via SWIFT's secure IP network
(SIPN), a highly secure and reliable network. Full redundancy, advanced
recovery mechanisms and first class operations and customer support
services ensure continuous network availability for SWIFTNet services.

... snip ..

SWIFTNet Interfaces Qualification
http://www.swift.com/index.cfm?item_id=2451

other refs:

Securities Markets Infrastructures
http://www.swift.com/index.cfm?item_id=2437

Banking Markets infrastructures
http://www.swift.com/index.cfm?item_id=57981

from above:

Additionally, SWIFT is now complementing its position in the wholesale,
high value clearing market by extending its portfolio of SWIFTNet
messaging solutions to the low-value payments and ACH market.

... snip ...

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Gary Green]

I do not remember, if I ever knew, the SWIFT network backbone.  All I ever
had to deal with was writing the communications to send packets of
information to custodian banks.  About the time I started working on the
project, SWIFT was just getting started as, I believe, a rebadged version of
something else.  The name of which is totally lost in the recesses of my
mind.

My system was a combination of OS/2 and Windows (3.1 no less).  Because it
was trade (read mutual fund trade) based we could process over 65,000
transactions a day.  Of course the information packets were small, sometime
less than 256 bytes.  It sounds like your system was needed for much larger
file transfers which poses a different set of problems...

Not to start another stroll down the OS/2 lane, I remember someone asking me
why I did not write everything for Windows.  I tried to explain to him that
OS/2 would wrap the PID number and still keep on running.  I tried it with
Windows and after about 1,000 transactions it would crash because of memory
leaks or any host of other reasons.

Ah... those were the days... :)

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Ed Gould
Sent: Friday, January 11, 2008 1:27 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

I don't recall too much about SWIFT. Just that it existed that is about all.
Was it connected with TANDEM (et al) ?
In anycase we had only two customers that didn't do SNA and I think one of
them was using RSCS (bisync I believe) and some RJE emulator oh yes JES3 I
had almost forgot. Bisync was pretty good (but not as good as SNA, IMO).
There were 3rd party options of course but that entailed a healthy invest in
software. The company I worked for was the cheapest of the cheap so any 50K
(or more) investment would have been looked on with extreme displeasure. We
did look at one or two but there were issue that we just could not live
with. One was that software (I am sorry  I can't remember the vendor name)
was a little (read a lot) OS dependent. I had talked to a user about
incompatibilities between the vendors software level. Trying to get everyone
synced up was a nightmare according to him (especially when it came to 200+)
. We wanted to make this as transparent as possible and we exceeded far
beyond my original estimates. The IBM utility we used was sold (it *NEVER
abended* because of program logic and we typically used it 10K times a day
(or more) the number of times a day gets a little murky as we had two data
centers and we transmitted to our DR site many times more files that 10K. I
can't say enough about a solid product like this was. Once installed it
hasn't been touched in 20 years over many OS releases and zero problems,
show me any equal numbers from any OEM and we can talk.

I am no longer with that company but I was pretty proud as to what I
implemented in such a short amount of time.

Ed


  <http://e-mail-servers.com/1c7cfcab4f28fab40bc238c9aee06e71worker.jpg> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Gary Green]

Okay Bruce...  I forgot that you were the poster.  I believe another forum
member mentioned this...

Would XMIT/RECEIVE work?  I ask because at another location/client there was
a need to keep backup copies of numerous mainframe files off of the
mainframe.  So on a periodic basis, a mainframe job would kick off, "unload"
the files to XMIT format and then FTP that file down to a PC/workstation
attached to the host network.  On that workstation I had a VB script that
would monitor the incoming directory and whenever it found a new file it
would then either email or FTP the file to another location. (email or FTP
was destination specific).  Once the file was "sent", it would be moved to
another directory to get it out of the way...  Worked great!

During testing I had a "receive" job defined to the job scheduler so
whenever a specific file was created, the receive job would be kicked off
using the file name as a JCL parameter.  This file would then be "received"
back into the original format.  To test the process I would drag-n-drop a
file into a different PC/workstation directory (separate from the one
mentioned above).  Another VB script would see the new file and then FTP
that file to the host.  Once there, the job would be kicked off and the file
received.

Again, JMTC

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Bruce Baxter
Sent: Thursday, January 10, 2008 10:53 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

I'm the original poster, and yours is an intersting thought.  There's no
financial partners involved that I know of, and S.W.I.F.T. would not really
be applicable.

Some of the other posters mentioned security issues, and they're not really
a concern here since the transfers that occur on public networks are done
securely and encrypted.  At issue, I think are the data conversions at
either end, which, with FTP require knowledge of the Code Pages and
converstions that have taken place.


On Thu, 10 Jan 2008 21:10:04 -0500, Gary Green <[EMAIL PROTECTED] SYSTEMS.COM>
wrote:

>I lost track of who posted the original inquiry, so take this for what
>it's worth.
>
>If the requirement is in the financial industry, could the
>communications between the two/various systems use S.W.I.F.T. (Society
>for Worldwide Interbank Financial Telecommunications)?  It's been some
>time since I wrote anything for SWIFT, but it was extremely secure and
>most financial institutions should be linked in.
>
>When I did some work, it was used in the securities market, primarily
>for payments, foreign exchange, securities, etc...  However, there were
>rumblings that the SWIFT organization was thinking about opening up the
>network for other financial "transactions"; which I took to mean data
>exchange...
>
>JMTC
>
>-Original Message-
>From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
Behalf
>Of Ed Gould
>Sent: Thursday, January 10, 2008 8:52 PM
>To: IBM-MAIN@BAMA.UA.EDU
>Subject: Re: File Transfer conundrum
>
>On Jan 10, 2008, at 11:16 AM, Hal Merritt wrote:
>
>> My guess is that many shops are implementing PC to PC transfers and
>> buying some really expensive software to facilitate the process.
>> That is
>> host>pc>network>pc>host.
>>
>> So far, we have been somewhat successful in insisting on a fully
>> automated, z/os based solution on our side. Our most compelling
>> argument is that we are not supposed to move sensitive data in the
>> open over any network, nor is the data supposed to reside in the open
>> on any server, even in a buffer. As much as the PC folks don't like
>> to admit it, they simply cannot meet that requirement.  They can come
>> close, but there seems to always be a point where the data can be
>> intercepted.
>>
>>
>
>Hal,
>
>Even though my bosses boss was big on PC's he also knew that PC file
>transfer was well lets say less than perfect. He was the person who
>pushed SNA and made the budget available to convert to SNA and the
>hardware associated with it. He lobbied for the money because he knew
>that pc file transfer in the financial community was less than good.
>We in the financial community put data integrity on a pedestal that
>tested everything every step of the way. A long time ago they had an
>OEM (name
>withheld) vendor and there was no data integrity checking all they
>cared about was being cheap. We were racked over the coals all the time
>as we
were
>essentially sending out good data but the data along the line was being
>corrupted somewhere along the line.
>Usually it was a some important field that needed to have to have the
>entire file retransmitted. more than a few t

Re: File Transfer conundrum

[Paul Gilmartin]

On Thu, 10 Jan 2008 15:54:23 -0700, Roger Bolan wrote:

>I agree with John.  TERSE is the simplest.   You just terse the file on
>the source z/OS system, then use binary for all transfers.   Then there is
>no problem of EBCDIC/ASCII, or RDW.  The binary file arrives unchanged.
>Then use TERSE to unpack it and it automatically restores the correct DCB
>attributes.   The only thing I have ever had to remember to do was specify
>directory blocks when untersing a PDS and no directory blocks for a
>sequential file.
>
>If the file gets to an intermediate server where you can use FTP from your
>target z/OS system to get it directly, then you can do this:
>
>BINARY  < don't forget this.  ASCII mode by default does not work for
>traces>
>LOCSITE recfm=fb lrecl=1024 cylinders primary=100 secondary=10 blksize=0
> 
>mget Px.* ( replace 
>quit< to get out of FTP>
>
How much of this stuff couldn't be automated with a well-designed tool?

(see:

"z/OS V1R7.0 DFSMSdfp Utilities"
B.0   Appendix B.  Unload Partitioned Data Set Format

... )

The header records of the unload data set contain most (all?) of this
information.  I've experimented with doing this in a Rexx program.
IIRC, the only way I could infer the directory block count was to
read the directory block records and count them; the only way I could
infer space was to make a guess based on the size of the unload
data set.

But SMP/E does this well for its TLIBs; the packager doesn't need to
specify any information to IEBCOPY -- it gets it all from the DSCB,
and the installer specifies only DSPREFIX to SMP/E -- it dynamically
allocates based on the information in the unload data set (but does
it need two passes to do this?)  Couldn't TERSE do as well if the
developers chose to do so?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Ed Gould]

On Jan 10, 2008, at 8:10 PM, Gary Green wrote:

I lost track of who posted the original inquiry, so take this for  
what it's

worth.

If the requirement is in the financial industry, could the  
communications

between the two/various systems use S.W.I.F.T. (Society for Worldwide
Interbank Financial Telecommunications)?  It's been some time since  
I wrote

anything for SWIFT, but it was extremely secure and most financial
institutions should be linked in.

When I did some work, it was used in the securities market,  
primarily for

payments, foreign exchange, securities, etc...  However, there were
rumblings that the SWIFT organization was thinking about opening up  
the

network for other financial "transactions"; which I took to mean data
exchange...

JMTC
---SNIP--


I don't recall too much about SWIFT. Just that it existed that is  
about all. Was it connected with TANDEM (et al) ?
In anycase we had only two customers that didn't do SNA and I think  
one of them was using RSCS (bisync I believe) and some RJE emulator  
oh yes JES3 I had almost forgot. Bisync was pretty good (but not as  
good as SNA, IMO). There were 3rd party options of course but that  
entailed a healthy invest in software. The company I worked for was  
the cheapest of the cheap so any 50K (or more) investment would have  
been looked on with extreme displeasure. We did look at one or two  
but there were issue that we just could not live with. One was that  
software (I am sorry  I can't remember the vendor name) was a little  
(read a lot) OS dependent. I had talked to a user about  
incompatibilities between the vendors software level. Trying to get  
everyone synced up was a nightmare according to him (especially when  
it came to 200+) . We wanted to make this as transparent as possible  
and we exceeded far beyond my original estimates. The IBM utility we  
used was sold (it *NEVER abended* because of program logic and we  
typically used it 10K times a day (or more) the number of times a day  
gets a little murky as we had two data centers and we transmitted to  
our DR site many times more files that 10K. I can't say enough about  
a solid product like this was. Once installed it hasn't been touched  
in 20 years over many OS releases and zero problems, show me any  
equal numbers from any OEM and we can talk.


I am no longer with that company but I was pretty proud as to what I  
implemented in such a short amount of time.


Ed
 


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Bruce Baxter]

I'm the original poster, and yours is an intersting thought.  There's no 
financial 
partners involved that I know of, and S.W.I.F.T. would not really be applicable.

Some of the other posters mentioned security issues, and they're not really a 
concern here since the transfers that occur on public networks are done 
securely and encrypted.  At issue, I think are the data conversions at either 
end, which, with FTP require knowledge of the Code Pages and converstions 
that have taken place.


On Thu, 10 Jan 2008 21:10:04 -0500, Gary Green <[EMAIL PROTECTED]
SYSTEMS.COM> wrote:

>I lost track of who posted the original inquiry, so take this for what it's
>worth.
>
>If the requirement is in the financial industry, could the communications
>between the two/various systems use S.W.I.F.T. (Society for Worldwide
>Interbank Financial Telecommunications)?  It's been some time since I wrote
>anything for SWIFT, but it was extremely secure and most financial
>institutions should be linked in.
>
>When I did some work, it was used in the securities market, primarily for
>payments, foreign exchange, securities, etc...  However, there were
>rumblings that the SWIFT organization was thinking about opening up the
>network for other financial "transactions"; which I took to mean data
>exchange...
>
>JMTC
>
>-Original Message-
>From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On 
Behalf
>Of Ed Gould
>Sent: Thursday, January 10, 2008 8:52 PM
>To: IBM-MAIN@BAMA.UA.EDU
>Subject: Re: File Transfer conundrum
>
>On Jan 10, 2008, at 11:16 AM, Hal Merritt wrote:
>
>> My guess is that many shops are implementing PC to PC transfers and
>> buying some really expensive software to facilitate the process.
>> That is
>> host>pc>network>pc>host.
>>
>> So far, we have been somewhat successful in insisting on a fully
>> automated, z/os based solution on our side. Our most compelling
>> argument is that we are not supposed to move sensitive data in the
>> open over any network, nor is the data supposed to reside in the open
>> on any server, even in a buffer. As much as the PC folks don't like to
>> admit it, they simply cannot meet that requirement.  They can come
>> close, but there seems to always be a point where the data can be
>> intercepted.
>>
>>
>
>Hal,
>
>Even though my bosses boss was big on PC's he also knew that PC file
>transfer was well lets say less than perfect. He was the person who pushed
>SNA and made the budget available to convert to SNA and the hardware
>associated with it. He lobbied for the money because he knew that pc file
>transfer in the financial community was less than good.
>We in the financial community put data integrity on a pedestal that tested
>everything every step of the way. A long time ago they had an OEM (name
>withheld) vendor and there was no data integrity checking all they cared
>about was being cheap. We were racked over the coals all the time as we 
were
>essentially sending out good data but the data along the line was being
>corrupted somewhere along the line.
>Usually it was a some important field that needed to have to have the entire
>file retransmitted. more than a few times a broker could not trade any X
>because data from our company was less than lets say accurate due to
>transmission and or equipment. Our side sent it out OK but somewhere along
>the line after it left something got corrupted. After we ditched the OEM
>vendor and we went to SNA we never had one piece of data go bad. We had
>cases where there was a tape issue on the receiving (data check or broken
>tape etc) that required a retransmission but not one in several million file
>transfers a day every was caused by "us". PC file transfer sucks PERIOD. I
>download on my MAC several hundred files a day and at least once a day 
some
>file is corrupted.
>
>Ed
>
>
>  <http://e-mail-
servers.com/ee2878cf37f593a6c827c73598cee1edworker.jpg>
>
>--
>For IBM-MAIN subscribe / signoff / archive access instructions,
>send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
>Search the archives at http://bama.ua.edu/archives/ibm-main.html

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Gary Green]

I lost track of who posted the original inquiry, so take this for what it's
worth.

If the requirement is in the financial industry, could the communications
between the two/various systems use S.W.I.F.T. (Society for Worldwide
Interbank Financial Telecommunications)?  It's been some time since I wrote
anything for SWIFT, but it was extremely secure and most financial
institutions should be linked in.

When I did some work, it was used in the securities market, primarily for
payments, foreign exchange, securities, etc...  However, there were
rumblings that the SWIFT organization was thinking about opening up the
network for other financial "transactions"; which I took to mean data
exchange...

JMTC

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On Behalf
Of Ed Gould
Sent: Thursday, January 10, 2008 8:52 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

On Jan 10, 2008, at 11:16 AM, Hal Merritt wrote:

> My guess is that many shops are implementing PC to PC transfers and
> buying some really expensive software to facilitate the process.
> That is
> host>pc>network>pc>host.
>
> So far, we have been somewhat successful in insisting on a fully
> automated, z/os based solution on our side. Our most compelling
> argument is that we are not supposed to move sensitive data in the
> open over any network, nor is the data supposed to reside in the open
> on any server, even in a buffer. As much as the PC folks don't like to
> admit it, they simply cannot meet that requirement.  They can come
> close, but there seems to always be a point where the data can be
> intercepted.
>
>

Hal,

Even though my bosses boss was big on PC's he also knew that PC file
transfer was well lets say less than perfect. He was the person who pushed
SNA and made the budget available to convert to SNA and the hardware
associated with it. He lobbied for the money because he knew that pc file
transfer in the financial community was less than good. 
We in the financial community put data integrity on a pedestal that tested
everything every step of the way. A long time ago they had an OEM (name
withheld) vendor and there was no data integrity checking all they cared
about was being cheap. We were racked over the coals all the time as we were
essentially sending out good data but the data along the line was being
corrupted somewhere along the line. 
Usually it was a some important field that needed to have to have the entire
file retransmitted. more than a few times a broker could not trade any X
because data from our company was less than lets say accurate due to
transmission and or equipment. Our side sent it out OK but somewhere along
the line after it left something got corrupted. After we ditched the OEM
vendor and we went to SNA we never had one piece of data go bad. We had
cases where there was a tape issue on the receiving (data check or broken
tape etc) that required a retransmission but not one in several million file
transfers a day every was caused by "us". PC file transfer sucks PERIOD. I
download on my MAC several hundred files a day and at least once a day some
file is corrupted.

Ed


  <http://e-mail-servers.com/ee2878cf37f593a6c827c73598cee1edworker.jpg> 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Ed Gould]

On Jan 10, 2008, at 11:16 AM, Hal Merritt wrote:


My guess is that many shops are implementing PC to PC transfers and
buying some really expensive software to facilitate the process.  
That is

host>pc>network>pc>host.

So far, we have been somewhat successful in insisting on a fully
automated, z/os based solution on our side. Our most compelling  
argument
is that we are not supposed to move sensitive data in the open over  
any

network, nor is the data supposed to reside in the open on any server,
even in a buffer. As much as the PC folks don't like to admit it, they
simply cannot meet that requirement.  They can come close, but there
seems to always be a point where the data can be intercepted.




Hal,

Even though my bosses boss was big on PC's he also knew that PC file  
transfer was well lets say less than perfect. He was the person who  
pushed SNA and made the budget available to convert to SNA and the  
hardware associated with it. He lobbied for the money because he knew  
that pc file transfer in the financial community was less than good.  
We in the financial community put data integrity on a pedestal that  
tested everything every step of the way. A long time ago they had an  
OEM (name withheld) vendor and there was no data integrity checking  
all they cared about was being cheap. We were racked over the coals  
all the time as we were essentially sending out good data but the  
data along the line was being corrupted somewhere along the line.  
Usually it was a some important field that needed to have to have the  
entire file retransmitted. more than a few times a broker could not  
trade any X because data from our company was less than lets say  
accurate due to transmission and or equipment. Our side sent it out  
OK but somewhere along the line after it left something got  
corrupted. After we ditched the OEM vendor and we went to SNA we  
never had one piece of data go bad. We had cases where there was a  
tape issue on the receiving (data check or broken tape etc) that  
required a retransmission but not one in several million file  
transfers a day every was caused by "us". PC file transfer sucks  
PERIOD. I download on my MAC several hundred files a day and at least  
once a day some file is corrupted.


Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Tony Harminc]

On Wed, 9 Jan 2008 15:24:32 -0600, Bruce Baxter
<[EMAIL PROTECTED]> wrote:

>We've routinely exhanged files with business partners running on z/OS
>machines using tape for years.
>
>We're now in the process of converting a number of these to electronic
>means, using in part FTP.

[snip - common problems with character codes, record boundaries, etc. etc.]


There are commercial solutions out there that will make all this stuff Just
Work. Naturally I'd suggest the products made by my company, but I'd also
suggest looking more generally, to get an idea of what's available. In
particular, if you are exchanging unencrypted data that used to rely on
physical security of tapes, and it is now flowing through unknown layers in
an organization you don't control, I would be worried. (Well, tapes should
be encrypted too, of course, but that's another story.)

Anyway - there are products that not only make the data flow correctly
across multiple platforms, but also provide control and audit, so management
can see who sent what when, and to whom, and indeed control their ability to
do so. In our case, products written by mainframe-knowledgable people, even. 

FTP (and TERSE and XMIT the like for packaging) are very handy for ad-hoc
transfers (I use them all the time), but for production use, particularly
between business partners, most enterprises find they need something much
more, well, enterprise-scale, with consistent security, as well as the
ability to get the data there with bit-for-bit-identical results.


Tony H.
www.proginet.com

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Roger Bolan]

I agree with John.  TERSE is the simplest.   You just terse the file on 
the source z/OS system, then use binary for all transfers.   Then there is 
no problem of EBCDIC/ASCII, or RDW.  The binary file arrives unchanged. 
Then use TERSE to unpack it and it automatically restores the correct DCB 
attributes.   The only thing I have ever had to remember to do was specify 
directory blocks when untersing a PDS and no directory blocks for a 
sequential file. 

If the file gets to an intermediate server where you can use FTP from your 
target z/OS system to get it directly, then you can do this: 

BINARY  < don't forget this.  ASCII mode by default does not work for 
traces>
LOCSITE recfm=fb lrecl=1024 cylinders primary=100 secondary=10 blksize=0   
  
mget Px.* ( replace 
quit< to get out of FTP>

Roger Bolan
AFP Infoprint Software on z/OS, z/VM, z/VSE
Phone: 303-354-2134, tieline 419-2134, local x62134
Fax: 1-800-=865-9053/1-303-354-2036
e-mail: [EMAIL PROTECTED]
infoprint.com

6300 Diagonal Highway
Bldg 004, G8
Boulder CO, 80301


IBM Mainframe Discussion List  wrote on 01/09/2008 
02:28:55 PM:

> The simpliest thing, in my opinion, is to use TERSE on both ends and do
> binary transfers. 
> 
> --
> John McKown
> Senior Systems Programmer
> HealthMarkets
> Keeping the Promise of Affordable Coverage
> Administrative Services Group
> Information Technology

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Bruce Baxter]

Their other partners also have similar problems.  I'm trying to coalesce a 
group 
of the techies that deal with this issue to have a conference call with techies 
at the shop generating this data.

Bottom line: FTP is clearly not a drop in replacement for good old fashioned 
tapes.

On Thu, 10 Jan 2008 08:18:56 -0600, Zaromil Tisler  wrote:

>On Wed, 9 Jan 2008 22:13:13 -0600, Bruce Baxter <...> wrote:
>
>>ftp directly between the two systems, quite simply isn't an option.  Our
>>business partner chose the mechanism in palce to deal with multiple other
>>shops, and they're not likely to want to do this sort of one-off thing for us.
>
>And how does your partner deliver files to other shops? Do they also get
>corrupted data or do all of them use non-EBCDIC environments?
>
>--
>Zaromil
>
>--
>For IBM-MAIN subscribe / signoff / archive access instructions,
>send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
>Search the archives at http://bama.ua.edu/archives/ibm-main.html

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Bruce Baxter]

An MQ Series solution would not be viable for this application on two counts.

First, I doubt the sender would go for this because it would be too much work 
on their side to maintain connections to all it's clients.

Secondly, MQ has lots of overhead compared to an FTP style process.  All the 
MQMD and other channel related traffic, etc, and in addition, how would you 
ever know if the file was fully sent, without some additional protocol layer on 
top of it?

On Thu, 10 Jan 2008 10:37:38 +, Grant Ward Able 
<[EMAIL PROTECTED]> wrote:

>Has MQSeries been considered? I'd have thought it would have solved most
>problems like this. (unless one of the partners doesnt actually have
>MQ installed!)
>
>--
>Regards - Grant
>
>Grant Ward Able
>Senior Systems Architect
>DTCC
>
>
>
>
>"John S. Giltner, Jr." <[EMAIL PROTECTED]>
>Sent by: IBM Mainframe Discussion List 
>10/01/2008 02:51
>Please respond to
>IBM Mainframe Discussion List 
>
>
>To
>IBM-MAIN@BAMA.UA.EDU
>cc
>
>Subject
>Re: File Transfer conundrum
>
>
>
>
>
>
>Any reason why you can't ftp directly between the two z/OS system?
>
>If security is an issue you could use either IPSec tunnels between the
>two systems or setup IBM SecureFTP server (SSL'ed FTP).
>
>
>
>Bruce Baxter wrote:
>> We've routinely exhanged files with business partners running on z/OS
>> machines using tape for years.
>>
>> We're now in the process of converting a number of these to electronic
>> means, using in part FTP.  This is being done at the behest of one of
>our
>> business partners, who (IMHO) hasn't thought through all the issues that
>the
>> use of FTP introduces in this process.  The central issue as I see it is
>that the
>> mainframes at either end of the pipeline are both EBCDIC and record
>oriented,
>> and the servers and ftp processes that lie between them to facilitate
>these
>> transfers do not have any inherent concept of record oriented files like
>the
>> mainframe.
>>
>> I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED
>data
>> separately.
>>
>> The first files that we received were FIXED BLOCK, and had been
>translated
>> from EBCDIC to ASCII, most likely at the first transfer of the file from
>z/OS to
>> an ASCII based server platform (either Windows or AIX).  When they
>arrived
>> on our z/OS system, we had issues of data corruption because the data
>> contained zoned decimal data.  After some discussion, we agreed that
>we'd
>> transfer these files in BINARY mode at all steps along the way.  Thus,
>all we
>> had to do was ensure that the LRECL used for the destination dataset on
>> z/OS was the same as the source dataset.  This seems to be working OK.
>>
>> Most recently, we've been having problems with other files that are
>VARIABLE
>> BLOCKED.  We received the first of these files last week, transmitted
>from end-
>> to-end in BINARY mode.  What we got was not at all what we expected.
>> We've discovered that the initial FTP from z/OS to the server stripped
>off all
>> information regarding record length and thus record delineation. Because
>
>> there aren't any RDWs in front of every record, ftp doesn't know how
>long the
>> records are and just plunks the data into the destination dataset in
>chunks of
>> LRECL-4.  I did a bunch of research on z/OS FTP and there doesn't appear
>to
>> be any way to make it convey record length/delineation information to
>and
>> ASCII platform other than to use ASCII mode.  z/OS FTP appears to have
>> mechanisms for conveying this between two z/OS FTP systems, but that's
>not
>> possible here.  For the present time, we've had the file shipped with
>the initiatl
>> movement translating the data from EBCDIC to ASCII and all subsequent
>> transfers until the last one back to z/OS in BINARY mode.  However, I'm
>> concerned about the possibility of data corruption if the translate
>tables used
>> in the first step and the last step of this files travel aren't exact
>inversions of
>> each other.  This would certainly be possible of the initial ASCII
>transfer were
>> done to a Windows Code Page 1252 system  and the last transfer (having
>> CP1252 data) were translating between UTF-8 and CP037.
>>
>> I'm interested in other folks war stories and what they've implemented
>for best
>> practices.  I've made clear to our developers and end

Re: File Transfer conundrum

[Hal Merritt]

My guess is that many shops are implementing PC to PC transfers and
buying some really expensive software to facilitate the process. That is
host>pc>network>pc>host. 

So far, we have been somewhat successful in insisting on a fully
automated, z/os based solution on our side. Our most compelling argument
is that we are not supposed to move sensitive data in the open over any
network, nor is the data supposed to reside in the open on any server,
even in a buffer. As much as the PC folks don't like to admit it, they
simply cannot meet that requirement.  They can come close, but there
seems to always be a point where the data can be intercepted. 

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
Behalf Of Zaromil Tisler
Sent: Thursday, January 10, 2008 8:19 AM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

On Wed, 9 Jan 2008 22:13:13 -0600, Bruce Baxter <...> wrote:

>ftp directly between the two systems, quite simply isn't an option.
Our
>business partner chose the mechanism in palce to deal with multiple
other
>shops, and they're not likely to want to do this sort of one-off thing
for us.

And how does your partner deliver files to other shops? Do they also get

corrupted data or do all of them use non-EBCDIC environments?

-- 
Zaromil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html
NOTICE: This electronic mail message and any files transmitted with it are 
intended
exclusively for the individual or entity to which it is addressed. The message, 
together with any attachment, may contain confidential and/or privileged 
information.
Any unauthorized review, use, printing, saving, copying, disclosure or 
distribution 
is strictly prohibited. If you have received this message in error, please 
immediately advise the sender by reply email and delete all copies.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Hal Merritt]

Try a suffix of '.BIN' on all files transferred as well as the 'BIN' FTP
option. Nothing seems to work 100% of the time, but that seems to work
most often. 

 
 
-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
Behalf Of Bruce Baxter
Sent: Wednesday, January 09, 2008 10:13 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

ftp directly between the two systems, quite simply isn't an option.  Our

business partner chose the mechanism in palce to deal with multiple
other 
shops, and they're not likely to want to do this sort of one-off thing
for us.

I also took some of the previous comments regarding the RDW site command

(which I'd seen before) and did some testing.  As stated by another
poster, 
there doesn't seem to be any way to get it to honor this and strip it
off on 
the final FTP.  

I also have to deal with the issue that EITHER the EBCDIC code pages on 
either end don't match (less likely) or the ASCII code pages don't match
(more 
likely) since the end user has subsequently reported to me that they've
seen 
some data corruption due to the translation.  I guess they were
expecting a 
cent sign character in someof their data.

 ves/ibm-main.html
NOTICE: This electronic mail message and any files transmitted with it are 
intended
exclusively for the individual or entity to which it is addressed. The message, 
together with any attachment, may contain confidential and/or privileged 
information.
Any unauthorized review, use, printing, saving, copying, disclosure or 
distribution 
is strictly prohibited. If you have received this message in error, please 
immediately advise the sender by reply email and delete all copies.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Zaromil Tisler]

On Wed, 9 Jan 2008 22:13:13 -0600, Bruce Baxter <...> wrote:

>ftp directly between the two systems, quite simply isn't an option.  Our
>business partner chose the mechanism in palce to deal with multiple other
>shops, and they're not likely to want to do this sort of one-off thing for us.

And how does your partner deliver files to other shops? Do they also get 
corrupted data or do all of them use non-EBCDIC environments?

-- 
Zaromil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Havelock, Glenn A]

If there is a copy of XCOM on both sides that would likely work well. Sometimes 
people have copies of it and are not aware that such copies are on z/os.

Regards,

Glenn

( Sent from blackberry)

Glenn Havelock
Cell 908 398 7726
[EMAIL PROTECTED]



- Original Message -
From: IBM Mainframe Discussion List 
To: IBM-MAIN@BAMA.UA.EDU 
Sent: Thu Jan 10 05:37:38 2008
Subject: Re: File Transfer conundrum

Has MQSeries been considered? I'd have thought it would have solved most 
problems like this. (unless one of the partners doesnt actually have 
MQ installed!)

-- 
Regards - Grant

Grant Ward Able
Senior Systems Architect
DTCC




"John S. Giltner, Jr." <[EMAIL PROTECTED]> 
Sent by: IBM Mainframe Discussion List 
10/01/2008 02:51
Please respond to
IBM Mainframe Discussion List 


To
IBM-MAIN@BAMA.UA.EDU
cc

Subject
Re: File Transfer conundrum






Any reason why you can't ftp directly between the two z/OS system?

If security is an issue you could use either IPSec tunnels between the 
two systems or setup IBM SecureFTP server (SSL'ed FTP).



Bruce Baxter wrote:
> We've routinely exhanged files with business partners running on z/OS 
> machines using tape for years.
> 
> We're now in the process of converting a number of these to electronic 
> means, using in part FTP.  This is being done at the behest of one of 
our 
> business partners, who (IMHO) hasn't thought through all the issues that 
the 
> use of FTP introduces in this process.  The central issue as I see it is 
that the 
> mainframes at either end of the pipeline are both EBCDIC and record 
oriented, 
> and the servers and ftp processes that lie between them to facilitate 
these 
> transfers do not have any inherent concept of record oriented files like 
the 
> mainframe.
> 
> I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED 
data 
> separately. 
> 
> The first files that we received were FIXED BLOCK, and had been 
translated 
> from EBCDIC to ASCII, most likely at the first transfer of the file from 
z/OS to 
> an ASCII based server platform (either Windows or AIX).  When they 
arrived 
> on our z/OS system, we had issues of data corruption because the data 
> contained zoned decimal data.  After some discussion, we agreed that 
we'd 
> transfer these files in BINARY mode at all steps along the way.  Thus, 
all we 
> had to do was ensure that the LRECL used for the destination dataset on 
> z/OS was the same as the source dataset.  This seems to be working OK.
> 
> Most recently, we've been having problems with other files that are 
VARIABLE 
> BLOCKED.  We received the first of these files last week, transmitted 
from end-
> to-end in BINARY mode.  What we got was not at all what we expected. 
> We've discovered that the initial FTP from z/OS to the server stripped 
off all 
> information regarding record length and thus record delineation. Because 

> there aren't any RDWs in front of every record, ftp doesn't know how 
long the 
> records are and just plunks the data into the destination dataset in 
chunks of 
> LRECL-4.  I did a bunch of research on z/OS FTP and there doesn't appear 
to 
> be any way to make it convey record length/delineation information to 
and 
> ASCII platform other than to use ASCII mode.  z/OS FTP appears to have 
> mechanisms for conveying this between two z/OS FTP systems, but that's 
not 
> possible here.  For the present time, we've had the file shipped with 
the initiatl 
> movement translating the data from EBCDIC to ASCII and all subsequent 
> transfers until the last one back to z/OS in BINARY mode.  However, I'm 
> concerned about the possibility of data corruption if the translate 
tables used 
> in the first step and the last step of this files travel aren't exact 
inversions of 
> each other.  This would certainly be possible of the initial ASCII 
transfer were 
> done to a Windows Code Page 1252 system  and the last transfer (having 
> CP1252 data) were translating between UTF-8 and CP037.
> 
> I'm interested in other folks war stories and what they've implemented 
for best 
> practices.  I've made clear to our developers and end-users that ftp is 
> certainly not a direct replacement for tape transfers.  It would appear 
that we 
> need lots of information about all the systems and transformations done 
in 
> moving the file from one system to another.  FTP doesn't convey this 
sort of 
> information in any way shape or form. 
> 
> What sort of options are there?
> 
> - Transmit/Receive would certainly be one, but would add a lot of 
overhead to 
> the process.
> - Removal of all non-display data from the files and subje

Re: File Transfer conundrum

[Ceruti, Gerard G]

What about scp ?, covers the UN*X world.

Regards
Gerard Ceruti 
may the 'z' be with you


-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
Behalf Of Grant Ward Able
Sent: 10 January 2008 12:38 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: Re: File Transfer conundrum

Has MQSeries been considered? I'd have thought it would have solved most

problems like this. (unless one of the partners doesnt actually have

MQ installed!)

-- 
Regards - Grant

Grant Ward Able
Senior Systems Architect
DTCC




"John S. Giltner, Jr." <[EMAIL PROTECTED]> 
Sent by: IBM Mainframe Discussion List 
10/01/2008 02:51
Please respond to
IBM Mainframe Discussion List 


To
IBM-MAIN@BAMA.UA.EDU
cc

Subject
Re: File Transfer conundrum






Any reason why you can't ftp directly between the two z/OS system?

If security is an issue you could use either IPSec tunnels between the 
two systems or setup IBM SecureFTP server (SSL'ed FTP).



Bruce Baxter wrote:
> We've routinely exhanged files with business partners running on z/OS 
> machines using tape for years.
> 
> We're now in the process of converting a number of these to electronic

> means, using in part FTP.  This is being done at the behest of one of 
our 
> business partners, who (IMHO) hasn't thought through all the issues
that 
the 
> use of FTP introduces in this process.  The central issue as I see it
is 
that the 
> mainframes at either end of the pipeline are both EBCDIC and record 
oriented, 
> and the servers and ftp processes that lie between them to facilitate 
these 
> transfers do not have any inherent concept of record oriented files
like 
the 
> mainframe.
> 
> I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED 
data 
> separately. 
> 
> The first files that we received were FIXED BLOCK, and had been 
translated 
> from EBCDIC to ASCII, most likely at the first transfer of the file
from 
z/OS to 
> an ASCII based server platform (either Windows or AIX).  When they 
arrived 
> on our z/OS system, we had issues of data corruption because the data 
> contained zoned decimal data.  After some discussion, we agreed that 
we'd 
> transfer these files in BINARY mode at all steps along the way.  Thus,

all we 
> had to do was ensure that the LRECL used for the destination dataset
on 
> z/OS was the same as the source dataset.  This seems to be working OK.
> 
> Most recently, we've been having problems with other files that are 
VARIABLE 
> BLOCKED.  We received the first of these files last week, transmitted 
from end-
> to-end in BINARY mode.  What we got was not at all what we expected. 
> We've discovered that the initial FTP from z/OS to the server stripped

off all 
> information regarding record length and thus record delineation.
Because 

__

Standard Bank Disclaimer and Confidentiality Note

This e-mail, its attachments and any rights attaching hereto are, unless the 
context clearly indicates otherwise, the property of Standard Bank Group Limited
and/or its subsidiaries ("the Group"). It is confidential, private and intended 
for the addressee only. Should you not be the addressee and receive this e-mail 
by
mistake, kindly notify the sender, and delete this e-mail, immediately and do 
not disclose or use same in any manner whatsoever. Views and opinions
expressed in this e-mail are those of the sender unless clearly stated as those 
of the Group. The Group accepts no liability whatsoever for any loss or
damages whatsoever and howsoever incurred, or suffered, resulting, or arising, 
from the use of this email or its attachments. The Group does not warrant the 
integrity
of this e-mail nor that it is free of errors, viruses, interception or 
interference. Licensed divisions of the Standard Bank Group are authorised 
financial services providers
in terms of the Financial Advisory and Intermediary Services Act, No 37 of 2002 
(FAIS).
For information about the Standard Bank Group Limited visit our website 
http://www.standardbank.co.za
___

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Grant Ward Able]

Has MQSeries been considered? I'd have thought it would have solved most 
problems like this. (unless one of the partners doesnt actually have 
MQ installed!)

-- 
Regards - Grant

Grant Ward Able
Senior Systems Architect
DTCC




"John S. Giltner, Jr." <[EMAIL PROTECTED]> 
Sent by: IBM Mainframe Discussion List 
10/01/2008 02:51
Please respond to
IBM Mainframe Discussion List 


To
IBM-MAIN@BAMA.UA.EDU
cc

Subject
Re: File Transfer conundrum






Any reason why you can't ftp directly between the two z/OS system?

If security is an issue you could use either IPSec tunnels between the 
two systems or setup IBM SecureFTP server (SSL'ed FTP).



Bruce Baxter wrote:
> We've routinely exhanged files with business partners running on z/OS 
> machines using tape for years.
> 
> We're now in the process of converting a number of these to electronic 
> means, using in part FTP.  This is being done at the behest of one of 
our 
> business partners, who (IMHO) hasn't thought through all the issues that 
the 
> use of FTP introduces in this process.  The central issue as I see it is 
that the 
> mainframes at either end of the pipeline are both EBCDIC and record 
oriented, 
> and the servers and ftp processes that lie between them to facilitate 
these 
> transfers do not have any inherent concept of record oriented files like 
the 
> mainframe.
> 
> I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED 
data 
> separately. 
> 
> The first files that we received were FIXED BLOCK, and had been 
translated 
> from EBCDIC to ASCII, most likely at the first transfer of the file from 
z/OS to 
> an ASCII based server platform (either Windows or AIX).  When they 
arrived 
> on our z/OS system, we had issues of data corruption because the data 
> contained zoned decimal data.  After some discussion, we agreed that 
we'd 
> transfer these files in BINARY mode at all steps along the way.  Thus, 
all we 
> had to do was ensure that the LRECL used for the destination dataset on 
> z/OS was the same as the source dataset.  This seems to be working OK.
> 
> Most recently, we've been having problems with other files that are 
VARIABLE 
> BLOCKED.  We received the first of these files last week, transmitted 
from end-
> to-end in BINARY mode.  What we got was not at all what we expected. 
> We've discovered that the initial FTP from z/OS to the server stripped 
off all 
> information regarding record length and thus record delineation. Because 

> there aren't any RDWs in front of every record, ftp doesn't know how 
long the 
> records are and just plunks the data into the destination dataset in 
chunks of 
> LRECL-4.  I did a bunch of research on z/OS FTP and there doesn't appear 
to 
> be any way to make it convey record length/delineation information to 
and 
> ASCII platform other than to use ASCII mode.  z/OS FTP appears to have 
> mechanisms for conveying this between two z/OS FTP systems, but that's 
not 
> possible here.  For the present time, we've had the file shipped with 
the initiatl 
> movement translating the data from EBCDIC to ASCII and all subsequent 
> transfers until the last one back to z/OS in BINARY mode.  However, I'm 
> concerned about the possibility of data corruption if the translate 
tables used 
> in the first step and the last step of this files travel aren't exact 
inversions of 
> each other.  This would certainly be possible of the initial ASCII 
transfer were 
> done to a Windows Code Page 1252 system  and the last transfer (having 
> CP1252 data) were translating between UTF-8 and CP037.
> 
> I'm interested in other folks war stories and what they've implemented 
for best 
> practices.  I've made clear to our developers and end-users that ftp is 
> certainly not a direct replacement for tape transfers.  It would appear 
that we 
> need lots of information about all the systems and transformations done 
in 
> moving the file from one system to another.  FTP doesn't convey this 
sort of 
> information in any way shape or form. 
> 
> What sort of options are there?
> 
> - Transmit/Receive would certainly be one, but would add a lot of 
overhead to 
> the process.
> - Removal of all non-display data from the files and subjecting them to 
ASCII 
> translation at every step would also be an option, but that would likely 
be 
> rejected by our business partner as too much work.
> - Are there any options to z/OS FTP that would allow record formatting 
> information to be conveyed in the file, if we presume that we'd transfer 
it in 
> binary mode at every step.
> 
> Has anyone come across any c

Re: File Transfer conundrum

[Jim Mulder]

IBM Mainframe Discussion List  wrote on 01/10/2008 
01:34:47 AM:

> On Jan 9, 2008, at 11:17 PM, Skip Robinson wrote:
> > -SNP---
> > -- TSO XMIT/RECEIVE, which has 'always' been incorporated in z/OS
> > and its
> > ancestors
> > SNIP
> 
> I am not sure why you put quotes around always but IIRC that TSOe
> included xmit/receive as to when TSOe came out I think it was in the
> mid to late 80's (somebody please give an a real date).

  The earliest source code for INMXM appears to be for FMID JBB1114.
According to some FMID translator tool on our system:

JBB1114 
 PUT Support (Y) / MVS environment 
 - 
 Program  Supply   Title 
 5665-285  TSO/Extension R1.0 - MVS/370 IDTF (TSO/E>


  The TSO developers at that time used a different Change Activity
format, which did not contain a date.

  My userid.NAMES.TEXT   data set was created in 1984, but I am 
not sure if that coincided with TRANSMIT/RECEIVE in TSO/E, or
with the IBM internal XMIT/RECEIVE which was the ancestor of
TRANSMIT/RECEIVE.

Jim Mulder   z/OS System Test   IBM Corp.  Poughkeepsie,  NY

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Ed Gould]

On Jan 9, 2008, at 11:17 PM, Skip Robinson wrote:

-SNP---
-- TSO XMIT/RECEIVE, which has 'always' been incorporated in z/OS  
and its

ancestors
SNIP


I am not sure why you put quotes around always but IIRC that TSOe  
included xmit/receive as to when TSOe came out I think it was in the  
mid to late 80's (somebody please give an a real date).


Ed

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Skip Robinson]

In view of the responses so far, it looks like the only 'reasonable'
(immediate, cheap, self-realizable) solution is to transfer fixed length
records. To summarize suggested options:

-- IBM's TERSE/UNTERSE program, which has 'always' been freely available
via download to all mainframe customers but is now incorporated into z/OS,
which means full support and automatic maintenance

-- TSO XMIT/RECEIVE, which has 'always' been incorporated in z/OS and its
ancestors

The first yields a compressed file at the expense of perhaps considerable
CPU cycles on both ends, which may or may not be worth the cost given
today's network speeds.

The second involves little overhead and yields a NETDATA file that could
even be used on VM/CMS , which may or may not ever be useful.

Unless you're shipping gigantic files that would benefit from compression,
I would go with XMIT/RECEIVE because the actual transmitted file can be
'browsed' to determine original file name and attributes. A tersed file
looks like garbage until it's untersed.

Either method supports both FB and VB customer data, so a single process
would work with any type of file, reducing complexity and training needs.
As a far-out bonus, you could even transfer IDCAMS unloaded VSAM
files--including catalogs if you were so motivated.

Trying to get FTP to handle other than FB files in the realistic future
seems far more trouble the effort is worth.

.
.
JO.Skip Robinson
Southern California Edison Company
Electric Dragon Team Paddler
SHARE MVS Program Co-Manager
626-302-7535 Office
323-715-0595 Mobile
[EMAIL PROTECTED]


   
 Bruce Baxter  
 <[EMAIL PROTECTED] 
 .STATE.NY.US>  To 
 Sent by: IBM  IBM-MAIN@BAMA.UA.EDU
 Mainframe  cc 
 Discussion List   
 <[EMAIL PROTECTED]         Subject 
 .EDU> Re: File Transfer conundrum 
   
   
 01/09/2008 08:13  
 PM
   
   
 Please respond to 
   IBM Mainframe   
  Discussion List  
 <[EMAIL PROTECTED] 
   .EDU>   
   
   




ftp directly between the two systems, quite simply isn't an option.  Our
business partner chose the mechanism in palce to deal with multiple other
shops, and they're not likely to want to do this sort of one-off thing for
us.

I also took some of the previous comments regarding the RDW site command
(which I'd seen before) and did some testing.  As stated by another poster,

there doesn't seem to be any way to get it to honor this and strip it off
on
the final FTP.

I also have to deal with the issue that EITHER the EBCDIC code pages on
either end don't match (less likely) or the ASCII code pages don't match
(more
likely) since the end user has subsequently reported to me that they've
seen
some data corruption due to the translation.  I guess they were expecting a

cent sign character in someof their data.

On Wed, 9 Jan 2008 21:51:23 -0500, John S. Giltner, Jr.
<[EMAIL PROTECTED]> wrote:

>Any reason why you can't ftp directly between the two z/OS system?
>
>If security is an issue you could use either IPSec tunnels between the
>two systems or setup IBM SecureFTP server (SSL'ed FTP).
>
>
>
>Bruce Baxter wrote:
>> We've routinely exhanged files with business partners running on z/OS
>> machines using tape for years.
>>
>> We're now in the process of converting a number of these to electronic
>> means, using in part FTP.  This is being done at the behest of one of
our
>> business partners, who (IMHO) hasn't thought through all the issues that

the
>> use of FTP introduces in this process.  The 

Re: File Transfer conundrum

[Ed Gould]

On Jan 9, 2008, at 10:13 PM, Bruce Baxter wrote:

ftp directly between the two systems, quite simply isn't an  
option.  Our
business partner chose the mechanism in palce to deal with multiple  
other
shops, and they're not likely to want to do this sort of one-off  
thing for us.


I also took some of the previous comments regarding the RDW site  
command
(which I'd seen before) and did some testing.  As stated by another  
poster,
there doesn't seem to be any way to get it to honor this and strip  
it off on

the final FTP.

I also have to deal with the issue that EITHER the EBCDIC code  
pages on
either end don't match (less likely) or the ASCII code pages don't  
match (more
likely) since the end user has subsequently reported to me that  
they've seen
some data corruption due to the translation.  I guess they were  
expecting a

cent sign character in someof their data.



Bruce,

I can't answer your question directly but

I worked at a company that for years took ORDERS on how file transfer  
was to be done.


I was instrumental on getting every company to change over to use  
either BSC or SNA (mostly SNA) because JES3 (at the time did not  
support it and I think VM didn't either). It took me about 1 and 1/2  
years to get everyone changed over. I did it with a combination of  
talking to the techies DIRECTLY at the other companies  and I also  
met with the people who went out to talk with the really remote users  
and let them know via spec sheet what we offered. I got the best  
results when I was at GUIDE and talking to people from the respective  
companies and convinced them to change. I was doing an average of 1  
company a week (sometimes 2 or 3). I would get a call on Monday and  
go over the changes needed to JES2 and their 3745. A week later (or  
the length of time they asked for) I  would call them back and I  
would set up a schedule so in case of network issues I would have our  
hardware type there to look at the line to see what the issue was. In  
the year and a half I converted well over 200 users and I had (new)  
users on a wait list clamoring to get on board. There were about 3  
issues I bumped into 1. sysprog (or the network person) from the  
other company would get bumped for another project) and had to  
reschedule 2. The sysprog would think they were the expert and go off  
on their own and set up the JES2 parms or VTAM mode  table. 3. The  
line was not switched over to us properly for temporary testing.


The "other" quite minor issue was the program that sent and received  
the the data was an IBM FDP. Sometimes IBM would squawk about sending  
it out. I never did get a good answer as to why they squawked but as  
soon as I heard they had a license (it was extremely cheap $25 a  
month IIRC)  I would cut a tape and have it delivered in source  
format the next day. IBM gave out source and all they had to do was  
compile and link it and it was ready to go (no APF required). The  
sysprogs could handle that easily and it was normally done the day  
they got the tape. The other issue we ran into was a VTAM issue (its  
not an issue now but then it was) but we were able to get by rather  
easily by a small change on their end.


The point is if you let the techies talk without management in the  
middle things tend to go rather smoothly at least that is what I found.


Ed

ps: After I got the connection going and the network people were  
aware of the connection I turned the rest of the process over to our  
production support and they were average but OK and they got the user  
up and running within a week.


Without the help of the hardware network people I would have stumbled  
a little bit as we had a good group. I was OK with guessing about  
some of the hardware issues but they were excellent and could spot  
the issue in 2 or 3 minutes or less. Our hardest user was a bisync  
RJE station that would randomly drop data I just sicked our hardware  
people on them and they got the problem resolved after a month or so.


We had by the end of the 1.5 year well over 100 and probably closer  
to 200 people NJE and RJE and RSCS and other (too far back to  
remember what other were but they were unusual).


--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Bruce Baxter]

ftp directly between the two systems, quite simply isn't an option.  Our 
business partner chose the mechanism in palce to deal with multiple other 
shops, and they're not likely to want to do this sort of one-off thing for us.

I also took some of the previous comments regarding the RDW site command 
(which I'd seen before) and did some testing.  As stated by another poster, 
there doesn't seem to be any way to get it to honor this and strip it off on 
the final FTP.  

I also have to deal with the issue that EITHER the EBCDIC code pages on 
either end don't match (less likely) or the ASCII code pages don't match (more 
likely) since the end user has subsequently reported to me that they've seen 
some data corruption due to the translation.  I guess they were expecting a 
cent sign character in someof their data.

On Wed, 9 Jan 2008 21:51:23 -0500, John S. Giltner, Jr. 
<[EMAIL PROTECTED]> wrote:

>Any reason why you can't ftp directly between the two z/OS system?
>
>If security is an issue you could use either IPSec tunnels between the
>two systems or setup IBM SecureFTP server (SSL'ed FTP).
>
>
>
>Bruce Baxter wrote:
>> We've routinely exhanged files with business partners running on z/OS
>> machines using tape for years.
>>
>> We're now in the process of converting a number of these to electronic
>> means, using in part FTP.  This is being done at the behest of one of our
>> business partners, who (IMHO) hasn't thought through all the issues that 
the
>> use of FTP introduces in this process.  The central issue as I see it is 
>> that 
the
>> mainframes at either end of the pipeline are both EBCDIC and record 
oriented,
>> and the servers and ftp processes that lie between them to facilitate these
>> transfers do not have any inherent concept of record oriented files like the
>> mainframe.
>>
>> I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED 
data
>> separately.
>>
>> The first files that we received were FIXED BLOCK, and had been translated
>> from EBCDIC to ASCII, most likely at the first transfer of the file from 
>> z/OS 
to
>> an ASCII based server platform (either Windows or AIX).  When they 
arrived
>> on our z/OS system, we had issues of data corruption because the data
>> contained zoned decimal data.  After some discussion, we agreed that we'd
>> transfer these files in BINARY mode at all steps along the way.  Thus, all we
>> had to do was ensure that the LRECL used for the destination dataset on
>> z/OS was the same as the source dataset.  This seems to be working OK.
>>
>> Most recently, we've been having problems with other files that are 
VARIABLE
>> BLOCKED.  We received the first of these files last week, transmitted from 
end-
>> to-end in BINARY mode.  What we got was not at all what we expected.
>> We've discovered that the initial FTP from z/OS to the server stripped off 
all
>> information regarding record length and thus record delineation.  Because
>> there aren't any RDWs in front of every record, ftp doesn't know how long 
the
>> records are and just plunks the data into the destination dataset in chunks 
of
>> LRECL-4.  I did a bunch of research on z/OS FTP and there doesn't appear 
to
>> be any way to make it convey record length/delineation information to and
>> ASCII platform other than to use ASCII mode.  z/OS FTP appears to have
>> mechanisms for conveying this between two z/OS FTP systems, but that's 
not
>> possible here.  For the present time, we've had the file shipped with the 
initiatl
>> movement translating the data from EBCDIC to ASCII and all subsequent
>> transfers until the last one back to z/OS in BINARY mode.  However, I'm
>> concerned about the possibility of data corruption if the translate tables 
used
>> in the first step and the last step of this files travel aren't exact 
>> inversions 
of
>> each other.  This would certainly be possible of the initial ASCII transfer 
were
>> done to a Windows Code Page 1252 system  and the last transfer (having
>> CP1252 data) were translating between UTF-8 and CP037.
>>
>> I'm interested in other folks war stories and what they've implemented for 
best
>> practices.  I've made clear to our developers and end-users that ftp is
>> certainly not a direct replacement for tape transfers.  It would appear that 
we
>> need lots of information about all the systems and transformations done in
>> moving the file from one system to another.  FTP doesn't convey this sort 
of
>> information in any way shape or form.
>>
>> What sort of options are there?
>>
>> - Transmit/Receive would certainly be one, but would add a lot of 
overhead to
>> the process.
>> - Removal of all non-display data from the files and subjecting them to 
ASCII
>> translation at every step would also be an option, but that would likely be
>> rejected by our business partner as too much work.
>> - Are there any options to z/OS FTP that would allow record formatting
>> information to be conveyed in the file, if we presume that we'd trans

Re: File Transfer conundrum

[John S. Giltner, Jr.]

Any reason why you can't ftp directly between the two z/OS system?

If security is an issue you could use either IPSec tunnels between the 
two systems or setup IBM SecureFTP server (SSL'ed FTP).




Bruce Baxter wrote:
We've routinely exhanged files with business partners running on z/OS 
machines using tape for years.


We're now in the process of converting a number of these to electronic 
means, using in part FTP.  This is being done at the behest of one of our 
business partners, who (IMHO) hasn't thought through all the issues that the 
use of FTP introduces in this process.  The central issue as I see it is that the 
mainframes at either end of the pipeline are both EBCDIC and record oriented, 
and the servers and ftp processes that lie between them to facilitate these 
transfers do not have any inherent concept of record oriented files like the 
mainframe.


I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED data 
separately.  

The first files that we received were FIXED BLOCK, and had been translated 
from EBCDIC to ASCII, most likely at the first transfer of the file from z/OS to 
an ASCII based server platform (either Windows or AIX).  When they arrived 
on our z/OS system, we had issues of data corruption because the data 
contained zoned decimal data.  After some discussion, we agreed that we'd 
transfer these files in BINARY mode at all steps along the way.  Thus, all we 
had to do was ensure that the LRECL used for the destination dataset on 
z/OS was the same as the source dataset.  This seems to be working OK.


Most recently, we've been having problems with other files that are VARIABLE 
BLOCKED.  We received the first of these files last week, transmitted from end-
to-end in BINARY mode.  What we got was not at all what we expected.  
We've discovered that the initial FTP from z/OS to the server stripped off all 
information regarding record length and thus record delineation.  Because 
there aren't any RDWs in front of every record, ftp doesn't know how long the 
records are and just plunks the data into the destination dataset in chunks of 
LRECL-4.  I did a bunch of research on z/OS FTP and there doesn't appear to 
be any way to make it convey record length/delineation information to and 
ASCII platform other than to use ASCII mode.  z/OS FTP appears to have 
mechanisms for conveying this between two z/OS FTP systems, but that's not 
possible here.  For the present time, we've had the file shipped with the initiatl 
movement translating the data from EBCDIC to ASCII and all subsequent 
transfers until the last one back to z/OS in BINARY mode.  However, I'm 
concerned about the possibility of data corruption if the translate tables used 
in the first step and the last step of this files travel aren't exact inversions of 
each other.  This would certainly be possible of the initial ASCII transfer were 
done to a Windows Code Page 1252 system  and the last transfer (having 
CP1252 data) were translating between UTF-8 and CP037.


I'm interested in other folks war stories and what they've implemented for best 
practices.  I've made clear to our developers and end-users that ftp is 
certainly not a direct replacement for tape transfers.  It would appear that we 
need lots of information about all the systems and transformations done in 
moving the file from one system to another.  FTP doesn't convey this sort of 
information in any way shape or form.  


What sort of options are there?

- Transmit/Receive would certainly be one, but would add a lot of overhead to 
the process.
- Removal of all non-display data from the files and subjecting them to ASCII 
translation at every step would also be an option, but that would likely be 
rejected by our business partner as too much work.
- Are there any options to z/OS FTP that would allow record formatting 
information to be conveyed in the file, if we presume that we'd transfer it in 
binary mode at every step.


Has anyone come across any clear helpful best practice type information or 
sites?  I'd be interested in anything anyone has.


Regards,
Bruce Baxter
Manager of DP Tech Services
NYS Dept of Tax and Finance.



--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Paul Gilmartin]

On Wed, 9 Jan 2008 15:36:28 -0600, Thomas Kern wrote:

>A good philosophy is to use some program common to both sending and
>receiving systems that will compress and possibly encrypt the data,
>retaining the file's original record formatting as part of the new data
>file. Transfer that new data file and decrypt/expand it at the receiving
>site. XMIT can be used to simply reblock the data to retain record
>formatting, TERSE/DETERSE can be used to compress and reblock. PKZIP can
>also be used. If you have a mainframe based data encryption product, it
>might have its own compress&encrypt utility, if you can convince the
>receiving system to use the same product.
>
In the OP's situation, both endpoints are z/OS, so the middleman needn't
understand the format; it should be just bits to him.

Now that TERSE is becoming a base z/OS facility, it should be made an
option of FTP to save a separate job step:

LOCSITE TERSE

>On Wed, 9 Jan 2008 15:24:32 -0600, Bruce Baxter
> wrote:
>>use of FTP introduces in this process.  The central issue as I see it is that 
>>the
>>mainframes at either end of the pipeline are both EBCDIC and record oriented,
>>and the servers and ftp processes that lie between them to facilitate these
>>transfers do not have any inherent concept of record oriented files like the
>>mainframe.

I did "HELP LOCSITE" with z/OS FTP client.  It tells me:

RDw RDW from VB/VBS files is retained as data.
NORDw   RDW from VB/VBS files is discarded as not data.

Sigh.  Too literally true.  What I observed was that on PUT the RDWs
were transmitted as if they were data.  On GET, they were treated as
data, not as RDWs.  Didn't any coder or design reviewer or whatever
have the guts to say to the specifier, "The way you describe this is
probably not what the customer needs."?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Paul Gilmartin]

On Wed, 9 Jan 2008 16:31:03 -0500, Gray, Larry - Larry A wrote:
>
>Have you tried LOCSITE RDW on your send to keep the length header?
>
I just tried it.  It appears that on the send ("PUT") it does
keep and transmit the RDWs.  On the GET, however, it treats
them as data, and ISPF reports "INVALID CHARACTERS".

Does anyone perceive this as other than a bug?

-- gil

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Thomas Kern]

A good philosophy is to use some program common to both sending and
receiving systems that will compress and possibly encrypt the data,
retaining the file's original record formatting as part of the new data
file. Transfer that new data file and decrypt/expand it at the receiving
site. XMIT can be used to simply reblock the data to retain record
formatting, TERSE/DETERSE can be used to compress and reblock. PKZIP can
also be used. If you have a mainframe based data encryption product, it
might have its own compress&encrypt utility, if you can convince the
receiving system to use the same product.

/Tom Kern 


On Wed, 9 Jan 2008 15:24:32 -0600, Bruce Baxter
<[EMAIL PROTECTED]> wrote:
>We've routinely exhanged files with business partners running on z/OS
>machines using tape for years.
>
>We're now in the process of converting a number of these to electronic
>means, using in part FTP.  This is being done at the behest of one of our
>business partners, who (IMHO) hasn't thought through all the issues that the
>use of FTP introduces in this process.  The central issue as I see it is
that the
>mainframes at either end of the pipeline are both EBCDIC and record oriented,
>and the servers and ftp processes that lie between them to facilitate these
>transfers do not have any inherent concept of record oriented files like the
>mainframe.
>
>I'm going to treat FIXED BLOCK data separately from VARIABLE BLOCKED data
>separately.
>
>The first files that we received were FIXED BLOCK, and had been translated
>from EBCDIC to ASCII, most likely at the first transfer of the file from
z/OS to
>an ASCII based server platform (either Windows or AIX).  When they arrived
>on our z/OS system, we had issues of data corruption because the data
>contained zoned decimal data.  After some discussion, we agreed that we'd
>transfer these files in BINARY mode at all steps along the way.  Thus, all we
>had to do was ensure that the LRECL used for the destination dataset on
>z/OS was the same as the source dataset.  This seems to be working OK.
>
>Most recently, we've been having problems with other files that are VARIABLE
>BLOCKED.  We received the first of these files last week, transmitted from end-
>to-end in BINARY mode.  What we got was not at all what we expected.
>We've discovered that the initial FTP from z/OS to the server stripped off all
>information regarding record length and thus record delineation.  Because
>there aren't any RDWs in front of every record, ftp doesn't know how long the
>records are and just plunks the data into the destination dataset in chunks of
>LRECL-4.  I did a bunch of research on z/OS FTP and there doesn't appear to
>be any way to make it convey record length/delineation information to and
>ASCII platform other than to use ASCII mode.  z/OS FTP appears to have
>mechanisms for conveying this between two z/OS FTP systems, but that's not
>possible here.  For the present time, we've had the file shipped with the
initiatl
>movement translating the data from EBCDIC to ASCII and all subsequent
>transfers until the last one back to z/OS in BINARY mode.  However, I'm
>concerned about the possibility of data corruption if the translate tables used
>in the first step and the last step of this files travel aren't exact
inversions of
>each other.  This would certainly be possible of the initial ASCII transfer
were
>done to a Windows Code Page 1252 system  and the last transfer (having
>CP1252 data) were translating between UTF-8 and CP037.
>
>I'm interested in other folks war stories and what they've implemented for best
>practices.  I've made clear to our developers and end-users that ftp is
>certainly not a direct replacement for tape transfers.  It would appear that we
>need lots of information about all the systems and transformations done in
>moving the file from one system to another.  FTP doesn't convey this sort of
>information in any way shape or form.
>
>What sort of options are there?
>
>- Transmit/Receive would certainly be one, but would add a lot of overhead to
>the process.
>- Removal of all non-display data from the files and subjecting them to ASCII
>translation at every step would also be an option, but that would likely be
>rejected by our business partner as too much work.
>- Are there any options to z/OS FTP that would allow record formatting
>information to be conveyed in the file, if we presume that we'd transfer it in
>binary mode at every step.
>
>Has anyone come across any clear helpful best practice type information or
>sites?  I'd be interested in anything anyone has.
>
>Regards,
>Bruce Baxter
>Manager of DP Tech Services
>NYS Dept of Tax and Finance.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[Gray, Larry - Larry A]

NOTICE:
All information in and attached to the e-mail(s) below may be proprietary, 
confidential, privileged and otherwise protected from improper or erroneous 
disclosure.  If you are not the sender's intended recipient, you are not 
authorized to intercept, read, print, retain, copy, forward, or disseminate 
this message.  If you have erroneously received this communication, please 
notify the sender immediately by phone (704-758-1000) or by e-mail and destroy 
all copies of this message (electronic, paper, or otherwise).  Thank you.

Have you tried LOCSITE RDW on your send to keep the length header? 


Larry Gray
Large Systems Engineering
Lowe's Companies
336-658-7944

-Original Message-
From: IBM Mainframe Discussion List [mailto:[EMAIL PROTECTED] On
Behalf Of Bruce Baxter
Sent: Wednesday, January 09, 2008 4:25 PM
To: IBM-MAIN@BAMA.UA.EDU
Subject: File Transfer conundrum

We've routinely exhanged files with business partners running on z/OS
machines using tape for years.

(snip)

Has anyone come across any clear helpful best practice type information
or sites?  I'd be interested in anything anyone has.

Regards,
Bruce Baxter
Manager of DP Tech Services
NYS Dept of Tax and Finance.

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html

Re: File Transfer conundrum

[McKown, John]

The simpliest thing, in my opinion, is to use TERSE on both ends and do
binary transfers. 

--
John McKown
Senior Systems Programmer
HealthMarkets
Keeping the Promise of Affordable Coverage
Administrative Services Group
Information Technology

The information contained in this e-mail message may be privileged
and/or confidential.  It is for intended addressee(s) only.  If you are
not the intended recipient, you are hereby notified that any disclosure,
reproduction, distribution or other use of this communication is
strictly prohibited and could, in certain circumstances, be a criminal
offense.  If you have received this e-mail in error, please notify the
sender by reply and delete this message without copying or disclosing
it. 

--
For IBM-MAIN subscribe / signoff / archive access instructions,
send email to [EMAIL PROTECTED] with the message: GET IBM-MAIN INFO
Search the archives at http://bama.ua.edu/archives/ibm-main.html