Re: PIPEDDR and attached DASD
I have solved this problem and present the resolution for the benefit of others: it was an MTU mismatch between the VM TCPIP stacks (set at 8992) and what the network between would support (1500). Further testing I did had showed that ATTACH vs. MDISK and the versions o f PIPELINES and PIPEDDR were not the cause. Instead, sucess or failure was dependent on the data being transferred and whether it traversed the external network or not. After ripping apart PIPEDDR to understand it so that I could add debugging code and options I found that the transfer for a particular disk always failed at a unique track within that disk. Manually added pacing (via DELAY stages) would work (although *tediously* more slowly than expected based on the delay value used), but not if the delay was too small. Eventually, while looking for pacing information in the TCPIP stack I read the reference that Selecting an MTU size that is too large may cause a client application to hang. The light bulb went o n and after adjusting the MTU size used in the VM TCPIP stack everything works fine now. Brian Nielsen On Tue, 5 Apr 2011 12:20:59 -0500, Brian Nielsen bniel...@sco.idaho.gov wrote: Should PIPEDDR work with attached DASD or does it only support MDISKs? The documentation doesn't seem to say. I get an error with attached DAS D but it works fine with a full pack MDISK of the same DASD volume when doing a dump/restore over TCPIP. Using attached DASD will avoid label conflicts and also avoids maintaining hardcoded MDISKs with DEVNO's in the directory. Unfortunately, the DEFINE MDISK command doesn't have a DEVNO option. Here is what the failure looks like from the receiving and sending sides using attached DASD at both ends: - pipeddr restore * 6930 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6930. Sending user is BNIELSEN at VMP2 Receiving data from 172.16.64.45 PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.01 08:58:15 pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 PIPTCQ1015E ERRNO 32: EPIPE. PIPMSG004I ... Issued from stage 7 of pipeline 1 name ipread. PIPMSG001I ... Running tcpclient 172.16.64.44 11000 linger 10 reuseaddr U. Dump failed. Ready(01015); T=0.02/0.03 07:54:34 If I create full pack MDISKs via DEFINE MDISK (starting at cyl zero) it works fine, as shown below. pipeddr restore * 6930 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6930. Sending user is BNIELSEN at VMP2 Receiving data from 172.16.64.45 41 MB received. Data restored successfully. Ready; T=4.04/4.54 09:34:18 --- pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 -- All data sent to BNIELSEN AT VMP1 -- 41 MB transmitted. Dump completed. Ready; T=6.27/8.21 08:30:37 Brian Nielsen = ===
Re: PIPEDDR and attached DASD
Sometimes you have to vary the device off and on before the system detects a new VOLSER. -Original Message- From: Bruce Hayden bjhay...@gmail.com Sent: Apr 5, 2011 3:33 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: PIPEDDR and attached DASD I don't know why the mdisk works and the attached disk doesn't. If the label is still SCRTCH, then nothing was written on the disk. That seems like the TCP/IP connection wasn't established correctly. We should take this offline to work on it further. Dave Lewis Sterling Commerce An IBM company Connect:Direct Level 3 Support
Re: PIPEDDR and attached DASD
On Wednesday, 04/06/2011 at 10:57 EDT, Dave david_v_le...@earthlink.net wrote: Sometimes you have to vary the device off and on before the system detects a new VOLSER. When a guest DETACHes a volume, CP re-reads the volser. (btw, Dave, you have reply-to: set to your id.) Alan Altmark z/VM and Linux on System z Consultant IBM System Lab Services and Training ibm.com/systems/services/labservices office: 607.429.3323 mobile; 607.321.7556 alan_altm...@us.ibm.com IBM Endicott
PIPEDDR and attached DASD
Should PIPEDDR work with attached DASD or does it only support MDISKs? The documentation doesn't seem to say. I get an error with attached DASD but it works fine with a full pack MDISK of the same DASD volume when doing a dump/restore over TCPIP. Using attached DASD will avoid label conflicts and also avoids maintainin g hardcoded MDISKs with DEVNO's in the directory. Unfortunately, the DEFIN E MDISK command doesn't have a DEVNO option. Here is what the failure looks like from the receiving and sending sides using attached DASD at both ends: - pipeddr restore * 6930 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6930. Sending user is BNIELSEN at VMP2 Receiving data from 172.16.64.45 PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.01 08:58:15 pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 PIPTCQ1015E ERRNO 32: EPIPE. PIPMSG004I ... Issued from stage 7 of pipeline 1 name ipread. PIPMSG001I ... Running tcpclient 172.16.64.44 11000 linger 10 reuseaddr U. Dump failed. Ready(01015); T=0.02/0.03 07:54:34 If I create full pack MDISKs via DEFINE MDISK (starting at cyl zero) it works fine, as shown below. pipeddr restore * 6930 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6930. Sending user is BNIELSEN at VMP2 Receiving data from 172.16.64.45 41 MB received. Data restored successfully. Ready; T=4.04/4.54 09:34:18 --- pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 -- All data sent to BNIELSEN AT VMP1 -- 41 MB transmitted. Dump completed. Ready; T=6.27/8.21 08:30:37 Brian Nielsen
Re: PIPEDDR and attached DASD
PIPEDDR should work fine with attached DASD. I just tried it on the latest version and some older levels and they all worked fine with a 3390-3 with both the source disk and target disk attached. I have the latest Pipelines runtime module, if that makes a difference. PIPEDDR doesn't really do anything differently for attached DASD, it just passes the virtual address you specified to the trackread or trackwrite stage of Pipelines. On Tue, Apr 5, 2011 at 1:20 PM, Brian Nielsen bniel...@sco.idaho.gov wrote: Should PIPEDDR work with attached DASD or does it only support MDISKs? The documentation doesn't seem to say. I get an error with attached DASD but it works fine with a full pack MDISK of the same DASD volume when doing a dump/restore over TCPIP. Using attached DASD will avoid label conflicts and also avoids maintaining hardcoded MDISKs with DEVNO's in the directory. Unfortunately, the DEFINE MDISK command doesn't have a DEVNO option. Here is what the failure looks like from the receiving and sending sides using attached DASD at both ends: - pipeddr restore * 6930 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6930. Sending user is BNIELSEN at VMP2 Receiving data from 172.16.64.45 PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.01 08:58:15 pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 PIPTCQ1015E ERRNO 32: EPIPE. PIPMSG004I ... Issued from stage 7 of pipeline 1 name ipread. PIPMSG001I ... Running tcpclient 172.16.64.44 11000 linger 10 reuseaddr U. Dump failed. Ready(01015); T=0.02/0.03 07:54:34 If I create full pack MDISKs via DEFINE MDISK (starting at cyl zero) it works fine, as shown below. pipeddr restore * 6930 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6930. Sending user is BNIELSEN at VMP2 Receiving data from 172.16.64.45 41 MB received. Data restored successfully. Ready; T=4.04/4.54 09:34:18 --- pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 -- All data sent to BNIELSEN AT VMP1 -- 41 MB transmitted. Dump completed. Ready; T=6.27/8.21 08:30:37 Brian Nielsen -- Bruce Hayden z/VM and Linux on System z ATS IBM, Endicott, NY
Re: PIPEDDR and attached DASD
On Tue, 5 Apr 2011 13:55:57 -0400, Bruce Hayden bjhay...@gmail.com wrot e: PIPEDDR should work fine with attached DASD. I just tried it on the latest version and some older levels and they all worked fine with a 3390-3 with both the source disk and target disk attached. I have the latest Pipelines runtime module, if that makes a difference. PIPEDDR doesn't really do anything differently for attached DASD, it just passes the virtual address you specified to the trackread or trackwrite stage of Pipelines. I updated both systems PIPEDDR from 1.4.10 to 1.5.12 and the Pipelines runtime from 1.0111 to 1.0112. pipe query PIPINX086I CMS/TSO Pipelines, 5654-030/5655-A17 1.0112 (Version.Release/Mod) - Generated 3 Dec 2010 at 11:10:08. It still fails the same way (but there are now progress messages on the sending side): q 6607 DASD 6607 ATTACHED TO BNIELSEN 6607 R/W VZ6607 Ready; T=0.01/0.01 13:36:07 pipeddr restore * 6607 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6607. Sending user is BNIELSEN at VMP2 Receiving disk BNIELSEN 9D5E from 172.16.64.45 PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.02 13:39:17 - q 9d5e DASD 9D5E ATTACHED TO BNIELSEN 9D5E R/W SYSDRL Ready; T=0.01/0.01 13:32:49 pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 PIPTCQ1015E ERRNO 32: EPIPE. PIPMSG004I ... Issued from stage 9 of pipeline 1 name ipread. PIPMSG001I ... Running tcpclient 172.16.64.44 11000 linger 10 reuseaddr U. Cylinder 133 of 3339 completed (3%) Cylinder 266 of 3339 completed (7%) Cylinder 400 of 3339 completed (11%) Cylinder 533 of 3339 completed (15%) Cylinder 666 of 3339 completed (19%) Cylinder 800 of 3339 completed (23%) Cylinder 933 of 3339 completed (27%) Cylinder 1066 of 3339 completed (31%) Cylinder 1200 of 3339 completed (35%) Cylinder 1333 of 3339 completed (39%) Cylinder 1466 of 3339 completed (43%) Cylinder 1600 of 3339 completed (47%) Cylinder 1733 of 3339 completed (51%) Cylinder 1866 of 3339 completed (55%) Cylinder 2000 of 3339 completed (59%) Cylinder 2133 of 3339 completed (63%) Cylinder 2266 of 3339 completed (67%) Cylinder 2400 of 3339 completed (71%) Cylinder 2533 of 3339 completed (75%) Cylinder 2666 of 3339 completed (79%) Cylinder 2800 of 3339 completed (83%) Cylinder 2933 of 3339 completed (87%) Cylinder 3066 of 3339 completed (91%) Cylinder 3200 of 3339 completed (95%) Cylinder of 3339 completed (99%) Dump failed. Ready(01015); T=4.77/6.30 13:37:11 I verified that both DASD volumes are the same size (Mod-3's w/3339 cyls) : --- q dasd details 6607 6607 CUTYPE = 2105-E8, DEVTYPE = 3390-0A, VOLSER = VZ6607, CYLS = 3339 CACHE DETAILS: CACHE NVS CFW DFW PINNED CONCOPY -SUBSYSTEM YY Y -N N -DEVICE Y- - YN N DEVICE DETAILS: CCA = 07, DDC = -- DUPLEX DETAILS: -- PAV DETAILS: BASE VOLUME WITH 01 ALIAS VOLUMES CU DETAILS: SSID = 6600, CUNUM = 6600 Ready; T=0.01/0.01 13:48:14 --- q dasd details 9d5e 9D5E CUTYPE = 2105-E8, DEVTYPE = 3390-0A, VOLSER = SYSDRL, CYLS = 3339 CACHE DETAILS: CACHE NVS CFW DFW PINNED CONCOPY -SUBSYSTEM YY Y -N N -DEVICE Y- - YN N DEVICE DETAILS: CCA = 5E, DDC = -- DUPLEX DETAILS: -- CU DETAILS: SSID = 450D, CUNUM = 9D00 Ready; T=0.01/0.01 13:44:11 *** Are there any options that might help? Brian Nielsen
Re: PIPEDDR and attached DASD
On Tue, Apr 5, 2011 at 10:07 PM, Brian Nielsen bniel...@sco.idaho.gov wrote: PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.02 13:39:17 Sounds like PIPEDDR is not properly handling the termination of the TCP/IP connection (like the sender going AWOL while the last piece of data is still in transit). If the pipe leaks, subtle timing changes may get your feet wet. I never looked at what PIPEDDR does for flow control, but I do recall that I had to master similar things when I did mine... | Rob
Re: PIPEDDR and attached DASD
Additional note: After this failed transfer the receiving DASD has a label of SCRTCH. I would expect it to be SYSDRL after cyl 0 is transfere d. det 6607 DASD 6607 DETACHED Ready; T=0.01/0.01 14:15:13 q 6607 DASD 6607 SCRTCH Ready; T=0.01/0.01 14:15:15 Brian Nielsen On Tue, 5 Apr 2011 15:07:31 -0500, Brian Nielsen bniel...@sco.idaho.gov wrote: On Tue, 5 Apr 2011 13:55:57 -0400, Bruce Hayden bjhay...@gmail.com wrote: PIPEDDR should work fine with attached DASD. I just tried it on the latest version and some older levels and they all worked fine with a 3390-3 with both the source disk and target disk attached. I have the latest Pipelines runtime module, if that makes a difference. PIPEDDR doesn't really do anything differently for attached DASD, it just passes the virtual address you specified to the trackread or trackwrite stage of Pipelines. I updated both systems PIPEDDR from 1.4.10 to 1.5.12 and the Pipelines runtime from 1.0111 to 1.0112. pipe query PIPINX086I CMS/TSO Pipelines, 5654-030/5655-A17 1.0112 (Version.Release/Mod) - Generated 3 Dec 2010 at 11:10:08. It still fails the same way (but there are now progress messages on the sending side): q 6607 DASD 6607 ATTACHED TO BNIELSEN 6607 R/W VZ6607 Ready; T=0.01/0.01 13:36:07 pipeddr restore * 6607 11000 (listen noprompt Connecting to TCP/IP. Enter PIPMOD STOP to terminate. Waiting for connection on port 11000 to restore BNIELSEN 6607. Sending user is BNIELSEN at VMP2 Receiving disk BNIELSEN 9D5E from 172.16.64.45 PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.02 13:39:17 - q 9d5e DASD 9D5E ATTACHED TO BNIELSEN 9D5E R/W SYSDRL Ready; T=0.01/0.01 13:32:49 pipeddr dump * 9d5e 172.16.64.44 11000 Dumping disk BNIELSEN 9D5E to 172.16.64.44 PIPTCQ1015E ERRNO 32: EPIPE. PIPMSG004I ... Issued from stage 9 of pipeline 1 name ipread. PIPMSG001I ... Running tcpclient 172.16.64.44 11000 linger 10 reuseaddr U. Cylinder 133 of 3339 completed (3%) Cylinder 266 of 3339 completed (7%) Cylinder 400 of 3339 completed (11%) Cylinder 533 of 3339 completed (15%) Cylinder 666 of 3339 completed (19%) Cylinder 800 of 3339 completed (23%) Cylinder 933 of 3339 completed (27%) Cylinder 1066 of 3339 completed (31%) Cylinder 1200 of 3339 completed (35%) Cylinder 1333 of 3339 completed (39%) Cylinder 1466 of 3339 completed (43%) Cylinder 1600 of 3339 completed (47%) Cylinder 1733 of 3339 completed (51%) Cylinder 1866 of 3339 completed (55%) Cylinder 2000 of 3339 completed (59%) Cylinder 2133 of 3339 completed (63%) Cylinder 2266 of 3339 completed (67%) Cylinder 2400 of 3339 completed (71%) Cylinder 2533 of 3339 completed (75%) Cylinder 2666 of 3339 completed (79%) Cylinder 2800 of 3339 completed (83%) Cylinder 2933 of 3339 completed (87%) Cylinder 3066 of 3339 completed (91%) Cylinder 3200 of 3339 completed (95%) Cylinder of 3339 completed (99%) Dump failed. Ready(01015); T=4.77/6.30 13:37:11 I verified that both DASD volumes are the same size (Mod-3's w/3339 cyls ): --- q dasd details 6607 6607 CUTYPE = 2105-E8, DEVTYPE = 3390-0A, VOLSER = VZ6607, CYLS = 3339 CACHE DETAILS: CACHE NVS CFW DFW PINNED CONCOPY -SUBSYSTEM YY Y -N N -DEVICE Y- - YN N DEVICE DETAILS: CCA = 07, DDC = -- DUPLEX DETAILS: -- PAV DETAILS: BASE VOLUME WITH 01 ALIAS VOLUMES CU DETAILS: SSID = 6600, CUNUM = 6600 Ready; T=0.01/0.01 13:48:14 --- q dasd details 9d5e 9D5E CUTYPE = 2105-E8, DEVTYPE = 3390-0A, VOLSER = SYSDRL, CYLS = 3339 CACHE DETAILS: CACHE NVS CFW DFW PINNED CONCOPY -SUBSYSTEM YY Y -N N -DEVICE Y- - YN N DEVICE DETAILS: CCA = 5E, DDC = -- DUPLEX DETAILS: -- CU DETAILS: SSID = 450D, CUNUM = 9D00 Ready; T=0.01/0.01 13:44:11 *** Are there any options that might help? Brian Nielsen
Re: PIPEDDR and attached DASD
On Tue, 5 Apr 2011 22:15:31 +0200, Rob van der Heij rvdh...@gmail.com wrote: On Tue, Apr 5, 2011 at 10:07 PM, Brian Nielsen bniel...@sco.idaho.gov wrote: PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.02 13:39:17 Sounds like PIPEDDR is not properly handling the termination of the TCP/IP connection (like the sender going AWOL while the last piece of data is still in transit). If the pipe leaks, subtle timing changes may get your feet wet. I never looked at what PIPEDDR does for flow control, but I do recall that I had to master similar things when I did mine... Perhaps, but it's curious that it works for a full pack MDISK but not for an attached DASD. Could it be a VM service level related issue?? q cplevel z/VM Version 5 Release 4.0, service level 0802 (64-bit) Generated at 02/19/09 10:50:42 MDT IPL at 02/28/09 11:15:51 MDT Ready; T=0.01/0.01 14:22:24 netstat VM TCP/IP Netstat Level 540 TCP/IP Server Name: TCPIP Both sides are at the same level, and yes, the last time my main VM was IPL'd was over 2 years ago. Brian Nielsen
Re: PIPEDDR and attached DASD
I don't know why the mdisk works and the attached disk doesn't. If the label is still SCRTCH, then nothing was written on the disk. That seems like the TCP/IP connection wasn't established correctly. We should take this offline to work on it further. On Tue, Apr 5, 2011 at 4:27 PM, Brian Nielsen bniel...@sco.idaho.gov wrote: On Tue, 5 Apr 2011 22:15:31 +0200, Rob van der Heij rvdh...@gmail.com wrote: On Tue, Apr 5, 2011 at 10:07 PM, Brian Nielsen bniel...@sco.idaho.gov wrote: PIPTCQ1015E ERRNO 54: ECONNRESET. PIPMSG004I ... Issued from stage 3 of pipeline 3 name iprestore. PIPMSG001I ... Running tcpdata. PIPUPK072E Last record not complete. PIPMSG003I ... Issued from stage 2 of pipeline 1. PIPMSG001I ... Running unpack. Data restore failed. Ready(01015); T=0.01/0.02 13:39:17 Sounds like PIPEDDR is not properly handling the termination of the TCP/IP connection (like the sender going AWOL while the last piece of data is still in transit). If the pipe leaks, subtle timing changes may get your feet wet. I never looked at what PIPEDDR does for flow control, but I do recall that I had to master similar things when I did mine... Perhaps, but it's curious that it works for a full pack MDISK but not for an attached DASD. Could it be a VM service level related issue?? q cplevel z/VM Version 5 Release 4.0, service level 0802 (64-bit) Generated at 02/19/09 10:50:42 MDT IPL at 02/28/09 11:15:51 MDT Ready; T=0.01/0.01 14:22:24 netstat VM TCP/IP Netstat Level 540 TCP/IP Server Name: TCPIP Both sides are at the same level, and yes, the last time my main VM was IPL'd was over 2 years ago. Brian Nielsen -- Bruce Hayden z/VM and Linux on System z ATS IBM, Endicott, NY
Re: RESTORE DUMP WITH PIPEDDR
On Tue, Jan 26, 2010 at 6:56 AM, Kris Buelens kris.buel...@gmail.com wrote: It depends what you need to restore. DFSMS/VM's COPY command can copy CMS formatted minidisks to unlike disks, so also from 3390 to FBA. The VM spool for example need to be rebuilt and transferred using tape; MAINT 190, with its CMS nucleus, needs to be rebuilt too, sameway for SAPL. But DFSMS does not know how to copy the Linux payload from one disk to another one. The fact that it can do CMS disks is of little help here. For the z/VM data, I would say that not even a very experienced VM systems programmer would try this. It's easier to re-install z/VM 2nd level on FCP devices. With both systems available at the same time, it's easy to pick up any missing things by file-level copy. Whether you should fiddle with Linux data depends on the amount of data, your skills and the ability to re-install Linux. The simple answer is that you can't copy the data like this, other than backup and restore from logical (file level) backup. It may be a good opportunity to test that process and get a reality check... Rob
Re: RESTORE DUMP WITH PIPEDDR
Hello Continuous with this subject, trying to find the solution most direct and simple to realise it. I want that dump and restore everything from VM and not to involve in this to the tools linux . And I have another question: With DFSMS for z/vm it's possible this??? Dump and restore between these technologies of storage DUMP IN ECKD -- RESTORE IN FCP. Thanks four yuor help again. ATTE Victor Hugo 2009/12/3 Rob van der Heij rvdh...@gmail.com On Thu, Dec 3, 2009 at 9:03 AM, Victor Ochoa Avila vhoa...@gmail.com wrote: Thanks Daveit is a pleasure to know again of you We have new storage (IBM) cage with discs FCP, 5 TB and I have configured and ready for user, but my masters clones live in discs with technology ECKD… for that reason it was looking for a form to be able to restore them in discs FCP and not to install these clones again. Ah! If you still have the original disk you don't need PIPEDDR but you can just the pure thing without buffering or whatever. But it also leads to a next question... we don't have a pipeline stage to write to an FBA device when it is not accessed by CMS (thus CMS format). My ccw stage got a bit dusty since John did the track* stages... Something to try would be - define an EDEVICE for the LUN - define a 2nd (larger) EDEVICE and format/reserve with CMS - access the 2nd and write the blocks in the reserved part of the disk - use DDR to copy (with reorder) from the reserved part of the 2nd to the 1st disk By far easier would be to give a Linux guest both the original master and a LUN, and then use Linux tools to copy the file system. Once you have a copy on a LUN, you could still decide to use DDR to make copies (if you don't need Linux to modify the contents after copying). But it's almost Friday ;-) Rob -- Victor Hugo Ochoa Avila z/OS z/VM systems programmer Mexico, City.
Re: RESTORE DUMP WITH PIPEDDR
It depends what you need to restore. DFSMS/VM's COPY command can copy CMS formatted minidisks to unlike disks, so also from 3390 to FBA. The VM spool for example need to be rebuilt and transferred using tape; MAINT 190, with its CMS nucleus, needs to be rebuilt too, sameway for SAPL. 2010/1/26 Victor Ochoa Avila vhoa...@gmail.com Hello Continuous with this subject, trying to find the solution most direct and simple to realise it. I want that dump and restore everything from VM and not to involve in this to the tools linux . And I have another question: With DFSMS for z/vm it's possible this??? Dump and restore between these technologies of storage DUMP IN ECKD -- RESTORE IN FCP. Thanks four yuor help again. ATTE Victor Hugo 2009/12/3 Rob van der Heij rvdh...@gmail.com On Thu, Dec 3, 2009 at 9:03 AM, Victor Ochoa Avila vhoa...@gmail.com wrote: Thanks Daveit is a pleasure to know again of you We have new storage (IBM) cage with discs FCP, 5 TB and I have configured and ready for user, but my masters clones live in discs with technology ECKD… for that reason it was looking for a form to be able to restore them in discs FCP and not to install these clones again. Ah! If you still have the original disk you don't need PIPEDDR but you can just the pure thing without buffering or whatever. But it also leads to a next question... we don't have a pipeline stage to write to an FBA device when it is not accessed by CMS (thus CMS format). My ccw stage got a bit dusty since John did the track* stages... Something to try would be - define an EDEVICE for the LUN - define a 2nd (larger) EDEVICE and format/reserve with CMS - access the 2nd and write the blocks in the reserved part of the disk - use DDR to copy (with reorder) from the reserved part of the 2nd to the 1st disk By far easier would be to give a Linux guest both the original master and a LUN, and then use Linux tools to copy the file system. Once you have a copy on a LUN, you could still decide to use DDR to make copies (if you don't need Linux to modify the contents after copying). But it's almost Friday ;-) Rob -- Victor Hugo Ochoa Avila z/OS z/VM systems programmer Mexico, City. -- Kris Buelens, IBM Belgium, VM customer support
Re: RESTORE DUMP WITH PIPEDDR
Thanks Daveit is a pleasure to know again of you We have new storage (IBM) cage with discs FCP, 5 TB and I have configured and ready for user, but my masters clones live in discs with technology ECKD… for that reason it was looking for a form to be able to restore them in discs FCP and not to install these clones again. You have some idea of I can do how it Best Regards ATTE Victor Hugo Ochoa Avila BBVA CCR America Mexico City. 2009/12/2 Dave Jones d...@sys1.vsoft-software.com Hello, Victor. I'm afraid not...the PIPEDDR process produces a track by track dump of the (E)CKD DASD devices...it can only be restored to a like (E)CKD device. There is no way that I know of to restore such a file to a SCSI-type FCP device directly. DJ - Original Message - From: =?ISO-8859-1?Q?Victor_Hugo_Ochoa?= vhoa...@gmail.com To: IBMVM@LISTSERV.UARK.EDU Subject: RESTORE DUMP WITH PIPEDDR Date: Wed, 2 Dec 2009 19:17:24 -0600 somebody knows this: It's possible to restore backup obtained with PIPEDDR in discs technology CKD(origin) to a destiny with discs technology FCP(target)? Thanks ATTE Victor Hugo Ochoa Avila BBVA CCR America, Mexico City -- Victor Hugo Ochoa Avila z/OS z/VM systems programmer Mexico, City.
Re: RESTORE DUMP WITH PIPEDDR
On Thu, Dec 3, 2009 at 9:03 AM, Victor Ochoa Avila vhoa...@gmail.com wrote: Thanks Daveit is a pleasure to know again of you We have new storage (IBM) cage with discs FCP, 5 TB and I have configured and ready for user, but my masters clones live in discs with technology ECKD… for that reason it was looking for a form to be able to restore them in discs FCP and not to install these clones again. Ah! If you still have the original disk you don't need PIPEDDR but you can just the pure thing without buffering or whatever. But it also leads to a next question... we don't have a pipeline stage to write to an FBA device when it is not accessed by CMS (thus CMS format). My ccw stage got a bit dusty since John did the track* stages... Something to try would be - define an EDEVICE for the LUN - define a 2nd (larger) EDEVICE and format/reserve with CMS - access the 2nd and write the blocks in the reserved part of the disk - use DDR to copy (with reorder) from the reserved part of the 2nd to the 1st disk By far easier would be to give a Linux guest both the original master and a LUN, and then use Linux tools to copy the file system. Once you have a copy on a LUN, you could still decide to use DDR to make copies (if you don't need Linux to modify the contents after copying). But it's almost Friday ;-) Rob
Re: RESTORE DUMP WITH PIPEDDR
No - you'll have to restore it to CKD and then copy it to FCP.. Scott On Wed, Dec 2, 2009 at 6:17 PM, =?ISO-8859-1?Q?Victor_Hugo_Ochoa?= vhoa.mx@ gmail.com wrote: somebody knows this: It's possible to restore backup obtained with PIPEDDR in discs technology CKD(origin) to a destiny with discs technology FCP(target)? Thanks ATTE Victor Hugo Ochoa Avila BBVA CCR America, Mexico City
Re: RESTORE DUMP WITH PIPEDDR
No. DDR (of any flavor) is same-to-same type. You might get away with FBA to FCP, but not CKD. On 12/2/09 8:17 PM, Victor Hugo Ochoa vhoa...@gmail.com wrote: somebody knows this: It's possible to restore backup obtained with PIPEDDR in discs technology CKD(origin) to a destiny with discs technology FCP(target)? Thanks ATTE Victor Hugo Ochoa Avila BBVA CCR America, Mexico City
Re: RESTORE DUMP WITH PIPEDDR
Hello, Victor. I'm afraid not...the PIPEDDR process produces a track by track dump of the (E)CKD DASD devices...it can only be restored to a like (E)CKD device. There is no way that I know of to restore such a file to a SCSI-type FCP device directly. DJ - Original Message - From: =?ISO-8859-1?Q?Victor_Hugo_Ochoa?= vhoa...@gmail.com To: IBMVM@LISTSERV.UARK.EDU Subject: RESTORE DUMP WITH PIPEDDR Date: Wed, 2 Dec 2009 19:17:24 -0600 somebody knows this: It's possible to restore backup obtained with PIPEDDR in discs technology CKD(origin) to a destiny with discs technology FCP(target)? Thanks ATTE Victor Hugo Ochoa Avila BBVA CCR America, Mexico City
Re: RESTORE DUMP WITH PIPEDDR
On Thu, Dec 3, 2009 at 2:17 AM, =?ISO-8859-1?Q?Victor_Hugo_Ochoa?= vhoa...@gmail.com wrote: It's possible to restore backup obtained with PIPEDDR in discs technology CKD(origin) to a destiny with discs technology FCP(target)? If the orignal disk format was simple fixed block format (eg 4K like in CMS or Linux) it should not be too hard to get the payload out of the output of trackreadYou may find the diagram in http://www.rvdheij.nl/Presentations/2008-V56.pdf handy (pg 17) Goes like this: ... | trackdeblock | locate 17 | ckddeblock | locate 9 I don't know whether Bruce did something else with the data, you may need to unsquish it or unpack the data. What you get there are the blocks as they were stored on disk. When you want to copy them to another FB device you may have to shift a few blocks offset, but that should be it. For Linux there's also some stuff that depends on the size of the device. If it was a CMS disk, you would need to learn how to walk the FST pointers if you want to pull individual files out of it. The biggest challenge might be to process the input stream only once, but if you're willing to store it as a FB file for a moment, it is SMoP Whether you want to do that depends on how desperate you are. I've used it to harvest the unused blocks from a CMS disk and reconstruct the files that were erased from it... PS We might want to move the discussion to CMSPIP-L since the most significant contributor there does not subscribe to this list.. Sir Rob the Plumber
PIPEDDR
Where can I find the error code descriptions from PIPEDDR?
Re: PIPEDDR
You could get an error from Pipelines, which would be in the Pipelines documentation, or an error from the FTP stages. The only documentation about the ftp stages is part of the DRPC download package, and it doesn't list any possible errors, just the syntax for using them. As I stated in the help file for PIPEDDR: The FTP stages are not part of PIPEDDR, are not officially supported by IBM, and source code is not provided. Any errors may not be fixable. On Tue, Oct 20, 2009 at 3:26 PM, Bob Henry bob.he...@sungardhe.com wrote: Where can I find the error code descriptions from PIPEDDR? -- Bruce Hayden Linux on System z Advanced Technical Support IBM, Endicott, NY
Re: FW: PIPEDDR TERSE and other stuff
When I've had a need to move 3390 volume images to another data center, I've used CMSDDR to dump them to files, then transferred the files via either RSCS or ISFC. CMSDDR does some sort of packing on the files it creates. Empty 3390-3's result in far smaller output files than full ones do. Dennis I don't have a girlfriend. I just know a girl who would get really mad if she heard me say that. -- Mitch Hedberg From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] On Behalf Of Huegel, Thomas Sent: Thursday, July 12, 2007 06:40 To: IBMVM@LISTSERV.UARK.EDU Subject: [IBMVM] FW: PIPEDDR TERSE and other stuff Well maybe if I could get PIPEDDR to work to the remote IP (my network guys and I disagree a bit here, my problem not yours) then maybe I'd shutup about TERSE. I have a need to do exacatly what you mentioned 'transfer complete 3390s from one data centre to another'. A rare but PITA occurance. I was under the, maybe false, impresion that TERSE would do a better job of compression than PACK. It might simply be a 'the grass is greener' syndrom. I'll probably be happy once I can get it working... My thanks to the author for writing it and placing it on the download page... You wouldn't be willing to share your READER/URO mods would you? please please.. -Original Message- From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] Behalf Of Graeme Moss Sent: Wednesday, July 11, 2007 4:57 PM To: IBMVM@LISTSERV.UARK.EDU Subject: PIPEDDR TERSE and other stuff Greetings Listers, PIPEDDR can use TERSE or PACK stages for compression or use no compression.. It only fails if TERSE option is specified and the stage is not available. It is written so it does not rely on TERSE I have used PIPEDDR with PACK option to transfer complete 3390s from one data centre to another and it works well. It saved the delay of waiting for a courier to take a cartridge. Just an aside, I thought I would update the code so it used standard PIPES but when I did that TCPIP packets were lost. Maybe TCPCLIENT and TCPLISTEN have different code or maybe I screwed it up or maybe it was timing. Another aside. Thanks to the author Bruce Hayden publishing the code I was able to use PIPEDDR as a model and instead of using TRACKREAD and TRACKWRITE to read and write dasd I used READER and URO to read and write spool making a very simple NJETCP link. Cheers Graeme __ ella for Spam Control has removed VSE-List messages and set aside VM-List for me You can use it too - and it's FREE! http://www.ellaforspam.com __ ella for Spam Control has removed VSE-List messages and set aside VM-List for me You can use it too - and it's FREE! http://www.ellaforspam.com
FW: PIPEDDR TERSE and other stuff
Well maybe if I could get PIPEDDR to work to the remote IP (my network guys and I disagree a bit here, my problem not yours) then maybe I'd shutup about TERSE. I have a need to do exacatly what you mentioned 'transfer complete 3390s from one data centre to another'. A rare but PITA occurance. I was under the, maybe false, impresion that TERSE would do a better job of compression than PACK. It might simply be a 'the grass is greener' syndrom. I'll probably be happy once I can get it working... My thanks to the author for writing it and placing it on the download page... You wouldn't be willing to share your READER/URO mods would you? please please.. -Original Message- From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] Behalf Of Graeme Moss Sent: Wednesday, July 11, 2007 4:57 PM To: IBMVM@LISTSERV.UARK.EDU Subject: PIPEDDR TERSE and other stuff Greetings Listers, PIPEDDR can use TERSE or PACK stages for compression or use no compression.. It only fails if TERSE option is specified and the stage is not available. It is written so it does not rely on TERSE I have used PIPEDDR with PACK option to transfer complete 3390s from one data centre to another and it works well. It saved the delay of waiting for a courier to take a cartridge. Just an aside, I thought I would update the code so it used standard PIPES but when I did that TCPIP packets were lost. Maybe TCPCLIENT and TCPLISTEN have different code or maybe I screwed it up or maybe it was timing. Another aside. Thanks to the author Bruce Hayden publishing the code I was able to use PIPEDDR as a model and instead of using TRACKREAD and TRACKWRITE to read and write dasd I used READER and URO to read and write spool making a very simple NJETCP link. Cheers Graeme __ ella for Spam Control has removed VSE-List messages and set aside VM-List for me You can use it too - and it's FREE! http://www.ellaforspam.com __ ella for Spam Control has removed VSE-List messages and set aside VM-List for me You can use it too - and it's FREE! http://www.ellaforspam.com
Re: FW: PIPEDDR TERSE and other stuff
When I first started getting electronic PTFs (VMFPLCD stuff) that needed DETERSEing, I DETERSEd a few and then COPYFILE (PACKed them. The original TERSEd files were smaller, insignificant when dealing with a couple of blocks for a CMS file, but I think it would be significant for even a 3390-m3 and even more for the 3390-m9's that I have to deal with. I am not sure if PACK or even TERSE in a pipeline is appropriate. PACK is a compatibility feature (PACK option to COPYFILE has been there since I started in June 1976). TERSE may also be a file-level compression that restructures all of the data into a new format (please no flames but if I am wrong here, a correction would be appreciated since I don't have TERSE to test with). What PIPELINES needs (and this has been mentioned on the CMSPIP-L) is a record by record compression stage or two or three dependi ng on data content. /Tom Kern /301-903-2211 On Thu, 12 Jul 2007 08:39:52 -0500, Huegel, Thomas [EMAIL PROTECTED] wr ote: Well maybe if I could get PIPEDDR to work to the remote IP (my network g uys and I disagree a bit here, my problem not yours) then maybe I'd shutup a bout TERSE. I have a need to do exacatly what you mentioned 'transfer complet e 3390s from one data centre to another'. A rare but PITA occurance. I was under the, maybe false, impresion that TERSE would do a better job of compression than PACK. It might simply be a 'the grass is greener' syndr om. I'll probably be happy once I can get it working... My thanks to the author for writing it and placing it on the download page... You wouldn't be willing to share your READER/URO mods would you? please please..
Re: FW: PIPEDDR TERSE and other stuff
COPYFILE (PACK is very simple, I never looked in the actual details but it replaces multiple occurances of the same character, inexpensive but not lots of gains. TERSE is much like ZIP and VMARC. The FCOPY tool on the download lib has a PACK option too, but that works like TERSE, not like COPYFILE. A reason why COPYFILE (PACK is used to transfer files from VM into the PC world, is that a safe binary download/upload is only possible for files with a FIXED recordlength. COPYFILE (PACK makes every file F1024 (TERSE too and VMARC F80; FCOPY creates a RECFM V file) 2007/7/12, Thomas Kern [EMAIL PROTECTED]: When I first started getting electronic PTFs (VMFPLCD stuff) that needed DETERSEing, I DETERSEd a few and then COPYFILE (PACKed them. The original TERSEd files were smaller, insignificant when dealing with a couple of blocks for a CMS file, but I think it would be significant for even a 3390-m3 and even more for the 3390-m9's that I have to deal with. I am not sure if PACK or even TERSE in a pipeline is appropriate. PACK is a compatibility feature (PACK option to COPYFILE has been there since I started in June 1976). TERSE may also be a file-level compression that restructures all of the data into a new format (please no flames but if I am wrong here, a correction would be appreciated since I don't have TERSE to test with). What PIPELINES needs (and this has been mentioned on the CMSPIP-L) is a record by record compression stage or two or three dependi ng on data content. /Tom Kern -- Kris Buelens, IBM Belgium, VM customer support
Re: FW: PIPEDDR TERSE and other stuff
Kris Buelens wrote: COPYFILE (PACK is very simple, I never looked in the actual details but it replaces multiple occurances of the same character, inexpensive but not lots of gains. TERSE is much like ZIP and VMARC. The FCOPY tool on the download lib has a PACK option too, but that works like TERSE, not like COPYFILE. A reason why COPYFILE (PACK is used to transfer files from VM into the PC world, is that a safe binary download/upload is only possible for files with a FIXED recordlength. COPYFILE (PACK makes every file F1024 (TERSE too and VMARC F80; FCOPY creates a RECFM V file) From the 1985 internal announcement of TERSE: -- TERSE is a program which compresses data. It is based on a new algorithm developed by Victor Miller and Mark Wegman of IBM Research at Yorktown (Variations on a theme by Ziv and Lempel - Research RC 10630). The compression algorithm is adaptive, and does quite well on a variety of file types, usually much better than HUFF or HUFFMOVE. --- I could likely dig up a copy of RC10630 if anybody is really that interested. Ray
Re: FW: PIPEDDR TERSE and other stuff
On 7/12/07, Thomas Kern [EMAIL PROTECTED] wrote: TERSEd files were smaller, insignificant when dealing with a couple of blocks for a CMS file, but I think it would be significant for even a 3390-m3 and even more for the 3390-m9's that I have to deal with. Depends on what your criteria are. Do understand that terse takes quite some CPU cycles and depending on the resources available, that may become the bottleneck. When you transfer large chunks of data over long haul connections there will be latency issues that get important (and it may be more attractive to run multiple streams). And you may also find that your mainframe connection was improperly defined in the switch with an asymmetric bandwidth capping... Long before Bruce did his PIPEDDR, I wrote my IP-DDR package (after all, I got the track* stages for my birthday). It's initial raison d'etre was to keep a copy of the RACF database on a DR site. Since our RACF database was defined rather big, only a small number of tracks would be changed in a day. So my IP-DDR used a per-track checksum that was exchanged between both ends to decide whether a track should be sent at all. The per-track approach also allows for doing multiple streams in parallel and use a connection only for a small amount of data (to fool the bandwidth shaping). The checksum thing turned out to be very helpful also to restart a transfer after something interrupted it. Since it's not uncommon to have disks partially empty when you need to transfer them, the Piper created tracksquish to have cheap compression of empty tracks. I also used that a lot to keep a copy of a Linux disk with empty file system. I recently talked to John about the value of having zlib stages, but it did not happen yet... Rob
Re: FW: PIPEDDR TERSE and other stuff
At this point I am not concerned with CPU cycles spent on compression or on network bandwidth. I am trying to use PIPEDDR (with my own kludge for TAP E output). The CPU cycles for compression would not be as much of a problem as the CPU cycles I want to spent on data ENCRYPTION prior to writing out to tape. If my customer ever gets a backup data center of their own, I will look into PIPEDDR (with encryption) for network transfer of backups. /Tom Kern /301-903-2211 On Thu, 12 Jul 2007 17:45:25 +0200, Rob van der Heij [EMAIL PROTECTED] wrote: Depends on what your criteria are. Do understand that terse takes quite some CPU cycles and depending on the resources available, that may become the bottleneck. When you transfer large chunks of data over long haul connections there will be latency issues that get important (and it may be more attractive to run multiple streams). And you may also find that your mainframe connection was improperly defined in the switch with an asymmetric bandwidth capping... Long before Bruce did his PIPEDDR, I wrote my IP-DDR package (after all, I got the track* stages for my birthday). It's initial raison d'etre was to keep a copy of the RACF database on a DR site. Since our RACF database was defined rather big, only a small number of tracks would be changed in a day. So my IP-DDR used a per-track checksum that was exchanged between both ends to decide whether a track should be sent at all. The per-track approach also allows for doing multiple streams in parallel and use a connection only for a small amount of data (to fool the bandwidth shaping). The checksum thing turned out to be very helpful also to restart a transfer after something interrupted it. Since it's not uncommon to have disks partially empty when you need to transfer them, the Piper created tracksquish to have cheap compression of empty tracks. I also used that a lot to keep a copy of a Linux disk with empty file system. I recently talked to John about the value of having zlib stages, but it did not happen yet... Rob
PIPEDDR
On the VM download page there is a tool 'PIPEDDR'. In the description of this tool it mentions that it can use a 'TERSE' pipestage, then goes on to mention that this is only available on internal IBM systems .. duh !! Anyone know how one could get a hold of such an internal pipestage? _ ella for Spam Control has removed 11825 VSE-List messages and set aside 10492 VM-List for me You can use it too - and it's FREE! www.ellaforspam.com http://www.ellaforspam.com
Re: PIPEDDR
The best way to get the TERSE stage is to become an IBMer. It is one of the internal only stages that never make it to the customers. /Tom Kern /301-903-2211 --- Huegel, Thomas [EMAIL PROTECTED] wrote: On the VM download page there is a tool 'PIPEDDR'. In the description of this tool it mentions that it can use a 'TERSE' pipestage, then goes on to mention that this is only available on internal IBM systems .. duh !! Anyone know how one could get a hold of such an internal pipestage? _ ella for Spam Control has removed 11825 VSE-List messages and set aside 10492 VM-List for me You can use it too - and it's FREE! www.ellaforspam.com http://www.ellaforspam.com Shape Yahoo! in your own image. Join our Network Research Panel today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7
Re: PIPEDDR
The download page is a gold mine of unreleased/unsupported tools that just simply make life easier. I have a lot of stuff from there, and often modify it for my specific use. Understanding completly that there is no gaurentee and everything is 'use at your own risk' I have no qualms about 'giving it a try' and take responsibility for the consequences. I know of shops that forbid the use of 'unsupported' software ie from the download page,I never quite understood that philosophy, but that is thier decision. The point is, if it is good enough for IBM isn't it good enough for thier customers? Now we(I) need to convince them to put TERSE on the download. -Original Message- From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] Behalf Of Thomas Kern Sent: Wednesday, July 11, 2007 9:05 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: PIPEDDR The best way to get the TERSE stage is to become an IBMer. It is one of the internal only stages that never make it to the customers. /Tom Kern /301-903-2211 --- Huegel, Thomas [EMAIL PROTECTED] wrote: On the VM download page there is a tool 'PIPEDDR'. In the description of this tool it mentions that it can use a 'TERSE' pipestage, then goes on to mention that this is only available on internal IBM systems .. duh !! Anyone know how one could get a hold of such an internal pipestage? _ ella for Spam Control has removed 11825 VSE-List messages and set aside 10492 VM-List for me You can use it too - and it's FREE! www.ellaforspam.com http://www.ellaforspam.com Shape Yahoo! in your own image. Join our Network Research Panel today! http://surveylink.yahoo.com/gmrs/yahoo_panel_invite.asp?a=7 __ ella for Spam Control has removed VSE-List messages and set aside VM-List for me You can use it too - and it's FREE! http://www.ellaforspam.com
Re: PIPEDDR
On 7/11/07, Huegel, Thomas [EMAIL PROTECTED] wrote: The point is, if it is good enough for IBM isn't it good enough for thier customers? I'm pretty sure you don't want to have even a fraction of what IBM thinks is good for their own employees :-) Now we(I) need to convince them to put TERSE on the download. With the CMS Pipelines Runtimes Library (which you also need for the PIPEDDR package) you have already more than what IBM thinks is good... But 'terse' is not part of that. The Piper normally has a good reason not to put something in the runtime library, but I don't know the motivation in this particular case. You could ask on CMSPIP-L if you have strong desires. Rob
Re: PIPEDDR
On Wed, 11 Jul 2007 08:44:56 -0500, Huegel, Thomas [EMAIL PROTECTED] wr ote: On the VM download page there is a tool 'PIPEDDR'. In the description of this tool it mentions that it can use a 'TERSE' pipestage, then goes on to mention that this is only available on internal IBM systems .. duh !! Anyone know how one could get a hold of such an internal pipestage? Well there's a DETERSE MODULE on the 5E5 disk but I wonder if TERSE has anything to do with the TRSMAIN package available for zOS systems which PACKs and UNPACKs data so that it is compressed (I only mentions this because its gets delivered in a dataset with TERSE in the name: PTFLCG.TERSE409.LOADLIB) Seb
Re: PIPEDDR
I have never been adverse to trying 'unsupported' code. I have even convinced some management to allow 'unsupported' code for our use and for some of our better customers. If you prowl through the downloads page long enough, you will find severa l items that sound usefull but on deeper inspection, find that they are dependent upon the secret workings of some unavailable program. IBM shoul d have required from the start that ALL items on the download page be compl ete and with source code included. Yeah, I know IBM would never have been abl e to go against the OCO is really good for everyone philosophy, but we can dream. Even with those problems, the downloads page is a source of some great programs (Mailit and Rxserver in particular) so don't give up on it. And please add your voice to the call for the TERSE and other stages and othe r programs that are hinted at. /Tom Kern /301-903-2211 On Wed, 11 Jul 2007 09:43:40 -0500, Huegel, Thomas [EMAIL PROTECTED] wr ote: The download page is a gold mine of unreleased/unsupported tools that just simply make life easier. I have a lot of stuff from there, and often mod ify it for my specific use. Understanding completly that there is no gaurentee and everything is 'us e at your own risk' I have no qualms about 'giving it a try' and take responsibility for the consequences. I know of shops that forbid the use of 'unsupported' software ie from th e download page,I never quite understood that philosophy, but that is thie r decision. The point is, if it is good enough for IBM isn't it good enough for thie r customers? Now we(I) need to convince them to put TERSE on the download.
Re: PIPEDDR
Yes, that is the stuff. IBM sends us TERSEd files and we have to DETERSE them. On that better system that has to send more dumps to IBM there is t he TRSMAIN program to help reduce the bandwidth for dump transfer. But I thi nk it would violate multiple IBM legalese documents for us to reverse engine er that program into a VM usable module or pipe stage. I can understand that some code MUST be kept secret, but if it is good enough for one system, why isn't it good enough for the other systems (Do es VSE have a TERSE program for its dumps?)? /Tom Kern /301-903-2211 On Wed, 11 Jul 2007 10:49:36 -0500, Sebastian Welton [EMAIL PROTECTED] wr ote: Well there's a DETERSE MODULE on the 5E5 disk but I wonder if TERSE has anything to do with the TRSMAIN package available for zOS systems which PACKs and UNPACKs data so that it is compressed (I only mentions this because its gets delivered in a dataset with TERSE in the name: PTFLCG.TERSE409.LOADLIB) Seb = ===
Re: PIPEDDR
On Wednesday, 07/11/2007 at 12:08 EDT, Thomas Kern [EMAIL PROTECTED] wrote: Yes, that is the stuff. IBM sends us TERSEd files and we have to DETERSE them. On that better system that has to send more dumps to IBM there is the TRSMAIN program to help reduce the bandwidth for dump transfer. But I think it would violate multiple IBM legalese documents for us to reverse engineer that program into a VM usable module or pipe stage. I can understand that some code MUST be kept secret, but if it is good enough for one system, why isn't it good enough for the other systems (Does VSE have a TERSE program for its dumps?)? We are investigating the issues surrounding dump compression and encryption for the VM environment. It is my hope, though, that the solution to these problems will be more generally useful. Of course, if you're aren't having trouble getting data to the Support Center, or aren't complaining (to the Support Center) if you do, then I guess we don't have a problem to solve, and can move on to something else, eh? ;-) Alan Altmark z/VM Development IBM Endicott
Re: PIPEDDR
On Jul 11, 2007, at 11:37 AM, Alan Altmark wrote: We are investigating the issues surrounding dump compression and encryption for the VM environment. It is my hope, though, that the solution to these problems will be more generally useful. Hunh. A transparent passthrough compression/encryption facility. Gee, that sounds like a good idea. It also sounds *suspiciously similar* to an extant IBM z/VM component. Admittedly, one that I think should be performed in a CMS app rather than by a detour through an install-it-yourself Linux guest, but, well, we're all aware of each others' views on that. Adam
Re: PIPEDDR
Complaining to the Support Center is sometimes very effective. I complained a couple of years ago about the requirement that dumps be sent in COPYFILE packed format. After I gave them the numbers for one of our dumps packed by COPYFILE vs. VMARC, and indicated that our development network apparently wasn't the fastest thing around, they agreed that much smaller is better. They started accepting VMARC packed files. Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] On Behalf Of Alan Altmark Sent: Wednesday, July 11, 2007 9:37 AM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: PIPEDDR On Wednesday, 07/11/2007 at 12:08 EDT, Thomas Kern [EMAIL PROTECTED] wrote: Yes, that is the stuff. IBM sends us TERSEd files and we have to DETERSE them. On that better system that has to send more dumps to IBM there is the TRSMAIN program to help reduce the bandwidth for dump transfer. But I think it would violate multiple IBM legalese documents for us to reverse engineer that program into a VM usable module or pipe stage. I can understand that some code MUST be kept secret, but if it is good enough for one system, why isn't it good enough for the other systems (Does VSE have a TERSE program for its dumps?)? We are investigating the issues surrounding dump compression and encryption for the VM environment. It is my hope, though, that the solution to these problems will be more generally useful. Of course, if you're aren't having trouble getting data to the Support Center, or aren't complaining (to the Support Center) if you do, then I guess we don't have a problem to solve, and can move on to something else, eh? ;-) Alan Altmark z/VM Development IBM Endicott
Re: PIPEDDR
If you are restricting your solution to only dump transfer to the Support Center then I am not interested. I have bigger problems protecting my customer's data and getting it available at a Disaster Recovery site. If you are looking to add compression and encryption to the basic VM toolkit, then I am very interested, but the Support Center is the wrong place to complain about that. /Tom Kern /301-903-2211 On Wed, 11 Jul 2007 12:37:24 -0400, Alan Altmark [EMAIL PROTECTED] wrote: We are investigating the issues surrounding dump compression and encryption for the VM environment. It is my hope, though, that the solution to these problems will be more generally useful. Of course, if you're aren't having trouble getting data to the Support Center, or aren't complaining (to the Support Center) if you do, then I guess we don't have a problem to solve, and can move on to something els e, eh? ;-) Alan Altmark z/VM Development IBM Endicott =
Re: PIPEDDR
On Jul 11, 2007, at 11:58 AM, Alan Altmark wrote: Down, boy! Down! [I brandish a rolled-up newspaper in your general direction] :-) There are issues surrounding compression and encryption that have nothing to do with secure network transmission provided by the SSL server. What to do if you want to encrypt a dump on a tape for archive purposes and you don't have an encrypting tape drive? Who said anything about *network* transmission? I don't really understand your question though: if your tape drive won't do it, then you gotta do the crypto in (possibly-coprocessor- assisted) software, right? So you just encrypt each block via your...well, encrypting pipeline is the best term I can quickly come up with...on its way to the drive. You just need some method of making sure that you don't shoeshine the drive by forcing it to wait for blocks, but that's a simple matter of adjusting buffer sizes (and if necessary buffering to disk) on the way. Sure, you need an additional buffer to hold the encrypted blocks (as well as the plaintext blocks) and you need the overhead of the encryption facility, but, hey, no such thing as a free lunch. Assuming you pick to standard (and correctly-implemented) ciphers, then whether you do the en- or decryption in hardware or software simply becomes a question of speed and price, not functionality. And if you just want your data compressed, well, you were going to compress it before you encrypted it anyway, right (to squeeze out as much entropy as possible), so you just pick a null cipher and the same pipelining mechanism, eh? I mean, yeah, sure, it's not really that quite easy (key management being the most glaring problem to me), but it does seem like there's a lot of common data-block-manipulation-strategy you probably could factor out. Until you said it, I hadn't given any thought to using the SSL server in new and creative ways to accomplish file encryption. Hmmm.. vey interesting. ;-) Oh. Well, crap. Can we rewind so I can *charge* you for that idea? Adam
Re: PIPEDDR
On Wednesday, 07/11/2007 at 12:43 EDT, Adam Thornton [EMAIL PROTECTED] wrote: Hunh. A transparent passthrough compression/encryption facility. Gee, that sounds like a good idea. It also sounds *suspiciously similar* to an extant IBM z/VM component. Admittedly, one that I think should be performed in a CMS app rather than by a detour through an install-it-yourself Linux guest, but, well, we're all aware of each others' views on that. Down, boy! Down! [I brandish a rolled-up newspaper in your general direction] :-) There are issues surrounding compression and encryption that have nothing to do with secure network transmission provided by the SSL server. What to do if you want to encrypt a dump on a tape for archive purposes and you don't have an encrypting tape drive? Until you said it, I hadn't given any thought to using the SSL server in new and creative ways to accomplish file encryption. Hmmm.. vey interesting. ;-) Alan Altmark z/VM Development IBM Endicott
Re: PIPEDDR
Tom, VSE *does* have a TERSE utility (at least since z/VSE 3.1). It can be used on library members that are not type DUMP or PHASE. See the z/VSE 3.1 System Utility Guide.
Re: PIPEDDR
And I would suspect that TPF also has a TERSE program. I would not be surprised if somewhere in IBM/Linux there is a TERSE executable. /Tom Kern /301-903-2211 On Wed, 11 Jul 2007 14:29:56 -0400, Hooker, Don - OIT [EMAIL PROTECTED] wrote: Tom, VSE *does* have a TERSE utility (at least since z/VSE 3.1). It can be used on library members that are not type DUMP or PHASE. See the z/VSE 3.1 System Utility Guide.
Re: PIPEDDR
Right, they have it all over the place, but are keeping the pipe stage for themselves. -Original Message- From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] Behalf Of Thomas Kern Sent: Wednesday, July 11, 2007 1:40 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: PIPEDDR And I would suspect that TPF also has a TERSE program. I would not be surprised if somewhere in IBM/Linux there is a TERSE executable. /Tom Kern /301-903-2211 On Wed, 11 Jul 2007 14:29:56 -0400, Hooker, Don - OIT [EMAIL PROTECTED] wrote: Tom, VSE *does* have a TERSE utility (at least since z/VSE 3.1). It can be used on library members that are not type DUMP or PHASE. See the z/VSE 3.1 System Utility Guide. __ ella for Spam Control has removed VSE-List messages and set aside VM-List for me You can use it too - and it's FREE! http://www.ellaforspam.com
Re: PIPEDDR
Rob van der Heij wrote: The Piper normally has a good reason not to put something in the runtime library, but I don't know the motivation in this particular case. You could ask on CMSPIP-L if you have strong desires. Might just be that he hasn't felt the urge to wrangle with IBM Legal since the LZW patents expired. ¬R
Re: PIPEDDR
Since the Support Center now accepts VMARC packed files, is there really any reason to demand TERSE? Will TERSE results be smaller? I doubt it. The reason for my doubt is that I have VMARC packed a large (1.5G) TPF dump and found that it was exactly the same size as it was when processed by a proprietary routine created by a person whose doctoral dissertation was about data compaction. I suspect that everyone on this list has VMARC. If not, it is easily obtainable, and the price is right. Of course, you may have other reasons for wanting TERSE. If so, you can ignore the above and continue complaining or wishing, whichever fits. Regards, Richard Schuh -Original Message- From: The IBM z/VM Operating System [mailto:[EMAIL PROTECTED] On Behalf Of [EMAIL PROTECTED] Sent: Wednesday, July 11, 2007 2:29 PM To: IBMVM@LISTSERV.UARK.EDU Subject: Re: PIPEDDR For some reason, IBM doesn't want to make TERSE available to VM customers. You used to be able to order a package on IBMLINK called QSEN= D that was supposed to be used to pack VM dumps before sending them to the = support center, (back in the days when that was a frequent event :-) ). = QSEND is basically TERSE shipped as QSEND MODULE instead of TERSE MODULE.= I say it is/was basically the same as TERSE, except that the DETERSE MODULE for VM could not DETERSE a file created with QSEND, (IBM wanted it= to be a one way thing). I found out that the output created by QSEND was= different from TERSE by only one byte! (The first byte.) I figured out = how to ZAP DETERSE so that it would work with QSEND files as well as TERSEd files. I don't know if you can still order QSEND for VM on IBMLIN= K or not, (it is not a PTF, you had to order it as a package called QSEND).= Dale R. Smith Persona Non Grata
PIPEDDR TERSE and other stuff
Greetings Listers, PIPEDDR can use TERSE or PACK stages for compression or use no compression.. It only fails if TERSE option is specified and the stage is not available. It is written so it does not rely on TERSE I have used PIPEDDR with PACK option to transfer complete 3390s from one data centre to another and it works well. It saved the delay of waiting for a courier to take a cartridge. Just an aside, I thought I would update the code so it used standard PIPES but when I did that TCPIP packets were lost. Maybe TCPCLIENT and TCPLISTEN have different code or maybe I screwed it up or maybe it was timing. Another aside. Thanks to the author Bruce Hayden publishing the code I was able to use PIPEDDR as a model and instead of using TRACKREAD and TRACKWRITE to read and write dasd I used READER and URO to read and write spool making a very simple NJETCP link. Cheers Graeme
Re: PIPEDDR TERSE and other stuff
On 7/11/07, Graeme Moss [EMAIL PROTECTED] wrote: Just an aside, I thought I would update the code so it used standard PIPES but when I did that TCPIP packets were lost. Maybe TCPCLIENT and TCPLISTEN have different code or maybe I screwed it up or maybe it was timing. If you mean using the version of CMS Pipelines that comes with z/VM, I fear you would at least miss the trackread and trackwrite stage. Those are the ones that Bruce is using too. Those stages are very useful to handle non-CMS disks. The difference for the TCP/IP stages is only for some specific cases at termination of the connection. Rob
