Re: Fast JSON parsing - was Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Tue, Jul 21, 2015 at 5:52 AM, David Crayford dcrayf...@gmail.com wrote: On 20/07/2015 9:37 PM, John McKown wrote: Thanks for the pointer. This is a on my own project which has been put on the back burner. I'm fighting a hardware upgrade issue at home right now. I think the SATA interface on the motherboard of one of my systems is getting flaky. And the old, IDE, CD-ROM has gone totally silent, along with the main HD getting a lot of read errors. So I put in some more memroy, a 512 MiB SSD, and 2 (older) SATA HDs I had laying around; ordered new PCIe SATA controller DVD burner. I did a fresh install of RedHat Fedora on the SSD. Now in the process of re-installing all the software. Whew! Not easy for a man with arthritis. Looks like a good build. Just out of interest why do you favor RedHat Fedora? Versus what other distribution? RH was one of the first available. When I first tried, and failed, there was mainly only Slackware. I couldn't get that installed on my XT. I don't remember what else was available when I first got RedHat. I do remember that I had to pay for it and got access to RHN for updates. This was long before Fedora. I guess that I've stayed with RH simply because I'm used to it. I have also dabbled with OpenSUSE, on a laptop, and Linux Mint. For some reason, I never even tried Ubuntu (yes, Mint is an offshoot) or Debian. I definitely am not going DIY. The only thing RH has done, lately, which has been irritating was going with systemd. I'm learning it, but I really understood SYSTEM V init. And I agree with the philosophical argument that systemd violates the pristine UNIX motto of do _one_ thing and to it _well_. systemd is trying to do way too much, IMO. -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Fast JSON parsing - was Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Tue, Jul 21, 2015 at 12:45 PM, John McKown john.archie.mck...@gmail.com wrote: Versus what other distribution? RH was one of the first available. When I first tried, and failed, there was mainly only Slackware. I couldn't get that installed on my XT. Ahhh... Good ol' Slackware. I tried that when it first came out and used it for many years. Not sure if I ever tried SLS, which Slackware was forked from. My first distribution of Linux was nothing more than the boot/root floppies from tsx-11.mit.edu. After that and Slackware I also moved to Redhat. Currently I too am on Fedora ... for now. Not sure where else I can go. Debian is a political mess with the Systemd war and Ubuntu on the desktop took a nose dive after version 8(?), IMHO. Maybe I should check out BSD. But you weren't going to get any Linux distribution running on your XT. Linux has always required the memory management capabilities first made available in the 80386. However, I think Minix could run on a XT. systemd is trying to do way too much, IMO. Systemd is trying to do *everything*. And about the only distribution that doesn't force you to use it (yet) is good ol' Slackware. Jim -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Fast JSON parsing - was Re: opinion? LISTCAT (equivalent) with XML / JSON output
On 20/07/2015 9:37 PM, John McKown wrote: Thanks for the pointer. This is a on my own project which has been put on the back burner. I'm fighting a hardware upgrade issue at home right now. I think the SATA interface on the motherboard of one of my systems is getting flaky. And the old, IDE, CD-ROM has gone totally silent, along with the main HD getting a lot of read errors. So I put in some more memroy, a 512 MiB SSD, and 2 (older) SATA HDs I had laying around; ordered new PCIe SATA controller DVD burner. I did a fresh install of RedHat Fedora on the SSD. Now in the process of re-installing all the software. Whew! Not easy for a man with arthritis. Looks like a good build. Just out of interest why do you favor RedHat Fedora? -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Fast JSON parsing - was Re: opinion? LISTCAT (equivalent) with XML / JSON output
FYI, If you are interested in the most efficient JSON parser out there check out jsmn (jasmine) http://zserge.com/jsmn.html. It's an incredibly simple low level API that does no memory allocations and is a thing of absolute beauty. Ironically, simple APIs are always hard to design. Here's an example of how to use it http://alisdair.mcdiarmid.org/2012/08/14/jsmn-example.html. I compiled this in Metal/C so it can be used in any language that can knock-up the mappings. This could easily be offload to a zIIP. On 17/07/2015 10:30 PM, John McKown wrote: On Fri, Jul 17, 2015 at 9:03 AM, David Crayford dcrayf...@gmail.com wrote: IBM are going after JSON big time with the new z/OS Connect tooling which transforms JSON into data structures like COBOL copybooks etc. I hadn't heard of the z/OS Connect tooling. Sounds interesting. I see what I can find on it. That's all good but is JSON always the best choice when a simple text protocol can do the same job without the bloat https://github.com/influxdb/influxdb/pull/2696. XML is even worse. Thanks for the URL. I'll be looking at that. One reason that I was looking at JSON or XML is that not every entry in a LISTCAT has the same information in it. The referenced format does seem to accommodate this because it appears to me that each line is ad hoc formatted. One thing that I, personally, like about XML is that you have have a DTD which precisely defines what is and what is not acceptable as input. And, at least for me, the size of the data is not overly important. Given that I have a 2 TiB HD in my at home (and more on two separate NAS boxes), and most data centers have space in the 100s of TiB (or even more), I don't know if JSON's bloat, compared to the referenced format, is all that terrible. And, of course, for at rest data, the XML/JSON can be compressed (gzip, bzip, xz). If I were truly worried about it, I would probably use some sort of delimited format where the first field of each line identified the schema for the rest of the line. Such as: VL (VSAM LDS), VK (VSAM KSDS), VR (VSAM RRDS), VE (VSAM ESDS), AL (ALIAS), NV (NONVSAM), VV (VSAM VVRDS), and so on. I guess I was more trying to go with something which is current and standardized. The referenced format appears to be in flux and is not an ISO standard. Which reminds me that I really should go over to the ISO site(s) and see if they have had other textual format standard. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: Fast JSON parsing - was Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Mon, Jul 20, 2015 at 8:12 AM, David Crayford dcrayf...@gmail.com wrote: FYI, If you are interested in the most efficient JSON parser out there check out jsmn (jasmine) http://zserge.com/jsmn.html. It's an incredibly simple low level API that does no memory allocations and is a thing of absolute beauty. Ironically, simple APIs are always hard to design. Here's an example of how to use it http://alisdair.mcdiarmid.org/2012/08/14/jsmn-example.html. I compiled this in Metal/C so it can be used in any language that can knock-up the mappings. This could easily be offload to a zIIP. Thanks for the pointer. This is a on my own project which has been put on the back burner. I'm fighting a hardware upgrade issue at home right now. I think the SATA interface on the motherboard of one of my systems is getting flaky. And the old, IDE, CD-ROM has gone totally silent, along with the main HD getting a lot of read errors. So I put in some more memroy, a 512 MiB SSD, and 2 (older) SATA HDs I had laying around; ordered new PCIe SATA controller DVD burner. I did a fresh install of RedHat Fedora on the SSD. Now in the process of re-installing all the software. Whew! Not easy for a man with arthritis. -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Fri, Jul 17, 2015 at 9:03 AM, David Crayford dcrayf...@gmail.com wrote: IBM are going after JSON big time with the new z/OS Connect tooling which transforms JSON into data structures like COBOL copybooks etc. I hadn't heard of the z/OS Connect tooling. Sounds interesting. I see what I can find on it. That's all good but is JSON always the best choice when a simple text protocol can do the same job without the bloat https://github.com/influxdb/influxdb/pull/2696. XML is even worse. Thanks for the URL. I'll be looking at that. One reason that I was looking at JSON or XML is that not every entry in a LISTCAT has the same information in it. The referenced format does seem to accommodate this because it appears to me that each line is ad hoc formatted. One thing that I, personally, like about XML is that you have have a DTD which precisely defines what is and what is not acceptable as input. And, at least for me, the size of the data is not overly important. Given that I have a 2 TiB HD in my at home (and more on two separate NAS boxes), and most data centers have space in the 100s of TiB (or even more), I don't know if JSON's bloat, compared to the referenced format, is all that terrible. And, of course, for at rest data, the XML/JSON can be compressed (gzip, bzip, xz). If I were truly worried about it, I would probably use some sort of delimited format where the first field of each line identified the schema for the rest of the line. Such as: VL (VSAM LDS), VK (VSAM KSDS), VR (VSAM RRDS), VE (VSAM ESDS), AL (ALIAS), NV (NONVSAM), VV (VSAM VVRDS), and so on. I guess I was more trying to go with something which is current and standardized. The referenced format appears to be in flux and is not an ISO standard. Which reminds me that I really should go over to the ISO site(s) and see if they have had other textual format standard. -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
opinion? LISTCAT (equivalent) with XML / JSON output
I imagine that many here, like myself, are guilty of scraping an IDCAMS LISTCAT report using something like REXX to produce information. I do it in a REXX program to recreate DEFINE decks for VSAM data sets which have been lost. I am aware of the IGGCSI00 interface whcih can be used with REXX and other language. I have used it myself in a C program. But I am wondering if others think that it might be nice to have a program which can write the data in a LISTCAT which is formatted as XML, or maybe even JSON. I am considering writing such a program, which I will put on the CBT. Of course, it will use the IGGCSI00 callable service internally. I am posting this here just to get an idea about how popular (or not) such a program might be. I guess I'm looking for a bit of encouragement that it would be worth my while. -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
What clients do you expect to use this API? XML is old hat and being deprecated and JSON is only useful in a language that can encode/decode into an internal object. JSON is better but only if you are using a language that can encode/decode in JSON seamlessly. On 17/07/2015 8:39 PM, John McKown wrote: I imagine that many here, like myself, are guilty of scraping an IDCAMS LISTCAT report using something like REXX to produce information. I do it in a REXX program to recreate DEFINE decks for VSAM data sets which have been lost. I am aware of the IGGCSI00 interface whcih can be used with REXX and other language. I have used it myself in a C program. But I am wondering if others think that it might be nice to have a program which can write the data in a LISTCAT which is formatted as XML, or maybe even JSON. I am considering writing such a program, which I will put on the CBT. Of course, it will use the IGGCSI00 callable service internally. I am posting this here just to get an idea about how popular (or not) such a program might be. I guess I'm looking for a bit of encouragement that it would be worth my while. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Fri, Jul 17, 2015 at 7:57 AM, David Crayford dcrayf...@gmail.com wrote: What clients do you expect to use this API? XML is old hat and being deprecated and JSON is only useful in a language that can encode/decode into an internal object. I don't actually have any clients in mind. This is just for my amusement at present. I was thinking XML simply because z/OS has System XML and IBM has not articulated a z/OS policy for JSON. Examples of this is the COBOL PARSE XML verb, and the use of XML in CICS. Neither COBOL nor CICS natively support JSON. Also, although XML is not as popular as JSON (thanks mainly to JavaScript), it is still a major player. What would be a better output format? Most of my current programming is done on Linux (not z/OS), so I'm becoming fond of UNIX standard output such as XML and JSON. JSON is better but only if you are using a language that can encode/decode in JSON seamlessly. JSON is indeed very popular right now. And anything that I can encode in XML can be equivalently encoded in JSON. And vice-versa. XML is nice thanks to utilities such as XPATH and xslt. In fact, a person could use xslt to transform XML into JSON rather painlessly. XML is easy to read in Java. And it _can_ be read in JavaScript, python, ruby, and most other UNIX scripting languages. I appreciate the feedback. If you have a standard format that I should consider beyond XML and JSON, I'd appreciate a pointer to the documentation on it. -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
IBM are going after JSON big time with the new z/OS Connect tooling which transforms JSON into data structures like COBOL copybooks etc. That's all good but is JSON always the best choice when a simple text protocol can do the same job without the bloat https://github.com/influxdb/influxdb/pull/2696. XML is even worse. On 17/07/2015 9:45 PM, John McKown wrote: On Fri, Jul 17, 2015 at 7:57 AM, David Crayford dcrayf...@gmail.com wrote: What clients do you expect to use this API? XML is old hat and being deprecated and JSON is only useful in a language that can encode/decode into an internal object. I don't actually have any clients in mind. This is just for my amusement at present. I was thinking XML simply because z/OS has System XML and IBM has not articulated a z/OS policy for JSON. Examples of this is the COBOL PARSE XML verb, and the use of XML in CICS. Neither COBOL nor CICS natively support JSON. Also, although XML is not as popular as JSON (thanks mainly to JavaScript), it is still a major player. What would be a better output format? Most of my current programming is done on Linux (not z/OS), so I'm becoming fond of UNIX standard output such as XML and JSON. JSON is better but only if you are using a language that can encode/decode in JSON seamlessly. JSON is indeed very popular right now. And anything that I can encode in XML can be equivalently encoded in JSON. And vice-versa. XML is nice thanks to utilities such as XPATH and xslt. In fact, a person could use xslt to transform XML into JSON rather painlessly. XML is easy to read in Java. And it _can_ be read in JavaScript, python, ruby, and most other UNIX scripting languages. I appreciate the feedback. If you have a standard format that I should consider beyond XML and JSON, I'd appreciate a pointer to the documentation on it. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
You could use the IGGCSI00 Java wrapper that is already in the z/OS Java SDK: http://www-01.ibm.com/support/knowledgecenter/api/content/SSYKE2_8.0.0/com.ibm.java.zsecurity.api.80.doc/com.ibm.jzos/com/ibm/jzos/CatalogSearch.html It allows full access to IGGCSI00 functionality. Kirk Wolf Dovetailed Technologies http://dovetail.com On Fri, Jul 17, 2015 at 10:14 AM, John McKown john.archie.mck...@gmail.com wrote: On Fri, Jul 17, 2015 at 9:54 AM, Kirk Wolf k...@dovetail.com wrote: Would be very cool to have a tool that called IGGCSI00 with parameters like: - dsn and/or catalog filter - list of fields to emit - output format (JSON, pipe-delimited, XML) ( IMO pipe-delimited would be the most useful for piping into awk ) Its pretty easy to call IGGCSI00 from C / C++ / Metal C if you like that sort of thing. I've written a very simple C program which uses IGGCSI00. My language options (in no particular order) are: HLASM - difficult to program in, but fairly performant, most shops have HLASM. REXX - interface is a bit difficult (to me), not very performant, beloved of my manager. C - fairly easy to program in, performant, getting more popular on z/OS?, PL/I - my first love, I think it's still performant, not popular at all as best as I can tell. COBOL - beloved of business programmers, middle performance, wordy. FORTRAN - the less said the better. Java - an interesting idea, performant on an zAAP, write everywhere, run nowhere according to the critics. I'll most likely go with C. I'm a mid-level programmer in it, and I can use my Linux tools for editing and syntax checking. C++ would be interesting if I were going to distribute a IGGCSI00 class for other C++ programmers to use. But I'm not really very good with C++, no experience at all. -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Fri, Jul 17, 2015 at 10:19 AM, Farley, Peter x23353 peter.far...@broadridge.com wrote: Maybe also output format CSV (which could be either comma- or tab-separated) for direct input to spreadsheet software. If we had GNU awk 4.1+ available in z/OS Unix we would also have the gawk extension API which could be used to write a C program to invoke IGGCSI00 which could be invoked directly from an awk script (like the gawk XML extension). Then we would be in really nice shape. One can dream . . . Yes. I would love that. My manager would then kill me. He dislikes z/OS UNIX. It may be due to the fact that we are getting close to retirement and he doesn't really want to be bothered with it. Also, the company is cloud sourcing as much of IT as it can (all the infrastructure it can including the z/OS machine, all Intel machines (Windows Linux), with the z/OS processes being converted to Windows ). This makes it difficult to be very interested in new things that we'll never see. He's a very good programmer, an excellent manager, but not a computer geek. He has truly said that he'd prefer to dig ditches, if it just paid as well. I, OTOH, and a computer nerd and have been since college (mucho years ago). Peter -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
Re: opinion? LISTCAT (equivalent) with XML / JSON output
On Fri, Jul 17, 2015 at 10:39 AM, Kirk Wolf k...@dovetail.com wrote: You could use the IGGCSI00 Java wrapper that is already in the z/OS Java SDK: http://www-01.ibm.com/support/knowledgecenter/api/content/SSYKE2_8.0.0/com.ibm.java.zsecurity.api.80.doc/com.ibm.jzos/com/ibm/jzos/CatalogSearch.html It allows full access to IGGCSI00 functionality. Great idea. Thanks for the URL. Kirk Wolf Dovetailed Technologies http://dovetail.com -- Schrodinger's backup: The condition of any backup is unknown until a restore is attempted. Yoda of Borg, we are. Futile, resistance is, yes. Assimilated, you will be. He's about as useful as a wax frying pan. 10 to the 12th power microphones = 1 Megaphone Maranatha! John McKown -- For IBM-MAIN subscribe / signoff / archive access instructions, send email to lists...@listserv.ua.edu with the message: INFO IBM-MAIN
