I don't think this is an appropriate use of the list serve. Though I understand Kepa's message was in response To Rama's, this kind of email is too close to a sales pitch. It gives other vendors no choice but to respond in kind. Now, if we want the list to turn into a forum for vendors to discuss the quality of their products and benefits of its features in comparison to others, we can make that choice, but unless we're ready to open the door to everyone with a testing product to plug, I suggest this discussion change direction.
Whether you call it testing or certification, it seems the real topic here is to what extent does analysis need to occur? That's a valid discussion but product comparisons do not belong here. -----Original Message----- From: Kepa Zubeldia [mailto:[EMAIL PROTECTED]] Sent: Sunday, November 24, 2002 1:31 AM To: WEDI SNIP Testing Subworkgroup List Subject: Re: VALIDATION or Certification Rama, I agree with you that the work of comparing results between the three largest testing vendors was very much needed. Patrice and Mike are to be commended for all the work they put into this, and I hope there will be more extensive studies in the future. You bring up some great points. And since you invited me to address them, here it goes... First, in spite of the title of the session being "Validation or Certification", the specific instructions from Patrice to all three vendors were that we were ONLY to discuss testing. We were not supposed to discuss certification, only testing. I specifically asked Patrice about this because it seemed odd, given the title of the session and she specifically forbade me to discuss certification or HIPAAmetrics. My presentation was to be ONLY about testing outbound transactions using Velocedi. Since I was the first presenter, I stuck to the script. The ONLY thing we were supposed to do in our 10 minutes each was to test/analyze/validate the one file that Patrice had provided us with, and no other. No other discussion or presentation was to take place, per Patrice's instructions. Much to my dismay, the other two presenters chose to present their entire product lines rather than sticking to the rules. Patrice, given the situation, passed me a note inviting me to take a second turn of 5 minutes to make a "sales pitch", but I declined. Had I know that I could present not only about outbound testing but also about certification and inbound testing and inbound certification, I would have used my 10 minutes very differently. So, given that the only topic was testing/analysis/validation, you can probably understand why when you brought the topic of the difference between testing and certification from the audience, Patrice diverted your question and never addressed the certification part of it. She was not interested in discussing certification in this meeting, only testing. I hope this email addresses your question. Comparing the offering of these three vendors by only looking at the results of testing outbound EDI files is equivalent to comparing automobiles by only looking at their tires. Yes, they do all have four tires on the road. Yes, all tires are made out of rubber, steel, plastic, cotton and other fibers. Yet, some tires will separate the thread under certain conditions and other tires will work better in the snow or may have studs for ice. But, would you buy a car based on the tires? There are many other aspects that are much more important than the tires. The Minnesota case study only looked at the single aspect of outbound transactions test results, with the explicit exclusion of other important aspects, such as certification or inbound testing. So as far as testing is concerned, Patrice showed that the three main contenders can test the files with some differences. However, had we tested a different file, a more moderately sized file for example, we would had seen a number of significant differences in the quality and completeness of the results. This is left for a future exercise. For example, I tested a medium size file with 3,500 professional claims and it took 133 seconds (2m13s) using the Foresight software but only 3 seconds using Claredi's Velocedi, using exactly the same hardware. This sort of differences are not visible when the test files are as small as the file Mike and Patrice gave us. I will stop here, as I don't want to start benchmarking discussions. As for your request for clarification about the disclaimer in the test report... The test report is exactly that, a TEST report. It is NOT certification. For almost two years now I have been making a big distinction between testing and certification. Some people understand the difference, others don't understand the difference. I think it is clearly expressed in the white paper, but obviously it is not as clear as I think it is, or this topic would not come up so often. I will try to explain it in this email, but I suspect it is going to need a more detailed description, so I am going to start working on a Claredi white paper to make sure people understand what WE mean by certification. I am aware that most of our competitors do NOT understand the difference and therefore try to equate the two concepts as if they were one and the same. In their eyes, if you pass a test without errors, that equates to being certified HIPAA compliant. I believe that is not the case. Please keep reading. When Patrice approached Claredi about the case study, she was very explicit about wanting to show only outbound test results. She did NOT want to show inbound testing (with Claredi's Wibbler) and she did not want to show certification in either direction. So we gave her a temporary test account of the same kind we offer in our "trial run" accounts. Those accounts are established with only a limited set of capabilities and do NOT have access to Claredi's certification, they have access only to testing tools. In a "normal" (that includes certification) account, the report summary that contains the disclaimer you are citing also contains a few more sections between the "Action" and the "Support" areas of the summary. These other sections are labeled "Certification", "HIPAAmetrics", and "File Specific Info". Again, these sections are only available to Claredi customers, not to test accounts. Also, test accounts are restricted in other ways. I can give you the restriction details off-line. Since I don't think it would be appropriate to turn this email into a sales pitch, I will only describe the outbound "Certification" process. We also have a completely different process for inbound certification, and I could describe it another day if there is enough interest. Once a file has passed the Claredi testing/verification/validation (all synonyms) process without HIPAA errors, it becomes eligible for certification by Claredi. The "Certification" section of the summary page then shows a button labeled "Submit this file for certification". If the file has HIPAA errors the button does not appear. Unless the user clicks this button, the file is not taken through Claredi's certification process. It is only tested, not certified. This is why the summary report MUST NOT be construed as a certification report, even if it shows that a file does not have "HIPAA errors". And thus the disclaimer. As you can tell, the summary report shows three types of "errors": the "HIPAA Errors", with the standard 7 types defined in the SNIP white paper, the "Business Errors" section, and the "Warnings" section. Only the "HIPAA Errors" section must be clean for the file to be a candidate for certification as HIPAA compliant. The file could have "Business Errors" and "Warnings" and still be certified as HIPAA compliant, since the certification of compliance measures against the requirements of the HIPAA implementation guides. In the next few weeks we will be relaxing the cleanliness requirement even further. For a file to qualify as a candidate for certification, it will need to have at least one "business unit" (e.g. claim) without "HIPAA Errors" rather than the entire file being error free. The HIPAAmetrics certification will give a detail of the contents of the file, including the percentage of compliant and non-compliant claims. This is currently in "beta" with a restricted number of accounts and will appear in all the Claredi accounts in a few weeks, as well as in the certification details through the directory. But, back to the certification process... Once the file passes the testing clean enough to be "certifiable", and the user clicks on the "Submit for certification" button, Claredi takes the file through a process that looks at the data content. Based on the data content of the file we can identify patterns, such as (I will use the claim as an example) the presence of Billing Provider and/or Pay-To provider, or the presence of both Subscriber and Patient or only Subscriber, or the presence of secondary coverage, or the presence of a prior adjudication (COB claims), or the presence of a referring physician, or the maximum number of service lines submitted by this entity, or the presence of a "cluster of data" that indicates this is an anesthesiology claim, or a DME claim, or an ambulance claim, etc. We identify these data clusters through a process we call HIPAAmetrics. This is a unique Claredi pattern recognition process that has never before been applied to EDI data, and we have applied for a patent to cover it. The result of the process gives us the "metrics" of this particular file. I am sure you have heard me say many times that the fact that a file does not contain errors is not enough. You have to know what was actually contained in the file. Using HIPAAmetrics we can determine exactly what is in the file, from the business perspective. The testing phase looks at the EDI perspective and determines that the file (or the individual claim within the file) does not contain errors. The HIPAAmetrics looks at the healthcare contents and "characterizes" it. At the completion of the HIPAAmetrics phase Claredi can say: We have seen a file that contained the following data - Office visits, 234 without errors, 3 with errors - Primary claims, 180 without errors, 20 with errors - Primary claims with a second payer, 0 without errors, 20 with errors - Secondary claims, 0 without errors, 97 with errors - Consultations, 5 without errors, 40 with errors - etc. Claredi is making a statement of what we have seen. Because there are some claims that are "compliant", the provider could claim to be compliant. However, the statement from Claredi clearly shows that although this provider has the ability to generate HIPAA compliant office visits for "single payer" situations, the provider does not have the capability to create correct claims when two payers are involved. The HIPAAmetrics describe the profile that we have seen reflected in the EDI data itself. The HIPAAmetrics certification gives the detailed data profile necessary to make a determination of compliance. Without this sort of detail, all you would know is that the provider has generated something that did not have errors, but you don't know what that "something" was. Certainly none of the other testing entities (neither our competitors nor the translator validators) provide these details, and I suspect that they do not yet understand the difference between passing a "clean test" and being certified as HIPAA compliant. Both EDIFECS and Foresight show a pass/fail approach in their reports. Of course, by the end of this email their eyes will be wide open. So, there is a significant difference between a testing or validation service and Claredi's certification. Big difference. The problem is that if you don't understand the difference and assume that any entity that can produce an error free file is automatically "certified" you will certainly find a real problem when you look at the quality of the data that entity sends. And thus the bad rap that the "pseudo-certification" provided by the pass/fail approach of our competitors is giving to the word "certification". And this is also the reason why a vendor cannot really be certified unless all of its clients are individually certified. It is just too easy for the vendor to produce certain clean transactions, and for the clients to not be able to produce the transactions THEY need in a compliant form. Ditto for clearinghouses. >From the usability side, a clean test that does not have the HIPAAmetrics type of data analysis, still needs to be reviewed by an expert to determine whether the test is sufficient in volume and quality, and to determine what is it that was actually tested. Did the test include sufficient variety of business scenarios to make it relevant? Were those scenarios complete? Is the volume of the sample for each scenario big enough? These are some of the sort of things the EDI folks at the payer's end have been looking in provider's tests. Using an EDI test tool may tell you that the EDI data does not contain errors, but that is only a small part of the answer. Most translators can do that too, so, if that is all the EDI test tool does, it is not much better than the translator rules. The Claredi HIPAAmetrics answer these questions. The payer can then define what are the required parameters. Things as simple as saying "we need to see 95% clean claims in a test containing at least 200 claims" can be answered by HIPAAmetrics and the Claredi certification, and cannot be answered by any of the other test tools in the market today, without human intervention. I hope this email sheds some more light on these concepts. I have been talking about this for almost two years. It is (washed out) in the SNIP white paper. And yet, the concept is still a mystery to most, even a lot of experts. The reality is that it will take a while to grasp the concept. Those of us that have done extensive testing of healthcare EDI before HIPAA (NSF, UB92, etc.) understand the concept easily. For some reason the traditional non-healthcare EDI community (X12) has more difficulty understanding the concept. Now let me add another important difference between the "test results" and the HIPAAmetrics. The test results, by their nature, will contain PHI. If there are any errors, the PHI will most likely be part of the error message or description. Claredi's test results also contain PHI. So we have a bold statement on the test results report saying that they must be handled according to the appropriate privacy and security procedures. And this in spite or requiring a Business Associate Agreement with all our customers. The HIPAAmetrics reports, on the other hand, do NOT contain PHI and thus are shareable with the world. In fact, in a few weeks we will be releasing the HIPAAmetrics results as part of the Claredi directory describing the details of each certification obtained through Claredi. There is a concept of "community" testing that is being touted lately. The problem that I see is that if the "community" is comprised by more than one payer (e.g. several payers sharing the results of all the providers in the community) then all the payers in the "community" are seeing the PHI from all the testers. Unless the provider creates a separate test account for each one of the payers in the "community" and then sends the appropriate test files to each one of those accounts, so each of the payers will only see the test files that corresponds to that payer, there is a problem... All the payers in the "community" will be seeing (through the "community" testing center) the PHI that belongs to the other payers. As far as I can tell this is not kosher for healthcare privacy reasons. Even if the provider has a "business associate agreement" with the community test center, and a BAA with each one of the payers, and each payer has a BAA with each one of the payers in the "community", I still think that the "minimum necessary" rules would preclude this sort of bulleting board sharing of PHI within the testing "community". Yes, you guessed right. If instead of sharing the test results, like these "communities" do today, they were to share the HIPAAmetrics certification, the PHI would never be shared. Some people have seen the vision. We have offered it to other testing vendors and translator vendors. Some have expressed interest, others are not interested. Clearly we need to do a lot more education on this topic. The American Hospital Association understood the concept and is now endorsing Claredi as the exclusive HIPAA transactions testing and certification service to their members. So, going back to Patrice's case study, you cannot compare testing and certification if the only thing you look at is testing. Testing your own transactions is important. Some people will focus only on testing. That is just fine, as there are many different approaches to putting a trading partner into production. We believe out test tools are better than our competitors', but I am sure they believe theirs are better. But testing is only part of the picture. To establish a business relationship for HIPAA transactions, you must go beyond compliance testing. Knowing that the data you receive is clean is important, but you must also know what is contained (from the business perspective) in that clean data stream. Only Claredi gives you an automated way to determine that. We call it HIPAAmetrics Certification. For the last 20 years we have done without it, and we have been happy with just test tools. Some people are just happy with the current tools, other people want to explore a more automated way to put a trading partner into production. This country was built on freedom of choice. No need to put someone else down just because you don't like what they have to offer. If you want to see it in action, give me a call and I can show it to you. It is really simple... Kepa On Saturday 23 November 2002 12:54 pm, [EMAIL PROTECTED] wrote: > For the benefit of those who could not make it to the WEDI SNIP HIPAA IMPLEMENTATION SUMMIT held in Phoenix from Nov 18 - 21, 2002. > > There was a session on Nov 20th between 9.45 AM - 11.45 AM on the topic "Validation or Certification", moderated by Patrice Thaler of Allina Health System and Mike Ubl of Blue Cross of Minnesota and the main participants being three vendors of testing tools represented by their big chiefs... Namely Sunny Singh from EDIFECS, ED Hafner from Foresight Corp, Kepa Zubeldia from Claredi, presenting a case study of three vendors. > > First of my sincere thanks to Patrice, Mike and whoever helped in this case study, besides the vendors. > > Being the author of the vendor-neutral statement in the Testing and Certification White Paper (Hey John, why is my name missing in the acknowledgements section? ;-)), and being the first (one of the?) to suggest the formation of Vendor (of testing tools AND TRANSLATORS) Consortium in one of the teleconference calls in May/June of 2002, I am really glad that this case study was done. It is the first step in that direction of forming the vendor consortium to help the CEs be able to pick and choose any one of them vendors, for they ALL have to offer the same basic VALIDATION (not certification) capability by coming up with the same ERROR messages and similar warning messages for the same test files. This case study proved just that point and am truly delighted and am happy to share it with one and all. > > One more interesting thing and I would like Kepa Z to clarify for all of us... > > On page 3(of the handout we were given there), on a "Claredi - Passed Claim", there is a heavy disclaimer in BOLD, which reads "This is not a certification or verification of HIPAA compliance Summary of test report for confidential use by Claredi customer only". I know and agree that it is NOT CERTIFICATION. I can also understand why the disclaimer is there. But Kepa should explain to us why he touts Certification with an evangelical pitch and then this disclaimer... saying it is NOT CERTIFICATION! > > Another important point, as suggested by Larry Watkins, as a member of the X12N committee (correct me Larry if I am not saying this right), that the discrepancies in the standards are not that many and there is a very good process to address them. I can't agree more. > > The bottom line is, it is Testing, Verification and Validation and NOT CERTIFICATION. End of discussion. I have been saying this all along and one more time won't hurt ;-). That said, I really do believe all the vendors, have a lot to offer in the validation part of transaction testing. I have no bias or preference towards one vendor. The CEs (or their testing people) should make the choice. Don't let anyone fool you that it is soooooooo big and only I (my org) can lead you(r org) to salvation ;-). > > Accreditation, creating entities to certify and the process of certification is a long one and can be done, but vendor consortium will eliminate the need for certification all together, IMHO. > > This message is not intended to be against any individual or a flame and rake up any unnecessary things but to share the joy of possible formation of vendor consortium and also request the translator vendors to join in the vendor consortium. This will really help in reducing the chaos and eliminate the obfuscation and confusion that is there and being created... We don't have time and anybody who can chip in to make things simple to whatever extent should be commended... like Patrice and Mike and all those who participated in the case study. In Patrice's words, paraphrased, "It is easy and you can do it, and I have done it". > > Thanks for reading up to this point. > > Best Regards, --Rama. > > --- > The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. These listservs should not be used for commercial marketing purposes or discussion of specific vendor products and services. They also are not intended to be used as a forum for personal disagreements or unprofessional communication at any time. > > You are currently subscribed to wedi-testing as: [EMAIL PROTECTED] > To unsubscribe from this list, go to the Subscribe/Unsubscribe form at http://subscribe.wedi.org or send a blank email to [EMAIL PROTECTED] > If you need to unsubscribe but your current email address is not the same as the address subscribed to the list, please use the Subscribe/Unsubscribe form at http://subscribe.wedi.org > > -- This email contains confidential information intended only for the named addressee(s). Any use, distribution, copying or disclosure by any other person is strictly prohibited. --- The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. These listservs should not be used for commercial marketing purposes or discussion of specific vendor products and services. They also are not intended to be used as a forum for personal disagreements or unprofessional communication at any time. You are currently subscribed to wedi-testing as: [EMAIL PROTECTED] To unsubscribe from this list, go to the Subscribe/Unsubscribe form at http://subscribe.wedi.org or send a blank email to [EMAIL PROTECTED] If you need to unsubscribe but your current email address is not the same as the address subscribed to the list, please use the Subscribe/Unsubscribe form at http://subscribe.wedi.org --- The WEDI SNIP listserv to which you are subscribed is not moderated. The discussions on this listserv therefore represent the views of the individual participants, and do not necessarily represent the views of the WEDI Board of Directors nor WEDI SNIP. If you wish to receive an official opinion, post your question to the WEDI SNIP Issues Database at http://snip.wedi.org/tracking/. These listservs should not be used for commercial marketing purposes or discussion of specific vendor products and services. They also are not intended to be used as a forum for personal disagreements or unprofessional communication at any time. You are currently subscribed to wedi-testing as: [email protected] To unsubscribe from this list, go to the Subscribe/Unsubscribe form at http://subscribe.wedi.org or send a blank email to [EMAIL PROTECTED] If you need to unsubscribe but your current email address is not the same as the address subscribed to the list, please use the Subscribe/Unsubscribe form at http://subscribe.wedi.org
