23-Oct-2018 21:40:01-GMT,18244;000000000000 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit precedence: bulk Subject: Risks Digest 30.89
RISKS-LIST: Risks-Forum Digest Tuesday 30 October 2018 Volume 30 : Issue 89 ACM FORUM ON RISKS TO THE PUBLIC IN COMPUTERS AND RELATED SYSTEMS (comp.risks) Peter G. Neumann, moderator, chmn ACM Committee on Computers and Public Policy ***** See last item for further information, disclaimers, caveats, etc. ***** This issue is archived at <http://www.risks.org> as <http://catless.ncl.ac.uk/Risks/30.89> The current issue can also be found at <http://www.csl.sri.com/users/risko/risks.txt> Contents: MTR East Rail disruption caused by failure of both primary and backup (Hong Kong Free Press) Train stops in exactly the wrong place (Mark Brader) Texas straight-ticket voters report ballot concerns (Arthur Flatau, MikeA) Australian risks of voting systems (Sheldon) Re: U.S. Begins First Cyberoperation Against Russia Aimed at Protecting Elections (Monty Solomon) Tech support -- Hubble telescope (Rob Slade) Login glitch behind Tokyo Stock Exchange snafu (Nikkei Asian Review) State surveillance company leaked its own data, its customers' data, and its customers' victims' data (BoingBoing) "New Windows 10 1809 bug: Zip data-loss flaw is months old but Microsoft missed it" (Liam Tung via Gene Wirchenko) Driverless cars: Who should die in a crash? (bbc.com) Every minute for three months, GM secretly gathered data on 90,000 drivers' radio-listening habits and locations (BoingBoing) Surgery students 'losing dexterity to stitch patients' (bbc.com) In Cyberwar, There are No Rules (Foreign Policy) Lawmakers Seek Review of Pentagon Contract Thought to Favor Amazon (WiReD) The customer is always right ... re: Apple iPhones (Rob Slade) Fun with source code (Medium) A Dark Consensus About Screens and Kids Begins to Emerge in Silicon Valley (The New York Times) When Trump Phones Friends, the Chinese and the Russians Listen and Learn (NYTimes) Apple appears to have blocked GrayKey iPhone hacking tool (Lucas Mearian) Re: Toward Human-Understandable, Explainable AI (DJC) Re: Explainable AI Simulation for AVs (Richard Stein) Abridged info on RISKS (comp.risks) ---------------------------------------------------------------------- Date: Mon, 29 Oct 2018 22:06:46 +0800 From: Richard Stein <rmst...@ieee.org> Subject: MTR East Rail disruption caused by failure of both primary and backup (Hong Kong Free Press) https://www.hongkongfp.com/2018/01/11/mtr-east-rail-disruption-caused-failure-primary-backup-servers/ ------------------------------ Date: Mon, 29 Oct 2018 14:56:29 -0400 From: Mark Brader <m...@vex.net> Subject: Train stops in exactly the wrong place (Modern Railways) According to a short item on page 87 of the October issue of "Modern Railways", on August 21 a suspected shoplifter was chased into a train tunnel at Amsterdam's Schiphol Airport, requiring the train service to be temporarily shut down. But when they went to restart it, the entire computerized train management system crashed and would not come back up. As a result, all trains throughout the greater Amsterdam area were halted from some time in the evening rush hour until after midnight when the bug was finally identified and fixed. "It transpired", the article says, "that one train had been stopped at exactly the point where the software determines which platform a train should use" and hence "the software continuously detected a train arriving at the spot and proceeded to try and allocate the non-existent arrival (the train was already there!) 32,000 times before the system crashed." ------------------------------ Date: Sat, 27 Oct 2018 08:07:15 -0500 From: Arthur Flatau <flat...@acm.org> Subject: Texas straight-ticket voters report ballot concerns Austin American Statesman The idea that using hitting a button or other control while a screen is rendering is a user error is astounding. If the machine incorrectly interprets user input it is a bug plain and simple. Amid scattered complaints by straight-ticket early voters of both parties that their ballots did not, at first, correctly record their choice of either Democrat Beto O'Rourke or Republican Ted Cruz for U.S. Senate, state and local election officials are cautioning voters to take their time in voting and check the review screen for accuracy before casting ballots. The elections officials say the problems resulted from user error in voting on the Hart eSlate machines widely used in Texas -- including in Travis, Hays and Comal counties -- and are not the result of a machine glitch or malfunction. ``The Hart eSlate machines are not malfunctioning,'' said Sam Taylor, communications director for the Texas secretary of state's office. ``The problems being reported are a result of user error -- usually voters hitting a button or using the selection wheel before the screen is finished rendering.'' Taylor said the office is aware of a handful of complaints and that the voters were able to correct their ballots before casting their votes. https://www.statesman.com/news/20181026/texas-straight-ticket-voters-report-ballot-concerns [On the other hand, this explanation might be somewhat evasive. For example, see Kim Zetter' article on this subject: Voters in Texas aren't to blame for vote-switching in Cruz/O'Rourke race; a software issue known as a race condition or concurrency bug is, says Dan Wallach, who notes machine vendor failed to fix this and many other problems found with the Hart machines at least ten years ago. https://twitter.com/KimZetter/status/1057332585313910785 Note: Dan Wallach, Rebecca Mercuri, and I testified before the Houston City Council in July 2001 on why the these machines (still in use today) were likely to be vulnerable. PGN] ------------------------------ Date: Thu, 25 Oct 2018 20:59:15 -0500 From: mikea <mi...@mikea.ath.cx> Subject: Texas straight-ticket voters report ballot concerns (RISKS-30.89) People have been talking about voting machines registering a vote other than the one the voter intended. It happened to a friend in Collin County, Texas. She voted Straight Democratic Party on an electronic voting machine, and had her votes change to all Republican candidates for the same positions. It was good that she noticed this before she actually hit the button to register her votes. She noticed that the process was repeatable: straignt Democratic party changed to straight Republican party a second time, called an election judge over, and demonstrated it a third time. The election judge reluctantly took that voting machine out of service. I find myself wondering if the same thing happened to others who *didn't* notice before they completed the vote using that machine. My more paranoid self, noting that these machines have no paper ballots as a permanent record, wonders if the machine was somehow rigged to change straignt Democratic to straight Republican -- the more so because Collin County is pure, saturated RGB=(255,0,0) Republican. It also wonders how many more machines did the same change. My _extremely_ paranoid self wonders if there are documents circulating among a small subset of election officials, with titles like "How to rig FooCorp voting machines to help your side". An acquaintance who works for the election board in a Georgia county tells me that the reports that votes for the Democratic candidate for Governor were, at the ultimate moment being changed *in the voting machine* to votes for the Republican candidate -- again, on all-electronic machines that dont use paper ballots and have no audit trail. Paper ballots make true recounts possible. Who controls these voting machines controls the election. ------------------------------ Date: Tue, 23 Oct 2018 22:44:19 -0400 From: Sheldon <sheldon10...@gmail.com> Subject: Australian risks of voting systems (RISKS-30.88) The Australian experience with counting votes will not work for the US. I've been a DRO, someone who has run a poll, at Canadian Federal, Provincial and Municipal Elections. Counting by hand the less than 200 ballots for a Federal or Provincial election was no problem. There is a paper ballot and one office to count. I told the scrutineers (partisans who watched the count) that they had a few seconds to look at a ballot and object. Then, I'd decide. If they didn't like the decision, that ballot went an envelope for disputed ballots along with spoiled ballots. In case the vote was very, very close, they first looked at those questionable ballots.I was one of the first to get my ballot box back to the riding office. Counting by hand a municipal election where there were two different ballots and 5 offices on a ballot was a nightmare. After doing one, I never did another one. Now there are still two different ballots but, the ballots are counted by OCR. The Election lists are maintained by a non-partisan body. There are ID requirements but, with the liberals in power, very little is required. In the past, the position of election officials on the day of the election was partisan. Now, they are happy to take anyone. Of course, with the mad Doug Ford in power in Ontario, no one knows where his madness will lead. Ontario elected an idiot knowing he was an idiot. We just didn't know how much of an idiot he would be. ------------------------------ Date: Wed, 24 Oct 2018 18:06:58 +0800 From: Richard Stein <rmst...@ieee.org> Subject: Re: U.S. Begins First Cyberoperation Against Russia Aimed at Protecting Elections (Solomon, RISKS-30.84) https://techcrunch.com/2018/10/23/first-cyber-operation-gentle-approach-russian-trolls/ A line in a CV stating: "Recipient of US Cyber Command email advising to cease and desist election interference, and immediately end trolling in OCT2018" must be an honor among the Russian cyberwarrior cognoscenti. RISK: Does it justify a salary raise request? ------------------------------ Date: Thu, 25 Oct 2018 12:10:07 -0700 From: Rob Slade <rmsl...@shaw.ca> Subject: Tech support -- Hubble telescope Two weeks ago, the Hubble telescope experienced a gyroscope failure. Hubble has been very important, and has contributed enormously to our understanding of the universe. This is a hugely expensive device, which has had problems in the past. It's up in space where you can't exactly get someone to go and hit it with a hammer in hopes it'll start working again. NASA has tried a number of sophisticated procedures to get Hubble functioning again. They haven't worked. Now NASA has turned it off, and back on again. https://gizmodo.com/hubble-telescope-s-broken-gyroscope-seemingly-fixed-aft-1829934018 or https://is.gd/JgwOMu Hubble is working again ... When I'm dying in hosptial I want them to unplug all the tubes and plug them back in and see if that works ... ------------------------------ Date: Tue, 30 Oct 2018 14:58:54 -0400 From: Gabe Goldberg <g...@gabegold.com> Subject: Login glitch behind Tokyo Stock Exchange snafu (Nikkei Asian Review) https://asia.nikkei.com/Business/Markets/Login-glitch-behind-Tokyo-Stock-Exchange-snafu ------------------------------ Date: Wed, 24 Oct 2018 11:44:41 -0700 From: Lauren Weinstein <lau...@vortex.com> Subject: State surveillance company leaked its own data, its customers' data, and its customers' victims' data (BoingBoing) via NNSquad https://boingboing.net/2018/10/24/20-gb-of-internal-data.html ------------------------------ Date: Tue, 23 Oct 2018 18:31:07 -0700 From: Gene Wirchenko <ge...@telus.net> Subject: "New Windows 10 1809 bug: Zip data-loss flaw is months old but Microsoft missed it" Liam Tung, ZDNet, 23 Oct 2018 https://www.zdnet.com/article/new-windows-10-1809-bug-zip-data-loss-flaw-is-months-old-but-microsoft-missed-it/ A Feedback Hub user reported the latest Windows 10 October 2018 Update bug three months ago. Microsoft has fixed the issue in preview builds of the 19H1 version of Windows 10, so it should be fixed in 1809 soon. opening text: Windows 10 version 1809 update is still on ice due to the data-deletion bug embarrassingly missed by Microsoft during preview testing. But the few users who did get the Windows 10 October 2018 Update have now discovered its built-in zip tool is doing weird things when copying files. As one 1809 user reported on Reddit, this version of Windows 10 is missing the 'Do you want to replace these files' dialog when copying from a zip archive to a folder with an identically named file in it. The problem only seems to affect the built-in zip tool in Windows File Explorer rather than third-party zip tools. The dialog is an important flag when transferring a lot of files, since it's an opportunity for the user to choose whether to replace the identical file, skip replacing the file, or compare the information stored in both files before taking any action. Without the dialog, it could be easy to unintentionally overwrite non-identical files. ------------------------------ Date: Thu, 25 Oct 2018 15:29:00 -0400 From: Gabe Goldberg <g...@gabegold.com> CTA - Human-Machine Interfaces Evolve in Cars Switches and dials have been the norm for controlling things in cars, from the side mirrors to audio volume. But norms evolve. As automakers prepare for a world of shared self-driving cars, theyâre experimenting with an array of human-machine interface technologies, or HMIs, including interior-facing cameras, gesture and voice controls, and touch-sensitive surfaces â all augmented by ever-smarter computing platforms. Voice controls are en route to be the second most-prevalent interface by 2022, when itâs forecast to be in 80 percent of car HMIs, up from 48 percent in 2016, according to the consulting firm Frost & Sullivan. Data published last year in the firmâs Global Connected Car Market Outlook show touchscreens on top, with 90 percent market share by 2022, up from 29 percent two years ago. Multifunctional controllers (50 percent from 16 percent), handwriting recognition (30 percent from nine percent), digital instrument clusters (25 percent from seven percent) and head-up displays or HUDs (20 percent from five percent) follow. Only gesture controls will remain relatively rare in four years, with just five percent HMI penetration worldwide, but still up tremendously from 0.02 percent in 2016, Frost & Sullivan predicts. Theyâre helping the driver âget more accustomed to newer technologies, so that the user acceptance is there before he or she is going to give over control to the car in autonomous mode,â says Niranjan Manohar, research manager for connected car and automotive IoT (Internet of Things) at Frost & Sullivan in Detroit. https://www.cta.tech/News/i3/Articles/2018/September-October/Human-Machine-Interfaces-Evolve-in-Cars.aspx ------------------------------ Date: Sun, 28 Oct 2018 12:51:47 +0800 From: Richard Stein <rmst...@ieee.org> Subject: Driverless cars: Who should die in a crash? (bbc.com) https://www.bbc.com/news/technology-45991093 "To get closer to an answer - if that were ever possible - researchers from the MIT Media Lab have analysed more than 40 million responses to an experiment they launched in 2014. "Their Moral Machine has revealed how attitudes differ across the world." With a software update, an AV "born" in China can be tuned for trolley problem "death" preferences anywhere, just like language locales for international-friendly applications. All the AV needs to know, per the "Moral Machine," are passenger/occupant ages and species. RISK: Does the AV have the "right" to act on its own volition if there are no human occupants or the passenger "species" are marginalized (insects or bacteria)? ------------------------------ Date: Tue, 23 Oct 2018 11:11:03 -0700 From: Lauren Weinstein <lau...@vortex.com> Subject: Every minute for three months, GM secretly gathered data on 90,000 drivers' radio-listening habits and locations (BoingBoing) via NNSquad https://boingboing.net/2018/10/23/dont-touch-that-dial.html On September 12th, GM's director of global digital transformation Saejin Park gave a presentation to the Association of National Advertisers in which he described how the company had secretly gathered data on the radio-listening habits of 90,000 GM owners in LA and Chicago for three months in 2017, tracking what stations they listened to and for how long, and where they were at the time; this data was covertly exfiltrated from the cars by means of their built-in wifi. The company says it never sold this data, but the presentation to the advertising execs was clearly designed to elicit bids for it. Unless they had explicit fully-informed consent from drivers, this should be -- and may have been -- illegal! ------------------------------ Date: Tue, 30 Oct 2018 10:53:50 +0800 From: Richard Stein <rmst...@ieee.org> Subject: Surgery students 'losing dexterity to stitch patients' (bbc.com) https://www.bbc.com/news/education-46019429 "A professor of surgery says students have spent so much time in front of screens and so little time using their hands that they have lost the dexterity for stitching or sewing up patients." Western medical training today emphasizes computer simulation over the "human touch" to learn the art. Simulated triage procedure rehearsals, especially from mass shooting incidents or industrial accidents, can help prepare medical team readiness. Would a surgical patient feel reassured to know that their physician learned colectomy or appendectomy exclusively by computer simulation rather than acquired via hands-on experience? Should surgeons be required to publicly disclose performance statistics: # of hours simulation practice for specific surgery, # of hands-on vs. robot surgery assists, # of computer-assist fatalities and incidents, etc.? Intuitive Surgical can cite this article to promote their da Vinci Surgical System. ------------------------------ Date: Fri, 26 Oct 2018 10:55:32 +0800 From: Richard Stein <rmst...@ieee.org> Subject: In Cyberwar, There are No Rules (Foreign Policy) https://foreignpolicy.com/2018/09/12/in-cyberwar-there-are-no-rules-cybersecurity-war-defense/ "If a country or terrorist group decided to take out a sitting U.S. senator undergoing robotically assisted surgery and then covered its tracks, the perpetrator's identity would be hard to pinpoint, and there would be no clear U.S. legal precedent for classifying the hacking of hospital equipment as an assassination or an act of war. Nor do there appear to be clear protocols for retaliation." A verifiable cyberweapons treaty urgently required to establish rules of conduct and preempt escalation. ------------------------------ Date: Sun, 28 Oct 2018 21:44:23 -0400 From: Gabe Goldberg <g...@gabegold.com> Subject: Lawmakers Seek Review of Pentagon Contract Thought to Favor Amazon (WiReD) Amazon long has been considered the likely winner of JEDI contract, as it is one of the only cloud providers with the infrastructure, funds, and security clearance necessary to meet all of the Pentagonâs requirements. The criticism is more acute because of the Pentagonâs insistence on awarding JEDI to a single bidder, rather than several companies and contractors. Both Oracle and IBM have filed official protests with the US Government Accountability Office, on the grounds that the DODâs decision to award the $10 billion contract to just one company both restricts innovation and poses a massive security risk. âJEDI turns its back on the preferences of Congress and the administration, is a bad use of taxpayer dollars, and was written with just one company in mind,â IBM General Manager Sam Gordy said in a statement in advance of JEDIâs bid deadline. https://www.wired.com/story/lawmakers-seek-review-pentagon-contract-thought-favor-amazon/ ------------------------------ Date: Fri, 26 Oct 2018 10:59:22 -0700 From: Rob Slade <rmsl...@shaw.ca> Subject: The customer is always right ... re: Apple iPhones A while back, users of older Apple iPhones started making noises about their phones being "throttled" and running slower. Turns out they were right. Apple had found that, for certain applications, if the batteries were older (and possibly dying) the demands of the application could cause the phone to simply quit, and stop working. So an upgrade to the operating system checked for these conditions, and, if the battery showed signs of failing, would dial back the CPU cycles so that the crash wouldn't happen. Trouble is, they didn't tell people first, didn't allow any options, and people got upset. Now, they probably did the right thing, technically. (Politically, it wasn't so smart.) And now an Italian court has decided they did the wrong thing, and has fined them. (They have also fined Samsung, which may not be guilty of anything, for the same thing.) https://nakedsecurity.sophos.com/2018/10/26/apple-and-samsung-punished-for-slowing-down-old-smartphones/ or https://is.gd/523V2E If this ruling stands, it's going to make deciding on upgrades and fixes a very complicated business. Politically. (It was already complicated enough, technically ...) ------------------------------ Date: Sun, 28 Oct 2018 15:46:23 -0400 From: Gabe Goldberg <g...@gabegold.com> Subject: Fun with source code (Medium) Why the NSA Called Him After Midnight and Requested His Source Code https://medium.com/datadriveninvestor/why-the-nsa-called-me-after-midnight-and-requested-my-source-code-f7076c59ab3d ------------------------------ Date: Mon, 29 Oct 2018 21:53:57 +0800 From: Richard Stein <rmst...@ieee.org> Subject: A Dark Consensus About Screens and Kids Begins to Emerge in Silicon Valley (The New York Times) https://www.nytimes.com/2018/10/26/style/phones-children-silicon-valley.html Mental illness traced to wireless mobile device (WMD) addiction has a label: The 'iDisorder.' See (https://www.nytimes.com/2012/05/13/business/in-idisorder-a-look-at-mobile-device-addiction-review.html for a book review. Excessive mobile device usage, induced by applications that easily captivate, is unhealthy. Children are especially susceptible to overuse. While there's no equivalent to the US Surgeon General's "Smoking causes cancer" warning, strictly enforced mobile device access restrictions for adolescents constitute wise parental guidance. The National Institutes for Health archives several studies on the physiological effects arising from excessive mobile device usage. "The Potential Impact of Internet and Mobile Use on Headache and Other Somatic Symptoms in Adolescence. A Population-Based Cross-Sectional Study" at https://www.ncbi.nlm.nih.gov/pubmed/27255862. "Conclusion: Results highlighted the potential impact of excessive internet and mobile use, which ranges from different types of headache to other somatic symptoms. Further studies are needed to confirm these findings and to determine if there is a need for promoting preventive health interventions, especially in school setting." "Evaluation of mobile phone addiction level and sleep quality in university students" at https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3817775/. "Conclusion: The sleep quality worsens with increasing addiction level. It was concluded that referring the students with suspected addiction to advanced healthcare facilities, performing occasional scans for early diagnosis and informing the students about controlled mobile phone use would be useful." ------------------------------ Date: Wed, 24 Oct 2018 16:35:22 -0700 From: Lauren Weinstein <lau...@vortex.com> Subject: When Trump Phones Friends, the Chinese and the Russians Listen and Learn (NYTimes) https://www.nytimes.com/2018/10/24/us/politics/trump-phone-security.html When President Trump calls old friends on one of his iPhones to gossip, gripe or solicit their latest take on how he is doing, American intelligence reports indicate that Chinese spies are often listening -- and putting to use invaluable insights into how to best work the president and affect administration policy, current and former American officials said. Mr. Trump's aides have repeatedly warned him that his cellphone calls are not secure, and they have told him that Russian spies are routinely eavesdropping on the calls, as well. But aides say the voluble president, who has been pressured into using his secure White House landline more often these days, has still refused to give up his iPhones. White House officials say they can only hope he refrains from discussing classified information when he is on them. So, Trump's cellphone use is being routinely monitored by our adversaries. Perhaps part of his plan? ------------------------------ Date: Tue, 30 Oct 2018 13:05:51 -0700 From: Gene Wirchenko <ge...@telus.net> Subject: "Apple appears to have blocked GrayKey iPhone hacking tool" (Lucas Mearian) Lucas Mearian, Computerworld | Oct 25, 2018 Apple and two companies that have worked to enable iPhone de-encryption continue their back-and-forth efforts. https://www.computerworld.com/article/3268729/apple-ios/apple-appears-to-have-blocked-graykey-iphone-hacking-tool.html selected text: Apple has apparently been able to permanently block de-encryption technology from a mysterious Atlanta-based company whose blackbox device was embraced by government agencies to bypass iPhone passcodes. Atlanta-based Grayshift is one of two companies that claimed it could thwart Apple iPhone passcode security through brute-force attacks. The blackbox technology purportedly worked, as Grayshift's technology was snapped up by regional law enforcement and won contracts with Immigration and Customs Enforcement (ICE) and the U.S. Secret Service. All GrayShift customers sign very strict non-disclosure agreements, as any leaked information could help Apple close the vulnerabilities they are using, whether they find them themselves or buy zero-day flaws in Darknet, said Vladimir Katalov, CEO of Russian forensic tech provider ElcomSoft. "Honestly, we are not absolutely sure that the hole has been completely closed; or maybe they will still find a workaround, or develop/buy another way," Katalov said via email. "So that is [a] cat and mouse game that is still ongoing. Now..., GrayShift will probably spend even more efforts to hide their findings from the media. "That is probably good for law enforcement, but definitely bad for the community, as it leaves some doors still open," Katalov added. "That's only a question of time when GrayKey will become available to some criminals." [The usual about the cat and mouse game. What I am wondering is whether those non-disclosure agreements are actually enforceable?] ------------------------------ Date: Thu, 25 Oct 2018 09:57:27 +0200 From: DJC <d...@resiak.org> Subject: Re: Toward Human-Understandable, Explainable AI (RISKS-30.88) We're wary about giving present-day AI the power to make decisions, partly because we don't know *why* it makes particular decisions, so its objectivity, fairness, common sense, etc., are opaque. At least where human beings decide, we can ask them the basis for their decisions. But as a matter of fact -- honesty and integrity aside -- humans aren't very good at knowing the grounds for their important decisions. Daniel Kahneman got the Nobel Prize for studying the reality of how people decide; cf. his book "Thinking, Fast And Slow". He and his colleagues did many, many experiments to expose the *real* bases for how people make decisions; and those bases are often not only unknown to their subjects, but impossible for them to know, because they happen in inaccessible processes of their cognition. Yet some of those processes can be exposed through careful experimentation over people's concrete behavior -- not what they self-report, but what they *do*. And that was worth a Nobel Prize. Kahneman acknowledges the impossibility of knowing everything about how one makes one's decisions, much less controlling it all. (In his book he proposes some personal strategies to ameliorate how bad it can be.) So what hope have we of transparency for the ever-more-complex AI mechanisms into which, even already today, we have no insight at all? Should we demand that, at a certain level of "importance", an AI system should be subject to the kind of concrete experimentation that Kahneman carried out in his research? How do we even know what to look for? Though I'm all in favor of the kind of transparency Hani Hagras proposes, I find it difficult to imagine how we can effectively grasp and achieve it. I can, though, imagine that if you're planning to do something of consequence -- possibly bad consequence -- that can be accomplished only through mechanisms neither you nor anyone else can understand, it may be time to step back and, simply, not do it. And that notion isn't new with AI. ------------------------------ Date: Thu, 25 Oct 2018 18:37:52 +0800 From: Richard Stein <rmst...@ieee.org> Subject: Re: Explainable AI Simulation for AVs Explainable AI (XAI), per http://catless.ncl.ac.uk/Risks/30/88/%23subj3.1 posits that (T)ransparency, (C)ausality, (B)ias, (F)airness, (S)afety characteristics must be demonstrable for an AI platform to establish a basis for triage and public comprehension of exhibited AI behavior. As a release metric, suppose that AV operational control program (OCP), the vehicular equivalent of an aircraft Operational Flight Program (OFP) has to demonstrate viability V = T + C + B + F + S == 5 (assigning 1 point for each XAI viability factor if it passes the stimulus/response pass-fail criterion, 0 if not), and don't publish the OCP bits until it does. Publishing with a viability score of 4, should (S)afety fail, implies significantly compromised XAI. Potential unexplained defect escape and elevated risk of AV OCP underachievement -- meaning public safety traffic incident frequency is likely to be higher, placing the AV's brand in jeopardy. Note: Release viability includes additional factors that I'm not being explicit about. Memory/descriptor leak, basic OCP function/operation, performance, payload/message passing, built-in-self test, behavior under sensor/processor error or fail-over conditions, etc. comprise a big "foundational" readiness component to deterministically achieve before attempting XAI qualification. Given a pile of GPUs or equivalent, construct a fictitious city-scape, that also has rural and suburban characteristics (buildings, fireplugs, houses, bushes, trees, parks, squirrels, etc). Have people, dogs, motorcycles, and other obstacles pop out into the driving surface, or on sidewalks at various distances/times, at controlled intersections, randomly/unexpectedly cross the street on bicycles, wheelchairs, scooters, skateboards, etc. Vary the weather conditions, terrain, pavement markers, hostile WiFi DoS stimulus, earthquakes, lighting, etc. Conceal obstacles or scenery, and then reveal it (remove billboards or restaurant placards), throw in some bicycles that swerve to avoid "dooring" incidents, or even experience "dooring" and toss out some tacoed bicycle wheels and prostrate bicyclists. Use buses, streetcars, street sweepers, free-rolling baby trolleys, swerving vehicles, ambulances/emergency vehicles, small aircraft landing, overturned fuel trucks, fiddle with the sound system, a/c, power seats, windows and door locks, sunroof, etc. The AV simulation's stimulus must generate real-time perspective images and sensor signaling content as detected/interpreted by LiDAR, BlueTooth, WiFi, RADAR, or whatever comprises standard AV sensor suites. Each stimulus condition must trace to one or more of the XAI viability attributes: T, C, B, F, S. Run the simulation for at least and equivalent of ~160 kilometers (100 miles) @ 60 MPH/100 KPH duration with stops, traffic jams, parallel parking, highway merges, varying speeds, etc. and process the log files to show that V is achieved unconditionally or with five or more nines reliability. Then randomly modify it, and run again and repeat, for a total of ~1.6Mhours to show V deterministically achieves or over-achieves the viability score threshold required to publish. Publicly release all the AV OCP simulation stimulus conditions and processing results for review. https://teslatap.com/undocumented/model-s-processors-count/ says a model-S has ~65 cores among its LRUs (line replaceable units) suite. Call it 100 cores to host LRU software stacks for sensor stimulus. That implies 100 cores x 100 inputs/sec = 10000 inputs for the cores to process and output per second. 10000 events/sec x 3600 secs/H x 1.6H = ~58M simulated sensor stimulus inputs to generate, process, and output log for one scenario. Assumes the AV OCPs landscape is pre-generated, save for random physical perturbations (weather, obstacles, etc). Each scenario must be reproducible to assist thorough triage and reconstruct anomalies that generate a viability score less than 5. The scenario generator would be a "work of art" unto itself. To complete OCP qualification by divide and conquer in 1 week (24*7 = 168H) of wall clock means ~9Kcores + memory + disk + net, etc. rigged for real-time processing. Feed a credit card to Amazon Web Services and provision a hunk of data center (GovCloud, r5 instances, reserved for 1 year, etc) gives ~US$ 42M for data center with 1000 GBs of network I/O. $42M/52 weeks ~= US$ 807K/data center week. Given this XAI simulator qualification scenario, the key question I think, is what objective criteria are used to specify and constitute T, C, B, F, and S for stimulus input and measurement? What standards are relevant, and should these factors be legislated and subject to regulation by an independent, conflict-free panel? If there's regulatory oversight for AV OCP pre-deployment qualification, would AV XAI be achieved under an ethically reasonable, publicly acceptable, and sufficiently rigorous process that entitles manufacturer indemnification against AV incidents and fatal accidents? Can any manufacturer engineer and achieve to XAI's expected qualification rigor? ------------------------------ Date: Tue, 5 May 2018 11:11:11 -0800 From: risks-requ...@csl.sri.com Subject: Abridged info on RISKS (comp.risks) The ACM RISKS Forum is a MODERATED digest. Its Usenet manifestation is comp.risks, the feed for which is donated by panix.com as of June 2011. => SUBSCRIPTIONS: The mailman Web interface can be used directly to subscribe and unsubscribe: http://mls.csl.sri.com/mailman/listinfo/risks => SUBMISSIONS: to risks@CSL.sri.com with meaningful SUBJECT: line that includes the string `'. Otherwise your message may not be read. *** This attention-string has never changed, but might if spammers use it. => SPAM challenge-responses will not be honored. Instead, use an alternative address from which you never send mail where the address becomes public! => The complete INFO file (submissions, default disclaimers, archive sites, copyright policy, etc.) is online. <http://www.CSL.sri.com/risksinfo.html> *** Contributors are assumed to have read the full info file for guidelines! => OFFICIAL ARCHIVES: http://www.risks.org takes you to Lindsay Marshall's searchable html archive at newcastle: http://catless.ncl.ac.uk/Risks/VL.IS --> VoLume, ISsue. Also, ftp://ftp.sri.com/risks for the current volume or ftp://ftp.sri.com/VL/risks-VL.IS for previous VoLume If none of those work for you, the most recent issue is always at http://www.csl.sri.com/users/risko/risks.txt, and index at /risks-30.00 Lindsay has also added to the Newcastle catless site a palmtop version of the most recent RISKS issue and a WAP version that works for many but not all telephones: http://catless.ncl.ac.uk/w/r ALTERNATIVE ARCHIVES: http://seclists.org/risks/ (only since mid-2001) <http://the.wiretapped.net/security/info/textfiles/risks-digest/> *** NOTE: If a cited URL fails, we do not try to update them. Try browsing on the keywords in the subject line or cited article leads. Apologies for what Office365 and SafeLinks have done to URLs. I have tried to extract the essence. ==> Special Offer to Join ACM for readers of the ACM RISKS Forum: <http://www.acm.org/joinacm1> ------------------------------ End of RISKS-FORUM Digest 30.89 ************************