Benevolent Spinoffs in Technology: Smartphones and the Deaf Population

 

At the town recreation center where my wife and I take exercise from time to time, one day we passed a man who was using American Sign Language (ASL).  That in itself is not too unusual in Central Texas, as there is a long-established Texas School for the Deaf in Austin, and as a result we have a higher-than-average concentration of deaf and hearing-impaired people in the area. 

 

What was unusual is that he was by himself, directing his gestures and facial expressions to his smartphone, propped on a chair next to him.  It never occurred to me until that moment that the video-call feature is a huge step forward for deaf individuals who use ASL.  As long as the person at the other end also knows ASL, there's no need for any intervening interpreters or translation functions:  just look and talk, and this guy was enthusiastically doing just that.

 

This was the vision of some Cornell researchers back in 2009, when older and slower 2-G phone systems were barely able to handle still images, let alone the 10-frame-per-second or so data rates needed to convey ASL efficiently.  Sheila Hemami, a professor of electrical and computer engineering, led a four-year effort to squeeze video of both hand gestures and facial expressions over the limited bandwidth then available.  By 2009, they were testing 25 prototype units with deaf individuals in the Seattle area, and hoped that their pioneering efforts would take hold and spread.  Of course, when wider bandwidths and videoconferencing came along, the problem almost solved itself.

 

Even back then, deaf people could send text messages, and in fact that is the only way they could use the telecommunications network for many decades.  Because most people have normal hearing, the big priority for deaf people is to communicate with a hearing person who has no knowledge of ASL.  With the advent of the telephone, the only way a deaf person could use it  at first was to sign to a hearing person, and ask the hearing person to make the phone call.  This works, but requires a resident hearing person to help you.

 

The next step came when a deaf physicist and amateur radio operator, a deaf orthodontist, and the grandson of the founder of Saks Fifth Avenue used some surplus teletype machines to demonstrate typewritten communications over ordinary phone lines.  This was in 1964, and in the succeeding decades, so-called TTY communication became a kind of standard for deaf people.  Service agencies sprang up who would read the TTY messages over the phone to hearing people so a deaf person could use their TTY to communicate with anyone who had a phone.  This was a great step forward, but still required the deaf person to invest heavily in a specialized set of equipment and hire a service agency to do something that hearing people could achieve with only a telephone.  Nevertheless, it became quite popular in the deaf community for all sorts of uses.  In a discussion thread on Reddit, a former TTY agent, as they were called, expressed amazement at how many deaf persons were calling sex chat rooms when he worked for the agency.  That would certainly pose an ethical dilemma for the agent, but common-carrier law back then would prevent the agency from turning down such business, I suppose.

 

When text messaging became feasible directly on one's mobile phone, deaf people seized on it eagerly, even back when one had to use a 10-key pad and multiple taps to encode messages.  Things got a lot easier with the advent of touchscreen phones and the bandwidth improvements that came along with 3-G and 4-G systems.  For communicating with hearing persons, there are now Video Relay Service (VRS) companies which employ ASL interpreters to relay messages from an ASL user to someone who doesn't understand ASL.  And with the advent of AI systems that are fluent in video images and sound as well as text, it is only a matter of time before someone develops an AI app that will at least supplement the work of a human ASL interpreter, and possibly become good enough to be used without human assistance at all. 

 

The use of videoconferencing for ASL was not the target application when smartphone developers set out to make video over phones possible.  But deaf people were the unintended beneficiaries nevertheless, and unintended uses turn out to be some of the more interesting byways in the history of technology.  Ronald Kline, a historian of technology at Cornell, has written extensively about how farmers adapted Model-T automobiles to do everything from plowing to grinding corn.  While Henry Ford wasn't planning on this, he was probably pleased if it meant more Fords were sold. 

 

This is one reason why engineering can never be an entirely theoretical arm-chair exercise.  Engineers have to get their prototype products out in the field where people they don't know will use them in ways that their designers can't imagine.  Sometimes these ways will cause harm, and that is why prototype testing with safety in mind is so important.  But almost as often, people will find beneficial uses that the developers never thought of, and what begins as a minor side use can become the main use after a while. 

 

The most prominent example of unintended uses I can think of came about with the development of the Internet itself.  While certain far-seeing individuals anticipated access to huge amounts of data by consumers (Vannevar Bush's 1945 "Memex" paper comes to mind), it seems that almost nobody prior to 1980 or so anticipated the huge increase in commercial applications of highly interconnected computer networks.  From early email, which was mainly a peculiar way of evading the long-distance telephone network used by a few physicists to exchange technical data at a distance, the Internet has now become a well-nigh-essential utility for billions of consumers, and sustains the economies of the world.  While it has its significant drawbacks (the maleficent influences of social media and Internet-based hacking come to mind), overall the Internet and its spawn have proved to be beneficial to humanity in myriad ways that its early developers and users never anticipated.

 

So I'm glad that members of the deaf community can now make phone calls to their deaf friends with as much ease and facility as their hearing counterparts can—and all without bothering anybody at the next table in the restaurant.

 

Sources:  The 2009 Cornell research is described at https://news.cornell.edu/stories/2009/12/researchers-create-cell-phones-sign-language.  I also referred to the websites https://nagish.com/post/how-to-use-telephone-if-deaf, about the history of TTY at https://www.smecc.org/tty___tdd_history_and_resources.htm, and discussions about the various technologies at the Reddit site https://www.reddit.com/r/CasualConversation/comments/ci1pbr/i_just_saw_a_deaf_person_having_a_phone/. 

Will Apple's Vision Pro Be the Next iPhone?

 

Back in June of 2023, Apple announced its Vision Pro, which the Wikipedia article about it calls a "mixed reality" headset.  This week, in some parts of the world you can now buy your own Vision Pro—for $3,500.  While this will not be an obstacle for wealthy early adopters, the rest of us will probably wait until the beta-version bugs are worked out and the price comes down.  In the meantime, we can think about what this means for the future of humanity.

 

That sounds either presumptuous or silly, but there is no question that the advent of the smartphone has changed the course of world history, especially cultural, social, and political history.  Combined with the AI-fueled algorithms that maximize profits for Facebook, X, and their ilk at the expense of rational political discourse, we have seen the smartphone severely damage democracy in the U. S. and other places.  Yes, there are advantages to smartphones as well, but a serious debate over whether having them is a net gain or loss to society is one that we will probably never have, because they are here to stay. 

 

That is not yet the case for the Vision Pro, so let's spend a little thought on imagining what life would be like if Vision Pro headsets or their upgraded equivalents become as common as smartphones.  My speculations are aided by my watching an 8-minute video made by Joanna Stern of the Wall Street Journal, who went to a cabin at a ski resort with some video producers and wore a Vision Pro for most of 24 hours.

 

When you wear a Vision Pro, your entire visual field is mediated, in a literal sense.  You can't see anything directly.  All you see is a projection of two high-resolution video screens that go directly to your eyeballs.  In order to see anything, including the ordinary world around you, you have to use the multiple cameras mounted on the Vision Pro.  Everything you see goes into the cameras, through Apple's proprietary software and some of the 600 apps now available for the device, and only then do you get to see anything.

 

And it works the other way too.  Physically, the Vision Pro looks like a pair of unusually bulky ski goggles, with a headstrap to keep it on and a fanny-mounted battery pack that has to be recharged every two or three hours.  The outer surface of the goggles is also a video screen, and in order to present something other than a blank shiny surface to someone the wearer is talking with in person, the screen presents video images of the wearer's eyes.  This is after the wearer has taken a photograph of her or his entire face, so the system knows how to present a somewhat reasonable facsimile of the wearer's visage.

 

Videoconferencing is one of the big intended uses of Vision Pro, but you can't just point a camera at a roomful of people wearing bulky headsets that cover their faces.  Apple to the rescue—the 3-D photos of the wearer stored in the system are used to create "avatar" faces to present to the other people in the videoconference.

 

From all the reactions to Stern's avatar that she accumulated in her video calls using the Vision Pro, there was one unanimous opinion:  her avatar looked terrible.  Even Apple has not yet overcome the "uncanny valley" effect in trying to use computing to simulate the human visage.  According to the uncanny valley hypothesis, unless a human simulation is extremely authentic (the good side of the valley), people will sense that something is off and have a negative reaction to it.  At the other side of the valley, a cruder image is seen as merely cartoonish and not uncanny.  Maybe Apple should have gone that route, as most people would prefer to see an obviously artistic caricature of a friend, rather than an image that is like something that an undertaker might manage to do with a corpse.

 

That was probably the worst experience Stern had with the device.  Although Apple doesn't recommend cooking while wearing the Vision Pro, Stern went right ahead and chopped onions, and was delighted to find that the airtight seal around her eyes prevented her eyes from watering.  (Chopping onions in a pan of water, I am told, is just as effective, and $3,500 cheaper.)  And the 3-D movies available from some (not all) streaming services were impressive. 

 

You can record your own 3-D videos with either the Vision Pro or the latest iPhone (15, I believe), and Stern tried this feature out while skiing, another activity that Apple doesn't recommend for Vision Pro wearers.  But nothing bad happened on her bunny-run venture down the slopes, and the overall impression Stern left with her viewers is that this is still a prototype, but if they work out some bugs and get the battery life up and the power consumption down, along with the price, Apple may have finally found what Google tried to find with Google Glass and failed to do back in 2015:  a mass market for what most people still call virtual-reality or augmented-reality headsets.

 

Apple avoids both of those terms and insists that what the Vision Pro allows is something they call "spatial computing."  To my ears, this is a singularly unfortunate phrase, because it implies that the computer uses space somehow to calculate things.  Well, every computer that takes up space does that, so it's just going to be a label for the 3-D techniques that the Vision Pro allows you to use for setting up your workspace. 

 

Wearing a Vision Pro really cuts you off from ordinary reality in a much more radical way than using a smartphone does.  Everything that you see passes first through the guts of the machine, rendering your entire visual field subject to the whims of the Vision Pro designers.  Perhaps that sounds benign now.  But put this device in the hands of criminals, or even well-intentioned entertainers who simply want to thrill people, and it may open entirely new fields of horrors.  It's too early to tell, but there will be downsides, especially if the Vision Pro proves as popular as Apple hopes.  Let's just hope the downsides aren't too low.

 

Sources:  I referred to an Associated Press article on the commercial introduction of the Vision Pro at https://apnews.com/article/apple-vision-pro-spatial-computing-augmented-reality-7ec545a42403cf12e799200864e47d94, Joanna Stern's video report on it at https://www.youtube.com/watch?v=8xI10SFgzQ8, and the Wikipedia article "Apple Vision Pro."

Apple Versus the Feds: How a Smartphone Stymied the FBI

When Syed Farook and Tashfeen Malik died in a hail of gunfire last December 2 after killing 14 people at a San Bernardino office party, the FBI recovered Farook's iPhone within a few hours.  One of the critical unanswered questions about the San Bernardino shootings is whether the couple had outside help, and the data on the iPhone may hold the answer.  Problem is, the FBI can't get at the data, and Apple, the iPhone's maker, won't help them.

Why not?  Let's let Tim Cook, CEO of Apple, answer that one:  "[T]he U.S. government has asked us for something we simply do not have, and something we consider too dangerous to create. They have asked us to build a backdoor to the iPhone."  A little historical perspective is in order to put this situation into context.

With the advent of powerful digital computers, advanced encryption algorithms were designed and adopted by both sides of the Cold War (both the U. S. and the Soviet Union) for secret communications in the 1970s and onward.  The U. S. National Security Agency, long used to spying on analog communications in which good radios were the most elaborate equipment needed, found itself behind the technology curve and spent millions on advanced computing technology to maintain its ability to crack enemy codes.  The computing power of those early NSA computers now resides on your smartphone, and after a run-in with NSA a few years ago involving spying on Apple, the tech company and its president resolved to do a better job than ever in protecting its customers' privacy.  The latest iPhone operating system has a feature that not only encrypts the user's private data, but destroys the internal encryption key if it detects more than 10 attempts to unlock the phone using the 4-digit password.  After that happens, nobody but God can retrieve the data. 

At first the FBI was hoping that the phone was backed up to the iCloud, where the data might be recovered.  But it turns out that the automatic backup feature was turned off last October, possibly by Farook to avoid just such snooping.  After trying everything they could think of, including things Apple suggested, the FBI has asked Apple to do something that the firm claims is unprecedented. 

The FBI wants Apple to write a new operating system for Farook's phone that will allow unlimited password tries electronically, which will allow the FBI to access the phone's data.  They say it will only be used on Farook's phone, and so there is no risk to anybody else's phone.  The FBI has put this request in the form of a court order, and Tim Cook has vowed to fight it.

Why?  Apple claims the risks of that system getting loose, either accidentally or by command, are simply too great, and they have dug in their heels.  For example, it has been suggested that once it becomes generally known that Apple has developed such a backdoor, repressive regimes will order the firm to give it to them, or else kick Apple out of the country.

This is not the first time that Apple and the federal government have been at loggerheads over encrypted data.  In a 2014 case, Apple was ordered to extract data from an iPhone, but it is not immediately clear from the record whether they complied.  In both that case and the San Bernardino situation, the FBI cited as its authority the All Writs Act of 1789, which basically lets courts issue writs (orders) "necessary or appropriate in aid of their respective jurisdictions and agreeable to the usages and principles of law."   To the ears of this non-lawyer, it sounds like the law basically says you can do whatever you want, but the Act is typically hauled out as a kind of last resort, as subsequent case law has erected a set of four conditions that must be fulfilled before a court can issue an order under the Act.  Of course, the FBI thinks the conditions are fulfilled, and Apple doesn't.

Apple's stand is based on the idea, not that common among high-tech companies, that even Apple doesn't have any business with your personal data, which is why they designed the iPhone operating system to be so hard to crack.  This differs from practices of other firms, who happily mine their customers' private data for commercially valuable things like brand names and so on.  Privacy advocates from across the political spectrum have joined Cook in his opposition to the order, and the outcome of this case could have wide implications not only for the FBI and smartphones, but for digital privacy generally.

National Review commentator Kevin Williamson (from whose column I first learned about this matter) takes the view that the FBI is taking the easy way out by simply ordering Apple to do its job.  There is evidence to support this claim.  For example, in its instructions to Apple, the FBI asked them to rig a bluetooth link to the phone so they could try the 9999 different number combinations electronically, instead of having to make somebody sit there and do it by hand.  This apparently minor detail has the aroma of a royal order to underlings—"and while you're at it, fix it so I don't mess up my manicure wearing my fingers out on that touchscreen of yours."  Back in the days of telephone hacking in the 1960s, teenagers with time on their hands would amuse themselves by dialing all 9999 numbers in a given 3-digit telephone exchange (e. g. 292-0000 to 292-9999) just for the thrill of discovering the test and supervisory numbers the phone company used for long-distance routing and maintenance.  Apparently, the FBI can't be bothered with such tedium.

The matter is in the hands of lawyers now, and if the issue does indeed go all the way to the Supreme Court, its fate may well depend on whether President Obama gets to appoint a new member after Justice Scalia's recent demise, or whether the next president does, or whether a split Court ends up doing nothing (split decisions leave the lower court's decision standing).  Whatever happens, I admire Tim Cook for taking a principled and consistent stand for a cause that he could so easily abandon:  the notion that privacy still means something in a digital age.

Sources:  Kevin Williamson's column "Hurray for Tim Cook" can be found at National Review Online at http://www.nationalreview.com/article/431491/apples-tim-cook-right-resist-governments-demand.  I referred to articles by ABC News reporter Jack Date carried on Feb. 19, 2016 at http://abcnews.go.com/US/san-bernardino-shooters-apple-id-passcode-changed-government/story?id=37066070 and Feb. 17 at http://abcnews.go.com/US/fbi-iphone-apples-security-features-locked-investigators/story?id=36995221.  I also referred to an article in The Guardian online at http://www.theguardian.com/technology/2016/feb/19/apple-fbi-privacy-encryption-fight-san-bernardino-shooting-syed-farook-iphone, and Wikipedia articles on encryption software and the All Writs Act of 1789.

Apple, Google, and the Mystery of Music

Back in the dark ages of music reproduction, when state-of-the-art was a plastic box called a cassette tape, yours truly worked for a few summers between college semesters at an audiovisual repair shop.  Most of my time there, I spent in the back room trying to fix recalcitrant tape recorders and players, but once in a while we were short-handed and I actually had to deal with a customer.  One guy I remember in particular.  He had brought in a strange-looking cassette player shaped and colored more or less like a volleyball, and its guts were not cooperating with my attempts to fix it.  One day he stopped in and demanded to see the guy who was actually working on his unit.  I came out to the front counter and tried to explain the problem to him.  He listened patiently, and then all he said was, “Man, I gotta have my music.”

Multiply him by the seven billion or so people on this planet, and you can realize some of the consequences of the fact that music is a universal feature of human culture, and the scale of money to be made by whoever provides music to these masses.  Before mechanical sound reproduction was possible, this was by necessity a retail operation.  Either you made music yourself, which is still the main way to hear music in many parts of the Global South, or you supported a few specialists called musicians, who could entertain as many people at once as could hear the sounds they made—a few thousand at a time at most.  Then came the phonograph, radio, magnetic recording, digital audio, the iPod, and most recently, according to Wired.com, Google and Apple’s attempts to dominate the global supply of streaming music to smartphones. 

Wired reporter Matt Honan admits that music is now a commodity, like gasoline or pork bellies.  The important thing about a commodity is the bulk price that  the distributors pay to the suppliers, which Apple reportedly offered to set at six cents per 100 songs streamed.  If Apple prevailed, that would mean for every song streamed to an individual listener, the entity formerly known as the record label would receive six ten-thousandths of a dollar.  Just a few years ago, when 45-RPM discs were still being sold, royalties might have amounted to as much as a tenth of the cost of a record, say maybe sixty cents.  Of course, you could listen to a record over and over again, but the same is often true of streaming audio.  The point here is that the cash return to the label for each listen has diminished almost to the vanishing point. 

It is not clear that Apple’s low bid will win out, but something close to it probably will.  And in the nature of digital distribution, we will probably end up with one major worldwide supplier of music to the masses, just as Google is the major supplier of search-engine technology.  Is this a bad thing?

It depends on what you think music is for.  Many people I know, especially younger ones, but even including my wife (who turns 57 next Tuesday), view music as a sort of background accessory to their lives, like certain styles of clothing.  The music they listen (or listened) to in their 20s and 30s becomes a part of them, and they select it to accompany or even induce certain moods of relaxation, pleasure, and so on.  For many people, like the gentleman I waited on at the repair shop, life without music is unimaginable, and it forms a continuous undertone to their lives at work and home.  If this is all that music means to you, then it doesn’t matter that much who supplies it, just as the name of your electric utility company probably doesn’t matter to you as long as the price is reasonable and the delivery is reliable.  That’s what dealers in commodities do:  deliver the goods reliably at a reasonable price.

But if you believe music is the most direct path to one’s emotions, in a manner of speaking a direct line to the soul, and can represent the abstraction called beauty, which is one of the “three roads to God” (the other two being truth and goodness)—well, then, the matter becomes a little more serious.  While it is unlikely that Google, Apple, or whoever ends up in charge will discriminate against particular styles of music if such discrimination reduces the bottom line, we can expect censorship to become an issue, especially in countries or regions where restrictive regimes hold power.  In the past, revolutionaries and freedom fighters have used music to great advantage.  In the classic World War II film Casablanca, no one can forget the stirring cafe scene in which a group of German soldiers singing “Die Wacht am Rhein” are drowned out by the French patriots singing “La Marseillaise.”  Say Google becomes the music supplier of choice in China, and the government decides that anything with religious or political overtones is not allowed?  Google and Yahoo and other high-tech firms have already kowtowed to the demands of governments for censorship with regard to search-engine results, and why should songs be any different?

Fortunately, anyone with a pair of lungs and vocal cords can make some kind of music or other.  And even now, in secret house churches, in African villages, and anywhere two or more people decide to sing together, the mysterious thing called music is doing its work in putting human beings in touch with the transcendent.  And we don’t need Apple's or Google’s help with that.

Sources:  The Wired.com story “Why There Are So Many Streaming-Music Rumors Right Now” appeared on Mar. 8, 2013 at http://www.wired.com/gadgetlab/2013/03/why-there-are-so-many-streaming-music-rumors-right-now/.  My thoughts on music drew inspiration from a talk (downloaded from iTunes!) by Peter Kreeft entitled “Beauty,” which is available at http://www.peterkreeft.com/audio.htm.  And the name of the 19th-century patriotic song that the Germans sing in Casablanca is disclosed at http://www.vincasa.com/indexwacht.html.