Saving Everything?

Here is something to mull over -- my own field has long stressed careful selection, what we call appraisal, and I still believe in the concept, but this is something fun to debate. . . .

From the Chronicle of Higher Education issue dated February 9, 2007


On The Record, All the Time
Researchers digitally capture the daily flow of life. Should they?

By SCOTT CARLSON

The first day I came home with a digital audio recorder hanging around my neck, along with a sign that said "Warning: This conversation may be recorded," my wife shook her head in a way that conveyed deep embarrassment. Then she said: "Have you ever heard of Nixon?" On the train to work, my fellow commuters did double takes and snickered. People sat down next to me, then got up and moved to another seat. One woman saw my sign and hushed her friends: "Look, he's recording." Another fellow told me that someone was going to beat me up over my little stunt. Once a conductor gave me such a severe look that I thought he would throw me off the train at the next stop.

In the grocery store, clerks asked me if I was doing some kind of crazy university study. Sort of, I replied. "You're recording your life all the time?" people said. "Why would anyone want to do that?"

Good question. I had been asking a handful of researchers the same thing. Many of them are documenting all their conversations, movements, ideas, and correspondence through audio recorders, digital cameras, GPS trackers, pedometers, brain scanners, and other gadgets. The data they collect are mainly fodder for research, as they try to figure out how people might organize and make use of a deluge of information collected in business, medical, and social settings.

But they also predict that in the future everyone is going to be "lifelogging" — continually recording images from their vacations, conversations from business meetings, and even intimate confessions to friends.

"I fully believe that we will all be wearing this stuff all the time," said Mark T. Bolas, a visiting associate professor in the film school at the University of Southern California, as he hung a digital voice recorder around his neck when we met a few months ago. "The day before you die, your kids are going to look at you, when everybody else is doing this, and say, You mean you didn't record when you were growing up? You're just going to die and all of this is going to go away?"

Jim Gemmell, who leads lifelogging projects for Microsoft Research, says that we will one day glean information from our own lives the same way we now get information from Google.

"The personal corpus is going to enable you to be more reflective and come up with better ideas," he says.

If that sounds unreal, step back and consider how many digital pictures you took last year compared with how many photographs you took 10 or 20 years ago. Consider that the latest cellphones are equipped to record video and track your location. Think about the countless lives already documented on Web sites like Flickr, MySpace, YouTube, and Facebook.

A laboratory at Queen's University at Kingston, in Ontario, is working on a camera that starts recording when its wearer makes eye contact with another person — one of many lifelogging projects in academe. Later this year, major electronics companies will start selling gadgets and tools specifically designed for lifelogging, says Alex Pentland, a professor in the Massachusetts Institute of Technology's Media Lab and a pioneer of wearable computing. His lab is supported by some of those companies, and he is working on lifelogging tools that can decide, on the fly, whether a moment is worth recording.

But as they push into a future that is totally on the record, lifeloggers are also pondering and fretting over how this technology will alter society. What can lifelogging do for us? Could it improve scholarship? What are the legal risks? And how will lifelogging affect personal relationships, private conversations, and family histories?

I wanted to find out for myself. I went to an electronics store and bought a digital voice recorder for $100. I set up a time to talk with a lawyer who knew something about wiretapping (I had no desire to be a test case for lifelogging lawsuits), and I began to contemplate how life might change after I pressed Record.

Lifeloggers trace their history back to 1945, when Vannevar Bush, a prominent American scientist, wrote an essay for The Atlantic Monthly called "As We May Think." Scientists deal with an increasingly unmanageable trove of data and other information, Mr. Bush wrote, but technology could help. Mr. Bush imagined scientists wearing little cameras on their heads to record lab work. He conjured an image of a desklike machine that could store thousands of pages a day in microfilm. He called his device a memory extender, or "memex" — a term that some researchers use today to describe their own suite of lifelogging tools.

In the 1990s, MIT's Media Lab began dabbling in lifelogging through wearable computers, under the direction of Mr. Pentland and other researchers. Steve Mann, an associate professor of computer engineering at the University of Toronto who was once Mr. Pentland's student, wrote about his experiences wearing various recording devices in his book Cyborg. "One day," he wrote, "we will all feel naked without our wearable computers."

In the early 2000s, the federal Defense Advanced Research Projects Agency embarked on lifelogging research of its own. However, the project coincided with newspaper reports about the Total Information Awareness project — a huge government database that would have tracked Americans' electronic lives — which was causing angst among privacy advocates and the public. Darpa backed off and recast its grants to concentrate on lifelogging for soldiers on the battlefield.

Around the same time, Microsoft Research took up lifelogging with a project called MyLifeBits, which is devoted to figuring out how to store vast amounts of lifelogged information and how a lifelogger might find important kernels in a pile of chaff. The project's guinea pig is Gordon Bell, a 72-year-old computer engineer and entrepreneur who is recording every conversation he has, snapping pictures of every event he attends, saving every e-mail message he sends or receives, and even tracking himself through GPS, then archiving that information. So far, he has amassed some 160 gigabytes of data, more than the hard-drive space on most people's computers.

Mr. Gemmell says he has found that a lifelogger can home in on a recorded event — a conversation about a new idea with a colleague at a conference, for example — by associating it with other memories. He says it is not possible to search for specific words or phrases in an audio record, the same way you might search a document, to find a specific detail, at least not yet. But you might remember that the conversation happened in a specific city, on an overcast day, during the conference's opening reception. You could use GPS records, weather data, and your calendar entries to triangulate and the find the digitized "memory."

"We have so many little hooks in our memories, and we think associatively, so the more we bring in to the corpus, the more we find it's possible to find something," Mr. Gemmell says. In 1945, Mr. Bush said traditional library indexing, an "artificiality" based on letters and numbers, wouldn't work for his memex. Instead, the awesome "intricacy of trails" of memory would lead people to information.

But once you store all that stuff, what do you do with it? Microsoft Research offered grants of money and equipment to find out what university researchers could come up with. The proposals that came in surprised Mr. Gemmell.

Many of the winning projects deal with health. Doctors already record physiological data, like heart rhythms, through mobile devices, but researchers imagine that lifelogging tools could give physicians an even clearer picture of the factors influencing a person's health. Lifelogging itself could even be a form of therapy.

For example, Anind Dey, an assistant professor of computer science at Carnegie Mellon University, is using digital photos to try to exercise and improve the memories of early-stage Alzheimer's patients. He uses a lifelogging device invented by Microsoft Research called a SenseCam — a little black box, with a fisheye lens, that hangs around a lifelogger's neck and automatically snaps a picture about once a minute. A day recorded by a SenseCam looks like a stop-action film.

In the research, Mr. Dey is trying to figure out which visual cues are useful in prompting memories, and whether he can add other cues — like the day's temperature or information about an event — to trigger stronger ones.

Other researchers, like Alan F. Smeaton, are trying to find automated ways to navigate lifelogging data. Mr. Smeaton, a professor of computing at Dublin City University, in Ireland, specializes in video, and his research team has created programs that can scan video and automatically find, say, action sequences. They are now recording their days with SenseCams and creating programs that will scan the pictures to find significant changes in activity, which will help locate the most interesting events.

Along the way, his researchers have recorded the significant and the banal. One has an entire record of the first time he met his girlfriend and of their first date. Now and then a researcher forgot to cover up the camera when going to the restroom. Half-joking, Mr. Smeaton says he should invent a "bathroom detector or a white-tile detector" among the video-processing programs, so he can cut those scenes out of the record.

People have become accustomed to surveillance cameras that are used to combat terrorism in Ireland and England, he says, so he found that wearing the SenseCam was not controversial. But when he started dabbling in audio recording, many people objected, even the young geeks and techies in his department. "We had to have a 'whoops' button that deleted the previous minute," he says.

Audio lifelogging is socially taboo, legally treacherous, and, occasionally, emotionally jarring.

Daniel P.W. Ellis, associate professor of electrical engineering at Columbia University, knows this from having recorded his daily life for almost two years. About half the people he met were uneasy or annoyed when he told them he was recording them. His wife banned logging at home because she found it "creepy," he says.

Like Mr. Smeaton, Mr. Ellis is developing ways to pluck out useful clips from all that audio. Through analysis of the background noise, he is able to figure out where he was at any particular time — walking on the street, lecturing in class, meeting in his office, riding in the elevator, and so on. Then he can match that analysis with his electronic calendar to get a fairly accurate diary of what he was doing and whom he was talking to over the course of weeks or months. He can also click on the calendar entry and listen to the audio.

He sees his system as a powerful organizational tool, like an automated datebook. He recently met with officials at Google about his work, but nothing has come of it — yet. "They were interested," he says. "Some people there obviously felt that it fell within their scope."

Lately he has become interested in keeping the recorded conversations private. He has found that he can scramble the voices so the conversations are unintelligible but the speakers are still recognizable to the computer.

Privacy is not an abstract issue for him. His recordings, now archived on a pile of compact discs, sit on a bookshelf in his office. He took them out and set them on the table as he discussed his work.

"There are times when I think we're out of our depth here," Mr. Ellis says. "There are things that you don't want people to know about you and things you don't want to know about yourself."

Once, for example, Mr. Ellis and his wife got into an argument as they were leaving for vacation. That night he realized he had recorded the fight, which is somewhere in that pile of discs. "I never went back to listen to it, but there exists that recording of me being an asshole," he says. "I don't particularly like the idea of that existing."

On another day, his toddler fell down some stairs and hit his head on a stone floor. Mr. Ellis and his wife rushed their son to the hospital, and as Mr. Ellis was talking with the doctors about his son's condition, he realized that he was recording the whole event.

"We were in the waiting room, making conversation, and the doctor said, 'What's that on your belt?' I said, 'Oh, that's my MP3 player,'" Mr. Ellis says. "For all kinds of reasons, it didn't seem like something I wanted to raise at that point. When you're in that situation, it's like, Please just help me. I don't want to threaten you at all."

His son was fine, but Mr. Ellis has no desire to relive the memory. "The whole sequence of him falling exists in these recordings — him freaking out, and us saying that we've got to get him to the hospital. That's something that I've never gone back to listen to. I could, but it would be too upsetting."

I told Dan Ellis that I was planning on recording my own life for a few weeks, and he said that I would learn what he was talking about firsthand. Before I left, he offered to process my recordings when I was done.

If you're wondering whether lifelogging is legal, the answer is that it seems to be — at least in most settings, most of the time.

Emilio W. Cividanes, a lawyer in Washington who specializes in wiretapping issues, told me that I needed to consider wiretapping and eavesdropping laws when it came to recording my life. In most states, you need the consent of at least one party when recording conversations, but in 12 states, including Maryland, where I live, you need the consent of both parties.

However, says Mr. Cividanes, the laws in most states say that people in public places should not necessarily have an expectation of privacy, and therefore may be recorded, even without their consent. (Lifelogging in Illinois, Massachusetts, and Oregon would be risky, as those states have both strict laws for consent and no allowances for recording in public, he says.) Any legal case would rest on facts and circumstances, he explained.

I decided to play it safe and hang a sign around my neck announcing my recording to the world.

I was a freak. At the farmers' market, the man who sells Communist Party newspapers picked me out right away. "So if I told you my name" — and he told me his name and some information about himself — "you would record all of that?"

"I just did," I said.

Out in public, no one asked me to turn off my recorder, but few people went out of their way to talk to me. In the office, colleagues asked me to turn off the recorder every other day, usually to relate a juicy bit of gossip or gripe about some office drama. Journalists are accustomed to the conventions of going off the record, even in private life.

My wife, who is also a journalist, banned recording at home for the first week because she said I acted like I was "on stage." I had noticed that, too. I never really forgot that the recorder was on, and now and then I sensed I was talking differently, as if to a crowd. I consciously avoided saying things that might be deemed politically incorrect or downright gross, although some of that slipped out and into my memex.

One weekend I got tired of wearing the recorder and put it in a drawer. I felt liberated in a way that is hard to describe. That Sunday I found myself pacing the house and whispering to no one — something I often do when I'm alone and trying to work out ideas for stories I'm writing. I realized I rarely did this when I had the recorder on. It was like I was afraid someone would catch me acting schizophrenic.

But I'm probably the only person who will ever listen to the recordings, so what was I worried about?

Jeffrey Rosen, a law professor at George Washington University and an expert on privacy, explains my anxiety through a concept from Jewish law called hezzek re'iyyah, or "the injury caused by being seen." Jewish law says that the mere possibility of unwanted observation, even if no one is really watching, injures a person's sense of privacy.

"It's the uncertainty about whether our intimate moments are going to be observed that makes us more inhibited and less likely to let down our hair," Mr. Rosen says.

We act differently depending on whether we're with our family, our friends, or our business associates, a social flexibility "necessary to cultivate the bonds of intimacy," Mr. Rosen says. "If the goal of the lifeloggers is to record real life in intimate and formal contexts, they would have to be defeated in some respects because the candor would dry up."

Anyone who has ever thought seriously about privacy would shudder at the thought of a lifelogged world, Mr. Rosen says. "The standard techno-positivist enthusiasm — that this is inevitable and that we should get used to it — is wrong and dramatically understates the social cost of this sort of technology," he says. Louis Brandeis, the Supreme Court justice who wrote the most famous treatise on privacy in American law, "would be turning over in his grave," Mr. Rosen says.

Science fiction has warned us of this future. In his Hugo Award-winning book Hominids, Robert J. Sawyer created a parallel universe in which Neanderthals (that world's dominant and highly advanced humanoid species) wear recorders on their arms that upload data to an "alibi archive." The society is free of crime, but even innocent people face punishment if they don't have alibis.

The Final Cut, a 2004 film starring Robin Williams, depicted a near future in which microchips implanted in human brains record everything a person sees and hears from birth. After death, a "cutter" edits those memories into a flattering biopic for friends and family, finding and deleting disturbing images of, say, spousal abuse or child molestation. "How can you handle it?" a retired cutter says to a workaholic colleague. "People lying to each other, manipulating each other. The obscenity!"

Lifelogging researchers are well aware that their work treads philosophically, socially, and legally treacherous territory, and they take steps to protect their data. Mr. Bolas, of Southern Cal, considers other people's words their property, and he stores his files on a hard drive that is off the network and safe from hackers.

But who knows what the future holds? Leana Golubchik, an associate professor of computer science at the University of Southern California, has raised a nightmare scenario among lifeloggers in which authorities can subpoena details about private lives from lifelogging archives. Others have wondered whether lifelogging copyrighted information, like going to a movie, or picking up songs on the radio, could put someone at risk for infringement.

The lifeloggers have grappled with how technology and society might adjust to their new world. Perhaps lifeloggers could program their gadgets to record only those people who grant permission, and the machines could scramble everything else. Perhaps society will just come to accept pervasive surveillance and the end of plausible deniability.

At the same time, the researchers are interested in how lifelogging might change society in positive ways.

"Nobody could ever lie again," says Mr. Bolas. "Nobody could ever commit a crime again." In the lifelogged future, "a verbal contract is just as good as a written contract."

Jane Greenberg, an associate professor of information and library science at the University of North Carolina at Chapel Hill, is studying the ways that undergraduates use lifelogging tools to enhance their learning in a biology course. She imagines that lifelogging would allow her to be more productive, to reflect on her work, and to review forgotten ideas from the past.

There might be personal and historical advantages, too — something Ms. Greenberg and her colleagues wonder about together. "Wouldn't it be cool to have your grandfather's memex?" she says. "And wouldn't it be cool to have your memex and hand it down to somebody? This is me, these are my recordings."

Or, she wonders, what if future world leaders or other important figures had memexes? "It would be interesting if our president had one on," she says.

My father frequently tells a sentimental story about a day when he, as a 6-year-old, woke up before dawn to find my grandfather getting ready for his job on the kill floor at a meatpacking plant in St. Paul. My father asked my grandfather if he liked to work. "Not really," my grandfather replied flatly. "But there are just some things you have to do to take care of your family."

My father cherishes that memory for what it says about my grandfather's sense of loyalty and duty. Of course, it may have never happened. My father could have imagined it as a boy and later come to believe it was a memory. There was no recorder hanging around my grandfather's neck, so we'll never really know — and we may not want to.

In his book Mystic Chords of Memory, Michael Kammen, a historian at Cornell University, points out that people are in a constant process of forgetting, recasting, and re-remembering their histories to reflect and reinforce their values and beliefs.

In a lifelogged world, people would probably continue to romanticize the past, Mr. Kammen says in an interview, but lifelogging would operate as a reality check on some events.

He points out that technologies for recording everyday life go back as far as George Eastman's Brownie camera, which came on the scene more than 100 years ago. "Microhistory of individuals and families is not a new phenomenon, but [lifelogging] is a fascinating example of new technologies making it possible to do this in a deeper and richer way," he says. "What these people are doing is going to be a treasure trove for historians."

That is not a universally held point of view among historians, however. Jill Lepore, a professor of history at Harvard University, says some of the appeal of her profession is the intrinsic mystery of people and the stories she can pull together from scant evidence.

"There is no part of the sensibility of total recall of the minutiae of my life that appeals to me, and encountering another human being through that medium as a researcher feels a little unsavory," she says.

"If I could know what George Washington was thinking when he wrote his will, emancipating his slaves — sure, I would like to know that," she says. "Would I want him exposed to me in a way where I couldn't even have the curiosity of that question? ... It seems horrible."

Even if lifelogging catches on — and Ms. Lepore is skeptical — she wonders whether the records would be of any use to historians. Statesmen and other important figures are not likely to leave an unfiltered archive behind, she says. "Those are the people who historically burn their papers or have their wives burn them when they die."

I thought I would erase my electronic memory when I was done with my experiment. Over a few weeks I had amassed a mess of files — some stored at home, some at the office — that amounted to about 200 megabytes for each recorded day. The files contained a lot of dead air, as the recorder captured me typing, driving, or walking. But the files also contained embarrassing moments and secrets that belonged to me and my friends. I decided I could not send them to Mr. Ellis at Columbia for processing. I knew that destroying them would be the safest thing to do.

But so far, I haven't had the will. The recorder had captured funny stories and moments that were worth saving. It had logged night after night of reading books to my 4-year-old son before bed and some of the funny things he said out of the blue, and may never say again. Somewhere in its memory is the moment I heard my baby daughter say her own name clearly for the first time — I just haven't been able to find it yet. I wondered whether my kids would want a record of a day in the life of their father when they got older.

Mr. Ellis said he experienced a "nostalgia effect" when he listened to some of his old lifelogs, and even after just three weeks I felt it. On the weekend I left the recorder in a drawer, my wife and I got to talking about what our son was like when he was a baby, and we realized that even after a few years we couldn't remember what he looked like or sounded like, or the things he said when he first started babbling. It was one of the few times I regretted not wearing the recorder more often.

In the end, I saved my lifelogs onto a disc, then hid them in my home. Maybe someday I will comb through them, pick out the good parts, and destroy the rest. Unless an audio-scanning program for lifeloggers hits retail shelves soon, that project could take weeks. For now, I'll hold these memories close.

http://chronicle.com
Section: Information Technology
Volume 53, Issue 23, Page A30