home | archives | about us | feedback

Daily Relay

Tracking trends and development in digital culture

special section:
Mindjack Film
Fresh thinking on current and classic cinema

Mindjack T-Shirts
Only $15US
shipping included


Mindjack Release
Sign up to receive details of new issues

photo by
Martin Rotovnik

January 26 , 2005 | Part Three: Me and My Data Shadow

Theory is good; observation is better. One of us (Pesce) has been observing the growth of the Internet, user communities, and swarms for nearly 20 years, while the other (Fraser) has deep experience concerning the issues of mobility and usability. The concrescence[30] of observation and experience has lead us from theory into practice. We are presently designing technological probes, testing the theses we have laid out in this paper, in order to concretize our understanding.

One of us (Fraser), as the former Director of User Experience for Hutchinson Telecoms Australia, has a strong understanding both of the desires and the frustrations of mobile telephone users. These real-world insights have guided us into designs that require a minimum of user intervention. In other words, we have done our best to design things that “just work,” without a lot of care and feeding. An example of this philosophy, created by one of us (Fraser), is ImageShow, a Java J2ME application which fetches images from the photo sharing site Flickr, allowing the mobile to dip into “Flickrstream” for an endless supply of imagery, filtered by user name and meta tags:

What follows are three speculative use cases, shown principally from the user’s point of view, and drawn from the themes outlined in this paper.

Example One: Active Listening

Starting from the basic proposition that the mobile telephone ought to do everything within its power to be intensely aware of its environment, the first use case involves the emergent user experience of a mobile telephone programmed for “active listening.”   In this case, the mobile telephone maintains simultaneous connections across all available network connections, using these as probes into both proximal and virtual environments.   GSM/GPRS provides connectivity to the global network, while Bluetooth, and, to a lesser extent, IrDA, provides a probe into the proximal environment.

One of us (Pesce) has already constructed a Java J2ME application which turns the mobile telephone into an active listening device. The application constantly scans the “bluesphere” (the 10 meter radius around the mobile telephone), keeping a record of all the devices it sees, and reports these results, via GPRS, to a server on the Internet. The server then puts these results into a database, so they can be retrieved and analyzed as needed.

Our supposition was that it should be possible to construct a heuristic model of the user’s task modality – home, work, traveling, shopping, etc. – based on the information gathered through active listening to the bluesphere. Figure 2 shows a basic plot of the data gathered by the application, over a period of seventy-five minutes, covering a trip from Surry Hills (a central Sydney district) to the AFTRS campus in North Ryde (an outer suburb of Sydney), via public transport:   

As can be seen in the diagram, the first moments are stable, as there are, in general, a fixed number of Bluetooth devices within range of the mobile telephone when it is in the user’s home. The number drops during the walk to the train station, and rises dramatically while waiting on a crowded platform. The number of bluesphere devices drops again while the train is in transit to its destination station, rising slightly at the arrival platform, finally settling at a consistent number once the user arrives in the office. This pattern is regular and repeatable, day after day; this means that minimal machine intelligence is required to translate a scan of the bluesphere into an assessment of task modality.

It needs to be noted that active listening to the bluesphere returns more than just the raw number of Bluetooth devices; the mobile telephone also learns the unique addresses (the Bluetooth equivalent of Ethernet MAC addresses) and the “friendly names” of those devices. This means that it is possible to pinpoint the location of the mobile to within 10 meters when it is within range of some known, fixed-point Bluetooth device, such as the five Bluetooth devices which crowd the offices of the Emerging Media and Interactive Design Program at AFTRS, or the one which is always visible within Pesce's home. Again, it is a trivial operation for a server to translate a given Bluetooth address to a task modality; the mobile simply sends this data to the server, and the server tells the mobile that it is at home, in the office, and so forth.

Once the server is able to generate information about task modality, it can treat the mobile telephone as a network terminal, and reconfigure its display to present information which is relevant to the task.  

For example, in the office my mobile telephone could inform me of upcoming meetings; in transit it could warn me of rail delays and changing weather conditions; at home it could prompt me to turn on the television and watch a program recommended to me by a friend. None of this is difficult, but it is all quite useful, and this utility can be delivered to the user with a minimum of user interaction.

The more interesting phenomenon comes from a detailed analysis of a long-term recording of the bluesphere – over a period days to weeks. That analysis will show that certain devices come into the user’s bluesphere regularly. These occasions of proximity are the foundation for a model of an emergent social network. Rather than laboriously building the elements of the social network by hand, via a web site, the mobile telephone can simply listen to the bluesphere, SMS traffic and voice traffic, learn who the user is communicating with, when, and for how long. While this model will not necessarily be complete, it will be substantial, and will build itself without user intervention. That, in and of itself, is a powerful capability.

Active listening must extend beyond the network interfaces available to the mobile telephone into the real-world interfaces offered by the device. Most modern mobile telephones can make audio recordings. If this recording capability were kept on all the time, when combined with the analysis of the user’s emergent social network, it should be possible – and easy – to offer the user the opportunity to keep a full audio record of their day-to-day life. The mobile telephone would simply record audio constantly, storing it locally until it came in contact with a device it could use to synchronize this data with the user’s server. (The amount of data is small; just about 80MB covers an entire day’s recording.) The server could then present this data to the user, visualized across the time domain, with annotations showing who was around, participating in conversations, throughout the day. The user would simply click at the appropriate place on a web page, and hear their own conversation repeated back to them.  

Such features are as invaluable as they are dangerous. There are a host of privacy issues strewn throughout these examples, and these must be regarded as elements to be incorporated into the design of the system. Our basic belief is that users control the rights to their own data shadow, and that they share this data with others at their peril.

Example Two: Feeding the Hungry Social Network with Active Interventions

Once the system has grown a model of the user’s social network, it can then begin to feed that model as much information as can be gathered through active listening, augmented with “active interventions”. In an active intervention, the mobile telephone will interrupt the user in meaningful ways, designed to improve the quality of the user’s social network.

In this use case we’ll consider the Thanksgiving dinner I’m planning with friends; one of those friends, James, pops into a pub down the street; James and I have agreed to share data about our relative proximity, so I’m informed, via my mobile telephone, that James is in my neighborhood. I give him a call (which is noted by the active listener on both our mobile telephones), and drop down to the pub (also noted by the active listeners).  When our mobile telephones come into proximity, I get an alert on my mobile – the first instance of an active intervention. I’ve made a note to remind myself about Thanksgiving dinner, which is being held at James’ home. The note comes up, and I ask James about some details. No need to write any of this down, it’s all being recorded by my mobile telephone, and I’ll be able to play it back later.

After a few beers, James and I decide to go to see a film; but what film should we see? Since my mobile is in a task modality which indicates a social situation, it is already presenting information about social diversions. I see that “Wolf Creek” has gotten a rave review from my friend Nicola, so James and I decide to go to the film. The Dendy theatre uses a few Macs, with Bluetooth installed, so my mobile telephone notes this, tells the server, and my server guesses that I’m going to see a film.  

A few hours later, I receive a request – an active intervention – to rate the film I’ve seen. When I do so, that information is then shared within my social network, and contributes to the growing list of shared ratings[31]. I can also add my own review of the film, just by speaking into my phone; the server will later share that voice recording with anyone in my social network who wants to hear it.

Many of my social interactions are observed by my mobile telephone; all of these are recorded, mapped and analyzed. That said, not all of my social interactions can be observed by the mobile telephone; many of them take place through my laptop computer. My mobile telephone may be with me everywhere I go, but it can’t actively listen to my computer. My computer therefore needs to shoulder some of the burden. I will need to have an email client which notes who I receive email from, and who I send email to, adding that information to my data shadow. My web browser’s history also needs to be fed into that data shadow.  

There is a pattern here: we generate enormous data shadows – not just the ones related to our financial progress through the world, but others which relate to our social and informational presence. This information may be stored locally, but it is not collected, collated, or analyzed. In other words, we are depriving our data shadow of the constant stream of information we generate as we communicate. For this reason, it is our recommendation that software designers implement “audit trails” of user activity which can then be fed into dynamic digital social networks, enhancing their capability to model the user’s social network. It is relatively easy to do this within a system such as Gmail, which never forgets any transaction through it: you can simply scrape the data off Gmail. An extension for Firefox would do the same for web browsing. These are simple changes, which require no user intervention beyond setup, but they would provide the data shadow with a more complete recording of the user’s activities. This information should never be discarded; it is far too valuable.

Example Three: Sharing the Shadow

The value of the individual’s data shadow has not been overlooked by commercial interests. Wal-Mart, the largest retailer in the world, builds extensive data shadows on each of its customers, studying their buying patterns, constantly adjusting their store inventories to meet the needs of their customers. This is the basic premise that has driven the adoption of customer “loyalty” programs, such as the ColesMyer “FlyBuys” card.  (ColesMeyer is the largest retailer in Australia, with two main arms: Coles, a supermarket chain, and Myer, a mid-range department store chain.) ColesMyer offers incentives to regular customers; in return they build a data shadow of that customer’s purchase habits. This information is essential for ColeMyer’s purchasing plans; it also allows ColesMyer to target individual consumers with offerings that they are very likely to accept. In short, ColesMyer has a database, drawn from user interactions, which is of great benefit to them. They guard this data tightly; they don’t share it, even with the customers who created it. But there are good reasons to share that data with customers.

In our final use case, I am popping into the Coles at Surry Hills Marketplace, shopping for my Thanksgiving dinner. My mobile telephone knows I’ve entered Coles because they’ve setup a small Bluetooth transmitter which identifies the store.   Immediately the task modality of my mobile telephone changes, and it displays my shopping list. Behind the scenes, the server managing my own social network is having a detailed discussion with ColesMyer’s own substantial computing facility. My server knows what I’m shopping for, and negotiates with Coles to get the lowest prices for each of the items on my shopping list. All of these sale prices will be tied to my FlyBuys card, so when I pass through checkout, the sale prices are applied. These prices are for my eyes only, and really, I don’t even see them. What I do see is a offer for $5 off on a fine Thanksgiving turkey; that’s important enough to be bounced up to my attention. It’s the only thing that I’m aware of, even though a lot of communication has taken place, out of view, between my own server and ColesMyer.

Furthermore, when I pass through the check-out, ColesMyer will do me the favor of informing my server of what I’ve bought, what it all cost, and so forth. This means some items will be removed from my shopping list, without my intervention, while others will remain. It also means that I can track my purchases and my expenses without having to laboriously enter any data.

All of this is easy enough to implement: if there’s one thing we understand, post dot-com collapse, it’s how to make databases talk to each other. We have XML and other standards which provide roll-your-own protocols. There is some programming involved here, but nothing extraordinary. User setup and user intervention are both minimal. As long as all parties can agree on how to communicate, such a system just works.

Why would Coles offer this service to its customers? The answer is obvious: loyalty. Any store which could make my shopping experience as personalized and seamless as this use case is more likely to hold their customers. They get a better sense of their clientele, as well, because ColesMyer gets a peek at what their customers are looking for, not just what they’re buying today. ColesMyer can build models of my user behavior as a consumer, which will help them to offer me just what they want, just when I want it. That will help them tailor their stores’ inventories. A commercial enterprise is just as much a node within a social network as any individual; strong and constantly reinforced informational relationships between these nodes will tend to improve and strengthen the real-world relationship.

Conclusion: Playing the Future

It is no longer sufficient to consider electronic communication as a two-way affair; electronic communication in the twenty-first century involves swarms of individuals, engaged in common or closely-aligned or loosely-coupled tasks. The translation of human social networks into dynamic digital entities, fed continuously by devices which actively listen and actively intervene, creates the necessary precondition for the fifth stage in the evolution of Internet use, an era where our data shadows stand alongside our physical selves, working to maintain and improve our effectiveness across the breadth of our social networks.  

Many of the ideas explored in this paper are not ours, but have been drawn from longstanding research in ubiquitous computing[32], and recent work in the visualization and management of social networks[33]. The present work contributes to this discourse an awareness that the infrastructure for this transformation is already in place. This is a software problem, which means that in all likelihood it will be solved quickly.   We invite you to do your own research, play with these devices, learn from the users, and invent the platforms for our human future.


30. A. N. Whitehead, Process and Reality, Macmillan, New York, 1927, p. 309
31. Mark Pesce, “The You Portal”, Mobile Journeys, February 2005, Sydney, p. 5
32. Mark Weiser, “Hot Topic: Ubiquitous Computing”, IEEE Computer, October 1993, pp 71-72
34. Steven Blyth, “My Social Fabric”, http://www.stevenblyth.com/mysocialfabric/index.html

Mark Pesce is the co-creator of the Virtual Reality Modeling Language (VRML) - the first 3D interface to the internet - and the founder of the Interactive Media Program at USC's School of Cinema-Television. In 2000, Ballantine Books published Pesce's The Playful World: How Technology is Transforming our Imagination, which explored the world of interactivity through a detailed examination of the Furby, LEGO’s Mindstorms and the Playstation 2. In late 2003, Pesce was invited to the Australian Film Television and Radio School, with a mandate to redesign the curriculum to incorporate the new opportunities offered by interactive media.

Read Mark's blog: hyperpeople.

This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/2.0/ or send a letter to Creative Commons, 559 Nathan Abbott Way, Stanford, California 94305, USA.

Advertise in Mindjack
Email for Information


home | about us | feedback