January
26 , 2005
| Part Three: Me and My Data Shadow
Theory is good; observation is better. One
of us (Pesce)
has been observing the growth of the Internet, user communities,
and swarms for nearly 20 years, while the other (Fraser)
has deep experience concerning the issues of mobility and usability.
The concrescence[30]
of observation and experience has lead us from theory into practice.
We are presently designing technological probes, testing the
theses we have laid out in this paper, in order to concretize
our understanding.
One of us (Fraser), as the former Director
of User Experience for Hutchinson
Telecoms Australia, has a strong understanding both of the
desires and the frustrations of mobile telephone users. These
real-world insights have guided us into designs that require
a minimum of user intervention. In other words, we have done
our best to design things that “just work,” without
a lot of care and feeding. An example of this philosophy, created
by one of us (Fraser), is ImageShow,
a Java J2ME application which fetches images from the photo
sharing site Flickr,
allowing the mobile to dip into “Flickrstream” for
an endless supply of imagery, filtered by user name and meta
tags:
What follows are three speculative use cases,
shown principally from the user’s point of view, and drawn
from the themes outlined in this paper.
Example
One: Active Listening
Starting from the basic proposition that the
mobile telephone ought to do everything within its power to
be intensely aware of its environment, the first use case involves
the emergent user experience of a mobile telephone programmed
for “active listening.” In this case, the
mobile telephone maintains simultaneous connections across all
available network connections, using these as probes into both
proximal and virtual environments. GSM/GPRS provides
connectivity to the global network, while Bluetooth, and, to
a lesser extent, IrDA, provides a probe into the proximal environment.
One of us (Pesce) has
already constructed a Java J2ME application which turns
the mobile telephone into an active listening device. The application
constantly scans the “bluesphere” (the 10 meter
radius around the mobile telephone), keeping a record of all
the devices it sees, and reports these results, via GPRS, to
a server on the Internet. The server then puts these results
into a database, so they can be retrieved and analyzed as needed.
Our supposition was that it should be possible
to construct a heuristic model of the user’s task modality
– home, work, traveling, shopping, etc. – based
on the information gathered through active listening to the
bluesphere. Figure 2 shows a basic plot of the data gathered
by the application, over a period of seventy-five minutes, covering
a trip from Surry Hills (a central Sydney district) to the AFTRS
campus in North Ryde (an outer suburb of Sydney), via public
transport:
As can be seen in the diagram, the first moments
are stable, as there are, in general, a fixed number of Bluetooth
devices within range of the mobile telephone when it is in the
user’s home. The number drops during the walk to the train
station, and rises dramatically while waiting on a crowded platform.
The number of bluesphere devices drops again while the train
is in transit to its destination station, rising slightly at
the arrival platform, finally settling at a consistent number
once the user arrives in the office. This pattern is regular
and repeatable, day after day; this means that minimal machine
intelligence is required to translate a scan of the bluesphere
into an assessment of task modality.
It needs to be noted that active listening
to the bluesphere returns more than just the raw number of Bluetooth
devices; the mobile telephone also learns the unique addresses
(the Bluetooth equivalent of Ethernet MAC
addresses) and the “friendly names” of those
devices. This means that it is possible to pinpoint the location
of the mobile to within 10 meters when it is within range of
some known, fixed-point Bluetooth device, such as the five Bluetooth
devices which crowd the offices of the Emerging
Media and Interactive Design Program at AFTRS, or the one
which is always visible within Pesce's home.
Again, it is a trivial operation for a server to translate a
given Bluetooth address to a task modality; the mobile simply
sends this data to the server, and the server tells the mobile
that it is at home, in the office, and so forth.
Once the server is able to generate information
about task modality, it can treat the mobile telephone as a
network terminal, and reconfigure its display to present information
which is relevant to the task.
For example, in the office my mobile telephone
could inform me of upcoming meetings; in transit it could warn
me of rail delays and changing weather conditions; at home it
could prompt me to turn on the television and watch a program
recommended to me by a friend. None of this is difficult, but
it is all quite useful, and this utility can be delivered to
the user with a minimum of user interaction.
The more interesting phenomenon comes from
a detailed analysis of a long-term recording of the bluesphere
– over a period days to weeks. That analysis will show
that certain devices come into the user’s bluesphere regularly.
These occasions of proximity are the foundation for a model
of an emergent social network. Rather than laboriously building
the elements of the social network by hand, via a web site,
the mobile telephone can simply listen to the bluesphere, SMS
traffic and voice traffic, learn who the user is communicating
with, when, and for how long. While this model will not necessarily
be complete, it will be substantial, and will build itself without
user intervention. That, in and of itself, is a powerful capability.
Active listening must extend beyond the network
interfaces available to the mobile telephone into the real-world
interfaces offered by the device. Most modern mobile telephones
can make audio recordings. If this recording capability were
kept on all the time, when combined with the analysis of the
user’s emergent social network, it should be possible
– and easy – to offer the user the opportunity to
keep a full audio record of their day-to-day life. The mobile
telephone would simply record audio constantly, storing it locally
until it came in contact with a device it could use to synchronize
this data with the user’s server. (The amount of data
is small; just about 80MB covers an entire day’s recording.)
The server could then present this data to the user, visualized
across the time domain, with annotations showing who was around,
participating in conversations, throughout the day. The user
would simply click at the appropriate place on a web page, and
hear their own conversation repeated back to them.
Such features are as invaluable as they are
dangerous. There are a host of privacy issues strewn throughout
these examples, and these must be regarded as elements to be
incorporated into the design of the system. Our basic belief
is that users control the rights to their own data shadow, and
that they share this data with others at their peril.
Example
Two: Feeding the Hungry Social Network with Active Interventions
Once the system has grown a model of the user’s
social network, it can then begin to feed that model as much
information as can be gathered through active listening, augmented
with “active interventions”. In an active intervention,
the mobile telephone will interrupt the user in meaningful ways,
designed to improve the quality of the user’s social network.
In this use case we’ll consider the Thanksgiving
dinner I’m planning with friends; one of those friends,
James, pops into a pub down the street; James and I have agreed
to share data about our relative proximity, so I’m informed,
via my mobile telephone, that James is in my neighborhood. I
give him a call (which is noted by the active listener on both
our mobile telephones), and drop down to the pub (also noted
by the active listeners). When our mobile telephones come
into proximity, I get an alert on my mobile – the first
instance of an active intervention. I’ve made a note to
remind myself about Thanksgiving dinner, which is being held
at James’ home. The note comes up, and I ask James about
some details. No need to write any of this down, it’s
all being recorded by my mobile telephone, and I’ll be
able to play it back later.
After a few beers, James and I decide to go
to see a film; but what film should we see? Since my mobile
is in a task modality which indicates a social situation, it
is already presenting information about social diversions. I
see that “Wolf Creek” has gotten a rave review from
my friend Nicola, so James and I decide to go to the film. The
Dendy theatre uses a few Macs, with Bluetooth installed, so
my mobile telephone notes this, tells the server, and my server
guesses that I’m going to see a film.
A few hours later, I receive a request –
an active intervention – to rate the film I’ve seen.
When I do so, that information is then shared within my social
network, and contributes
to the growing list of shared ratings[31]. I can also add
my own review of the film, just by speaking into my phone; the
server will later share that voice recording with anyone in
my social network who wants to hear it.
Many of my social interactions are observed
by my mobile telephone; all of these are recorded, mapped and
analyzed. That said, not all of my social interactions can be
observed by the mobile telephone; many of them take place through
my laptop computer. My mobile telephone may be with me everywhere
I go, but it can’t actively listen to my computer. My
computer therefore needs to shoulder some of the burden. I will
need to have an email client which notes who I receive email
from, and who I send email to, adding that information to my
data shadow. My web browser’s history also needs to be
fed into that data shadow.
There is a pattern here: we generate enormous
data shadows – not just the ones related to our financial
progress through the world, but others which relate to our social
and informational presence. This information may be stored locally,
but it is not collected, collated, or analyzed. In other words,
we are depriving our data shadow of the constant stream of information
we generate as we communicate. For this reason, it is our recommendation
that software designers implement “audit trails”
of user activity which can then be fed into dynamic digital
social networks, enhancing their capability to model the user’s
social network. It is relatively easy to do this within a system
such as Gmail,
which never forgets any transaction through it: you can simply
scrape the data off Gmail. An extension for Firefox
would do the same for web browsing. These are simple changes,
which require no user intervention beyond setup, but they would
provide the data shadow with a more complete recording of the
user’s activities. This information should never be discarded;
it is far too valuable.
Example
Three: Sharing the Shadow
The value of the individual’s data shadow
has not been overlooked by commercial interests. Wal-Mart, the
largest retailer in the world, builds extensive data shadows
on each of its customers, studying their buying patterns, constantly
adjusting their store inventories to meet the needs of their
customers. This is the basic premise that has driven the adoption
of customer “loyalty” programs, such as the ColesMyer
“FlyBuys” card. (ColesMeyer is the largest
retailer in Australia, with two main arms: Coles, a supermarket
chain, and Myer, a mid-range department store chain.) ColesMyer
offers incentives to regular customers; in return they build
a data shadow of that customer’s purchase habits. This
information is essential for ColeMyer’s purchasing plans;
it also allows ColesMyer to target individual consumers with
offerings that they are very likely to accept. In short, ColesMyer
has a database, drawn from user interactions, which is of great
benefit to them. They guard this data tightly; they don’t
share it, even with the customers who created it. But there
are good reasons to share that data with customers.
In our final use case, I am popping into the
Coles at Surry Hills Marketplace, shopping for my Thanksgiving
dinner. My mobile telephone knows I’ve entered Coles because
they’ve setup a small Bluetooth transmitter which identifies
the store. Immediately the task modality of my mobile
telephone changes, and it displays my shopping list. Behind
the scenes, the server managing my own social network is having
a detailed discussion with ColesMyer’s own substantial
computing facility. My server knows what I’m shopping
for, and negotiates with Coles to get the lowest prices for
each of the items on my shopping list. All of these sale prices
will be tied to my FlyBuys card, so when I pass through checkout,
the sale prices are applied. These prices are for my eyes only,
and really, I don’t even see them. What I do see is a
offer for $5 off on a fine Thanksgiving turkey; that’s
important enough to be bounced up to my attention. It’s
the only thing that I’m aware of, even though a lot of
communication has taken place, out of view, between my own server
and ColesMyer.
Furthermore, when I pass through the check-out,
ColesMyer will do me the favor of informing my server of what
I’ve bought, what it all cost, and so forth. This means
some items will be removed from my shopping list, without my
intervention, while others will remain. It also means that I
can track my purchases and my expenses without having to laboriously
enter any data.
All of this is easy enough to implement: if
there’s one thing we understand, post dot-com collapse,
it’s how to make databases talk to each other. We have
XML
and other standards which provide roll-your-own protocols. There
is some programming involved here, but nothing extraordinary.
User setup and user intervention are both minimal. As long as
all parties can agree on how to communicate, such a system just
works.
Why would Coles offer this service to its customers?
The answer is obvious: loyalty. Any store which could make my
shopping experience as personalized and seamless as this use
case is more likely to hold their customers. They get a better
sense of their clientele, as well, because ColesMyer gets a
peek at what their customers are looking for, not just what
they’re buying today. ColesMyer can build models of my
user behavior as a consumer, which will help them to offer me
just what they want, just when I want it. That will help them
tailor their stores’ inventories. A commercial enterprise
is just as much a node within a social network as any individual;
strong and constantly reinforced informational relationships
between these nodes will tend to improve and strengthen the
real-world relationship.
Conclusion:
Playing the Future
It is no longer sufficient to consider electronic
communication as a two-way affair; electronic communication
in the twenty-first century involves swarms of individuals,
engaged in common or closely-aligned or loosely-coupled tasks.
The translation of human social networks into dynamic digital
entities, fed continuously by devices which actively listen
and actively intervene, creates the necessary precondition for
the fifth stage in the evolution of Internet use, an era where
our data shadows stand alongside our physical selves, working
to maintain and improve our effectiveness across the breadth
of our social networks.
Many of the ideas explored in this paper are
not ours, but have been drawn from longstanding research in
ubiquitous
computing[32], and recent
work in the visualization and management of social networks[33].
The present work contributes to this discourse an awareness
that the infrastructure for this transformation is already
in place. This is a software problem, which means that
in all likelihood it will be solved quickly. We invite
you to do your own research, play with these devices, learn
from the users, and invent the platforms for our human future.