Ideas Menu Search

The Quantified Us: Where Small Data Meets Big Data

Have you ever wondered how many steps you took today, or exactly what percentage of calories you burned walking versus running? Curious to know how stressed you are at work versus at home, or exactly how much REM sleep you got last night? Concerned about how pollution in the local environment might be affecting your allergies, or whether you’re consuming enough Vitamin A in your diet?

In many ways, there’s never been a better time to ask these questions – because there’s never been an easier way to answer them. As technology and sensors have become increasingly miniaturized and are integrated into everything from our homes to our phones and even our clothing, we’re able to capture more data than ever before about our health, our environments, and our lifestyles.

The evolution

For a dedicated minority, the ‘Quantified Self’ era has been met with enthusiasm. Not only can massive datasets about oneself be a route to enhanced self-discovery, they can also be a tool for running self-experiments and figuring out how to optimize health. But for the majority of us, the idea of continuous self-tracking is a novelty that results in little more than shallow insights about ourselves, which are no longer interesting after a few weeks. Just ask anyone who has bought a Fitbit which now rests untouched for months at the bottom of a junk drawer.

The true value of the Quantified Self movement lies far beyond the novelty of gratuitous data about ourselves and our past behaviors, which we might call a ‘first degree of meaning.’ The real promise of the Quantified Self is in a ‘second degree of meaning,’ where self-tracking helps motivate people toward self-improvement, and a ‘third degree of meaning,’ where people can use data to make better, more informed health decisions in the moments where they matter most. While we’re beginning to see the Quantified Self move toward these second and third degrees of meaning, many of the tools and services available today that emphasize self-tracking are still largely rooted in the novelty of recording behavior and serving up daily data. (Read more about the degrees of meaning of the Quantified Self).

As the Quantified Self inches its way toward more meaning and impact, another data-driven movement has been steadily brewing, one that is in many ways driven by the same breadth of data that the Quantified Self contributes to Big Data. Big Data offers businesses and organizations insights into the behaviors and trends of large populations. Where the Quantified Self has remained very much about individual enlightenment, retaining the grassroots flavor of its local meet-up origins, Big Data has become a buzzword in the commercial world and a rapidly adopted tool of big business.

For most of us, we’ve been exposed to Big Data’s impact primarily through targeted marketing campaigns and recommendation engines, pushing consumption and purchasing decisions (which music to listen to, which products to buy). Netflix, for example, can give you better recommendations about which movies you’ll enjoy because they have data on your previous viewing history and ratings from other people with whom you have things in common. The company’s continued investment in this type of technology (as covered on its tech blog) indicates the longevity of this trend.

The spaces between

But what lies in between the Quantified Self, with its emphasis on individual data, and Big Data, with its emphasis on aggregating data from larger populations? Is there a way we might we bridge these two movements with an eye toward doing more than promoting consumption, to help improve people’s lives in the long term?

Imagine a future where self-tracking harnesses the power of a whole population’s data to identify patterns and make meaningful recommendations about what we should do next. Imagine a future where we can fluidly move between our own data and the data of the collective to gain insights on how best to live the life we desire, and where we decide what privacy we give up because we control the benefit it brings us?

The Quantified Us

We are excited about the idea of the ‘Quantified Us,’ a term we use at Artefact which describes the important space between Big and Small data, and between the Quantified Self and crowd. The Quantified Us is based on a select group of people who share relevant characteristics and behaviors. This group may be centered on similar goals, health conditions, biometrics, personal qualities, environmental factors, or even similarity of emerging data patterns. When the members of a group with such similarities decide to participate in collective pooling their self-tracked data, the data-driven insights, and recommendations that result take on an added meaning, impact, and personal relevance.

We are already starting to see the beginnings of a Quantified Us movement starting to emerge though we feel its full potential is untapped:

  • PatientsLikeMe allows people to share personal health records so they can compare ‘treatments, symptoms, and experiences.’ The site also supports personal connections with the community, as well as the ability to track your own health data and to make your records available to medical researchers. These data, however, are positioned as a tool for the medical community to review and gain clinical insights.
  • Crohnology is a social network centered on people who suffer from Crohn’s disease and colitis. The website is focused on building collective knowledge about overall treatment approaches and offers ratings on treatment effectiveness. The community revolves around the sharing and aggregation of information, but only at an overall therapy approach level. The scope and depth of data and the relationships of variables that the patient can access, as well as the insights which might be gleaned, is limited.
  • StockTwits is another example of a social network which is starting to present some aspects of the Quantified Us. By using a followers model, this service connects investors who are interested in the same financial opportunities so they can learn and share insights before making trades. Though the insights can be very timely and represent the sentiment of an informed group, the ‘group’ is just defined by who decides to follow who and there is no collaboration or cooperation between individuals because no one is sharing their personal data.

The application

While these early, partial examples of the Quantified Us are headed in the right direction, they’re largely fragmented and still rely heavily on users to manually share their data and look for insights. A true implementation of the Quantified Us will likely tackle the challenge of helping these groups form, facilitating data collection, extracting insights, and turning insights into actionable recommendations for specific individuals. In the future world of a fully realized Quantified Us, we can work frictionlessly as a collective to positively affect the human as an individual.

To get to this future, we must be clear about what the Quantified Us is and how it achieves success. A successful Quantified Us strategy is:

  • Based on Quantitative Data – To be able to easily and accurately see trends and patterns from different groups and in different time frames, quantified core data is key. Until technology becomes much better at processing semantic information, the qualitative data will serve to augment patterns to add narrative and to build empathy. Such qualitative data is very important to the human experience of gaining insights, but it is secondary in Quantified Us strategies.
  • Selective, But Configurable – To make insights as relevant as possible, the collective of people sharing data needs to be carefully selected, but also configurable. This does not mean that some people will be kept from participating, rather, as an individual is looking for insights about themselves, they must be able to control the boundaries of the data set they are analyzing. Most simply, this sorting will be by who is included or excluded from the sample. For example, if a person who experiences migraines knows that caffeine primarily affects her migraine threshold, she should be able to exclude people from her sample collective who have other unrelated triggers. Defining a user experience that carefully balances the breadth of the community with the interests and needs of the individual user will be key to its success.
  • Driven by Democracy – The data set must always be defined and populated by those who will use it. The Quantified Us will have value because the people who have chosen to see the benefit in trading off their personal, private data for access to a larger set of data from which to gain understanding about themselves. Staying true to these grassroots origins and the interests of the users is key for the offering to stay fresh and meaningful. The role of design would be to support the experience the community creates for itself and to amplify and augment the directions the community pursues.
  • Focused on Individual Understanding and Decisions – The ultimate value of the Quantified Us is the ability to help individuals extract insights, change their behaviors, and make better decisions. If a Quantified Us platform just becomes a place where many users store their data, but the data are isolated and do not reveal relationships across participants, then the platform has failed. Similarly, if the platform does provide insights, but those findings are for anyone besides the individuals who contributed data, then the platform loses much of its personal relevance. These core values of understanding and decision-making targeted for individuals should motivate the entire experience with the platform.
  • Inclusive of the Full Data Journey – For Quantified Us strategies to take off and have meaning, they must successfully address all stages of data life. It is easy to be overly focused on the data analysis stage because that is where the obvious value is. But for that value to be possible, we need to get large amounts of data of a wide variety of types from the users into the system. And once the data is collected, how patterns are represented and how data in the Quantified Us interacts with data elsewhere must also be addressed. After the data is in use, there also needs to be a clear end of life for the data if that is what the person who contributed the data wants. Addressing this full data journey will be tantamount to the success of the Quantified Us.

The promise

Imagine a person with epilepsy who is trying to understand what has caused an increase in the frequency of his absence seizures. What if he could fluidly record biometric, environmental, and self-reported data, and compare them to the trigger patterns of others just like him?

This is how the Quantified Us can help people with complex, unpredictable conditions get a better understanding of what causes emergent events. This example focuses on epilepsy, but other similar conditions where triggers are unknown that could benefit from Quantified Us strategies include Crohn’s disease, migraines, arthritis, rheumatoid arthritis, asthma, and chronic fibromyalgia.

Now imagine a person with insulin-dependent diabetes whose blood sugars are running high at night, but who isn’t able or doesn’t feel motivated to understand why. What if she could see the profiles and data of others who share the same biological, behavioral, and therapeutic attributes as she does, learn about their success with controlling their blood sugar levels, and see where she falls relative to the “norm”? What if she was able to start a dialog with other people like her to ask questions about specific past cases, or to get emotional support when she needs it?

This is one example of how the Quantified Us can help prompt and motivate people with chronic conditions to manage their conditions more effectively, but you can see how this example might extend to other conditions where motivation, measurement, and management matter, like hypertension, COPD, and congestive heart failure.

It’s easy to imagine a variety of scenarios in which self-tracking combined with collective data sharing, can result in more informed management, deeper understanding, and heightened motivation. Ultimately the Quantified Us can help people take better care of themselves, more often.

The unknowns

The most obvious concerns around Quantified Us are lingering worries about personal privacy, liability issues for those who contribute and host data, and simply that the data themselves will be low quality and not able to inform meaningful insights. Other concerns are less obvious but equally real and will not be fully understood until we see how people create, use, and then identify with a Quantified Us approach:

  • Too Much Faith in the System – One fear is that people rely too much on the platform and blindly act on the most obvious patterns without fully processing all the information available to make a decision. Insulin pumps, for example, offer a wizard that makes recommendations for how much insulin the patient should administer based on current glucose levels and carbohydrate intake. Some doctors look for the wizard to be overridden at least 30% of the time because they see an advantage to having the patient make the effort to cognitively process how much insulin they take. Balancing system intelligence with user involvement and autonomy will be a design challenge in developing Quantified Us.
  • Too Many Data – When data interpretation is left up to end users, it may feel empowering at first, but it can quickly become overwhelming (if not impossible to keep up with). And when we’re faced with overwhelming amounts of information, particularly when that information might raise a red flag about our health and lifestyle choices, we may decide to tune it out entirely, or bury our heads in the sand – a tendency known as information avoidance, or the “ostrich effect” (as it is commonly known in financial situations). Presenting the data in a clear, actionable way will be a key priority for the Quantified Us.
  • Too Few Data – In contrast, what if, especially when the self-selected community is new, there is a very small dataset that leads to skewed patterns and insights? The fewer data present to review, the less confidence the user should have in conclusions drawn from the insights. On the other hand, even when many data are available, we may fall victim to a “confirmation bias” – where we filter out or ignore data that doesn’t confirm a personal hypothesis we hold about what’s happening and why.  So even in the face of a large number of data, individuals may still narrow in on a small subset of examples that aren’t large enough to confidently draw conclusions based upon. The shortcomings of specific data sets should always be reiterated so that the user can adjust confidence accordingly.
  • Too Quick to Conclude – A greater risk, perhaps, comes in misinterpreting our own data, and jumping to conclusions that may not be correct. When we encounter data, we tend to look for patterns in the data that will help us make sense of it – but the problem arises when we start to see patterns that don’t actually exist. Our tendency to see small patterns or streaks in what is really random data is known as the clustering illusion. Applying and reinforcing visual and data literacy principles will be an important part of the Quantified Us experience.
  • Too Much Reliance on Data – There is also a risk that we will rely too much on data to inspire people to change their behavior, but in reality the data by itself may not be a motivator. Many other factors drive successful behavior change, and successful Quantified Us approaches should look at what we know about how and why behavior change occurs and use that knowledge to better craft the way we present data and help people act on it.

A Better Me

From its origin, self-improvement through self-learning has been the goal and expectation of the Quantified Self movement, but it has yet to be fully realized. Unless we find ways of making the Quantified Self movement relevant beyond the first degree of meaning, it will always remain a novelty. One of the ways we can transition the Quantified Self movement to have more impact is to bridge the gap between Big and small data and to heighten the collective relevance of the data we track about ourselves. By uncovering insights about ourselves through looking closely at others who are like us in the most meaningful ways, we can chart new paths toward becoming the people we want to be. As the Quantified Self movement matures into one of the Quantified Us, our desire is to help it stay attuned to its ultimate promise: a better me.