What is Big Data?

We used to keep data in nice little structured loyalty-card piles, then a little thing called the internet came along and started spewing out huge unstructured chunks of ones and zeros. Now nearly everything we do leaves some kind of digital trace, and the Internet of Things is creating data at such velocity and in such large volumes that, in fact, ‘big’ doesn’t even come close to describing its size. It’s like calling the HSC a pop quiz.

What are we doing with it?

So we’ve collected huge amounts of data – go us! But what are we doing with it all? The short answer is: not as much as we could. While some companies and universities are starting to do some cool things (see below), the industry is crying out for qualified people who have the skills to collect, process and analyse big data so we can actually put it to good use.

How is it being used in everyday life?

Writing for The Nation, Dr Rohan Wickramasuriya, a Research Group Leader at UOW’s SMART Infrastructure Facility, points out that “recent technological advancements have made it possible to sift through mountains of data and uncover hidden intelligence.” We took a look at three seemingly normal everyday activities to see what treasures have been found.

Watching TV

With over 55 million subscribers worldwide, Netflix has access to a HUGE amount of data. Every time you sit down to chill, they get a little better at understanding what you like and even how long it will take until you’re hooked on a series. This data not only lets them suggest new content from their 75,000 micro-genres, it gives them the insights they need to make their own content. Based on big data, Netflix knew House of Cards had the recipe for success. They were so confident they commissioned two complete seasons without testing a pilot. Big data even gave them insights into what colours to include on its cover art.


A photo posted by Netflix US (@netflix) on

Using Social Media

As you can probably guess, social media clocks up the gigabytes pretty quickly. Just take a look at this infographic from Domo. While it may seem like a lot of cats, the data output is also being used for something a little more serious. Researchers at UOW have developed a system that turns geo-tagged tweets about flooding in Indonesia’s capital city, Jakarta, into valuable data in real-time. Citizens share flood information with social media peers while simultaneously providing Emergency responders with the data they need to support decision making in a disaster.

Big Data - Jakarta
The main street of Jakarta flooded on 18 February, 2013. AFP Photo / Adek Berry

Listening to music

Your taste in music says a lot you, and streaming services like Spotify are listening.
They use big data to suggest new music you might be interested in and – for those of us not willing to pay the premium – serve up ads you might want to hear. Spotify Running uses data from your phone's sensors to detect your running tempo and select music with the same beat to keep you motivated while your run. Record labels and artists can use Spotify’s big data to see what their fans are liking and even where they should tour. And don’t be surprised if you see Spotify create its own record label in the future that will sign artists based on, you guessed it, big data.


The future is data-full

The amount of data we’re producing isn’t going to slow down anytime soon and, while privacy issues around all this data and who actually owns it is a legitimate concern, the possibilities for big data are endless. You just have to know what questions to ask.

If you’re interested in big data, check out the Computer Science degrees at UOW, which includes the option to major in big data.