.webp)
The head of big data at Coles is an Irishman named Niall Keating. He's worked in the area of big data for four years, having previously been at Telstra.
If you aren't an Australian, Coles is a dominant supermarket chain in Australia and Telstra the largest telco in the land. Clearly both large enterprises work with a lot of data.
He's also a major advocate of the open source Hadoop eco-system for handling big data. This eco-system is developed by the open source focused Apache Foundation. He was also the first person in Australia to reach out to HortonWorks, a key tool in this open source big data world.
His job is to modernise and simplify the retailer's information systems, in particular, it's big data technologies.
"Coles has spent a lot of time with proof of concepts and trying out different technologies. And now in the last couple of years, we've really seen what we can do with that [big data] eco-system. It's now being deployed in earnest to solve business problems," Keating says.
Scott Gnau, Hortonworks Chief Technology Officer at HortonWorks, believes that big data technologies have become a more established part of the IT landscape. No longer are they just pipe dreams.
"Coles' experience really echoes or mirrors what we've seen in other regions. Starting with tests and pilots but now delivering more business value and it being worked into production," Gnau says.
Big data adoption in Australia
Keating also sees that Hadoop and big data technology adoption has been a little slow in Australia.
The challenges that he sees are culture and skills. Culturally, that IT departments in Australia are risk averse. He believes that they need to get that balance right of adopting new technologies versus not making mistakes and managing costs. "Every enterprise should have an innovation agenda," says Keating.
While he sees universities producing data scientists, there is a skills shortage in software engineers holding back big data adoption in Australia, the kind of people that can build scalable and distributed back end systems.
The future
Keating sees a number of start-ups being created around big data, either as technology providers or just at their core using big data to develop new products and services. Already corporates like Telstra and Westpac have invested in these type of ventures.
Read about an innovative example; Data Republic here.
"Data will be at the core of products going forward and drive customer experience. You can see this with Uber and AirBnb already," Keating says.
He also sees that technologies like IOT and Cloud are linked to the useful rise of big data in the enterprise.
"The next challenge is real time. Real time is quite easy, but real time at scale can be quite hard," he continues.
Gnau agrees that real-time is going to be the way of the future.
"It's important for the business to adjust and be able to react very quickly. Customers and even businesses will expect this real-time reaction in the future," Gnau says.
On Hadoop and Hortonworks​
Keating likes the Hadoop eco-system because it's open source foundation creates competition between engineers globally. Which in turn pushes the technology ecosystem forward.
"This creates opportunities but also that flexibility, innovation and constant change creates challenges for enterprises," he continues.
"That's where we need organisations like HortonWorks to put that wrapper around the open source innovation, to give us a solid platform to build our enterprise applications on top of," he explains.
HortonWorks sees their role as adding dependability, scalability, reliability to the Hadoop open source eco-system. Which in turn enables ease and acceleration of adoption of the tools.