Many organisations that have big data programs don’t use the cloud at all because it poses such a challenge. An often cited survey by Gartner carried out in 2014 showed that less than half of the enterprises with these programs were deploying cloud in any form. Most companies where the public cloud is utilised in some form use it largely for SaaS applications.
The pills are bitter so we are not going near them
It is interesting to note that despite the wide range of opportunities presented by using the cloud, many organisations don’t seem to be in a hurry to move big data processing to the cloud. When you think about benefits such as elasticity, scalability, and cost, you might imagine that there must be a valid reason why these companies haven’t all done the needful. At a certain point in time, the argument might have been issues surrounding security, but those are easing and now more than ever, there are a myriad of services available. That certainly sounds like more than enough incentive, but it’s not that easy.
The matter lies more in the infrastructure of an enterprise and the foundation it was built on. Maybe pills are not the best analogy, but the point is using the cloud is not the easiest thing to adopt even with the immense benefits. Think of an established organisation that has invested massively in their own data centres. Such an organisation would need to go through a painstaking process of learning, maintaining, and integrating two separate environments in various aspects including operational, cultural, and technical. So if a company does not start off with cloud as part of its normal processes, it would become a headache to deal with eventually. On the other hand, enterprises that kicked off with their business completely online had cloud-based systems right from the start.
There are a number of issues that may crop up when an enterprise tries to use big data in the cloud, however, there is always a way to deal with them, particularly if the enterprise selects the appropriate services for their requirements and plans ahead for all possible problems.
Clouds and data centres couldn’t be any more different
On-premises data centres and cloud services are entirely different and while many organisations know this, they don’t quite understand the implications. When it comes to price, configuration and installation, traditional servers in a data centre will require a different set of activities from cloud services. This means a company would have to acquire new skills or train staff if they decide to use and optimise cloud infrastructure.
“Even with setting up the most basic of services, it is a daunting challenge to integrate cloud services with a company’s current system, not to talk of managing the cloud and other aspects involved, such as integration, deployment and architecture maintenance,” says Brendan Wilde, cloud migration expert at Umbrellar Cloud Backup.
Finding the right skills is a challenge
One of the most serious issues with operating in the cloud is the big skill gap that must be filled. There are not enough skilled experts in cloud DevOps, development, and cloud architecture. There is a similar situation with big data as there is a shortage of skills as well. Even more difficult and expensive to find is expertise in both cloud and big data. Experts believe that gaining the required expertise will continue to be a problem unless there is greater collaboration across enterprises. There is also the problem of properly managing the talent an organisation does have. For example, many organisations are sceptical about disseminating new knowledge through the enterprise using certain third-party tools, making it a longer process to migrate to the cloud.
Security issues with the cloud
Gartner once predicted that 95% of cloud security failures through 2020 would be the fault of the customer. Human mistakes become prevalent when cloud policies are poorly defined or the cloud is used in an inconsistent, ad hoc manner. Security was once the primary concern, but organisations can choose vendors that have no problem working with their existing policies and systems. They can also learn to properly use the security features provided by cloud providers. Once enough attention is given to security, with loop holes covered and cloud security formally addressed, any enterprise can scale through.
Experimentation is one thing and production is quite another
An enterprise might deploy cloud for big data, but integrating the cloud with the data process production won’t be quite as easy. With a new cloud environment comes challenges such as change management, operational difficulties, complex data, and system integration. Determining and upholding service level agreements or SLA’s for cloud services is a struggle for enterprises. Also, it is very difficult to support the cloud as you might other services. Where you find enterprises that have successfully built their own capabilities for cloud analytics, they must have invested loads of money and many years of effort into the effort.
And data movement is the most challenging of all
The concern that is mostly brought up when data movement is discussed is how large datasets of up to petabyte size move into the cloud from data centres. However, this isn’t necessarily the best or most appropriate way to look at it. It is true that extremely large volumes of data are often moved by ground transportation, but these are not usually ongoing updates. Moving data to and from the cloud is the real problem, meaning data movement cannot easily be made a seamless aspect of the enterprise data flow. Instead, the focus should be on data pipelines, micro-batching updates, and streaming.
Standardisation of cloud services
It is not easy for a cloud provider to fit adequately with an enterprise’s current processes because vendors typically have different levels of operational support, SLAs, management, and monitoring. Their pricing models, capabilities, and labels are also very different. For example, while some services operate on a flat rate model, others adopt a pay-as-you-go model.