Building a modern data platform – what have we learned?

As I reach the end of this series, it raises the question “what have we learned?”. If you’ve read through it all, you’ve learned you are patient and I’ve learned that writing a series of posts actually takes quite a bit of time. But I digress!

Let’s start at the beginning – what is a modern data platform?

I’ve used the term throughout, but what does it mean? In the introductory post I stated “In today’s modern world however, storing our data is no longer enough, we need to consider much more” and that’s true as organisations now want their data to provide modern data platformcompetitive edge and insights, we also need to ensure we are “developing an appropriate data strategy and building a data platform that is fit for today’s business needs”. In essence those two areas neatly define a modern data platform, storing data is no longer enough and our platform needs to fit today’s rapidly changing demands, integrate with new technologies and give the scale and flexibility we need to turn our data into an asset, all of this while ensuring our data maintains its privacy, security and we maintain governance and control

It’s not storage

While storage plays an important part in any data strategy (our data has to live somewhere) it’s important to realise when we talk about a data platform, it’s not about storage, while the right storage partner plays a crucial part, the choice isn’t driven by media types, IOPS, or colour of bezel, it’s about a wider strategy and ensuring our technology choice enables us to provide the scale, flexibility and security a modern platform demands.

Break down walls

We have also learned that data cannot be stored in silo’s, be that an on-prem storage repository or its modern equivalent the “cloud silo” placing our data somewhere without consideration of how we move it so we can do what we need to with it quickly and easily, is not designing a modern data platform.

Data Insight is crucial

Where our data is held and on what, while important, pales when compared to the managing the futureimportance of insight into how our data is used. Our modern data platform must provide visibility into the who’s, where’s, when’s, what’s and why’s of data usage, who’s accessing it, where is it and when, if ever, are they accessing it, what are they accessing and why. Knowing this, is critical for a modern data platform, it allows us to build retention, security and compliance policies, it allows us to start to build effective data leak protections and be more efficient with our storage and control the costs and challenges that comes with our ever increasing reliance on data.

Without this insight you don’t have a modern data platform.

Data is everywhere

We have also learned that our data is everywhere, it no longer resides in the protected walls of our data centers, it’s living on a range of devices both sat inside and outside of those walls. That’s not just the data we have, it’s also the increasing range of devices creating data for us, our platform needs to be able to ingest, process and control all of it. Protecting data on the very edges of our network to the same degree that we protect, secure and govern that which sits inside our data centers is crucial.

Cloud, cloud and more cloud

Just a few years ago the prognosis for the data industry was that cloud was going to swallow it all and those who looked to use “traditional” thinking around data would be swept away by the cloud driven tide.

080118_0950_Optimisingt1.jpgNow while cloud is unlikely to wipe out all data life as we know it, cloud should certainly play a part in your data strategy, it has many of the attributes that make it an ideal repository, its flexibility, scale, even commercial models make it an attractive proposition.

But it has limits, however ensuring our data platform can integrate cloud where appropriate and maintain all of the enterprise control we need is a core part of a modern platform, you can’t design a modern platform without considering cloud.

It’s a platform

The reason I used the word platform, is because that is what it is, it’s not one component, it is built up of multiple components, as I’ve shown here, it’s storage, data management, governance, control, be it in the datacentre, on the edges of your network or utilising the cloud.

The days of our data just been about one element are gone, we need a strategy that looks at how we use data in its entirety.

Building a modern data platform

The point of this series has been to provide some practical examples of the tools and technologies I’ve used building modern data platforms. Not every platform uses all of these technologies all of the time and it doesn’t have to be these specific ones to build your platform. What is more important is the concept of a data platform and hopefully this series has introduced you to some areas you may not have considered previously and will help you design a platform to get the very best from your data assets.

If you have any questions, please leave a comment on the site, or contact me on twitter @techstringy or LinkedIn

If you’ve missed any of the series head back to the introduction where you’ll find links to all of the parts of the series.

Thanks for reading.

Advertisements

Building a modern data platform – exploiting the cloud

No modern data platform would be complete if we didn’t talk about the use of public cloud. Public cloud can play a very important part in building a modern data platform and provide us with capabilities we couldn’t get any other way.

In this part of our series we look at the benefits of public cloud, the challenges of adoption and how to overcome them and ensure we can embrace cloud as part of our platform.

Why is public cloud useful for our data?

If we look at the challenges normally associated with traditional approaches to data storage, scale, flexibility, data movement, commercials, then it quickly becomes clear how cloud can be valuable.

While these challenges are common in traditional approaches, these are the areas were public cloud is strongest. It gives us scale that is almost infinite, a consumption model were we pay for what we need as we need it and of course flexibility, the ability to take our data and do interesting things with it once it’s within the public cloud. From analytics and AI to the more mundane backup and DR, flexbility is one of the most compelling reasons for considering public cloud at all.

While the benefits are clear, why are more organisations not falling over themselves to move to cloud?

What’s it lacking?

It’s not about what public cloud can do, it is more about what it doesn’t that tends to stop organisations wholeheartedly embracing it when it comes to data assets.

As we’ve worked through the different areas of building a modern data platform our approach to data is about more than storage, it’s insight, protection, availability, security, privacy and these are things not normally associated with native cloud storage and we don’t want our move to cloud to mean we lose all of those capabilities or have to implement and learn a new set of tools to deliver them.

Of course there is also the “data gravity” problem, we can’t have our cloud based data siloed away from the rest of our platform, it has to be part of it, we need to be able to move data in to the cloud, out again, between cloud providers, all while retaining enterprise control and management.

So how do we overcome these challenges?

How to make the cloud feel like the enterprise?

When it comes to the modern data platforms, NetApp have developed into an ideal partner for helping to integrate public cloud storage. If we look back at part one of this series (Building a modern data platform-the storage) we discussed NetApp’s data services which are built into their ONTAP operating system making it the cornerstone of their data fabric strategy. What makes ONTAP that cornerstone is, as a piece of software, the ability for it to be installed anywhere, which today also means public cloud.

Taking ONTAP and its data services into the cloud provides us with massive advantages, it allows us to deliver enterprise storage efficiencies, performance guarantees and the ability to use the enterprise tools we have made a key part of our platform with our cloud based data as well.

NetApp has two ways to deploy ONTAP into public cloud. It can be installed as Cloud Volumes ONTAP, a full ONTAP deployment on top of native cloud storage, providing all of the same enterprise data services we have on-prem and extend them into the cloud and seamlessly integrate them with our on-prem data stores.

An alternative and even more straightforward approach, is having ONTAP delivered as a native service, no ONTAP deployment or experience necessary. You order your service enter a size, performance characteristics and away you go, with no concern at all with underlying infrastructure, how it works and how it’s managed. You are provided with enterprise class storage with data protection, storage efficiencies and performance service levels previously unheard of in native cloud storage, in seconds.

It’s not a Strategy without integration

While adding enterprise capabilities are great, the idea of a modern data platform relies on having our data in the location we need it, when we need it while maintaining management and control. This is where the use of NetApp’s technology provides real advantage. The use of ONTAP as a consistent endpoint provides the platform for integration, allowing us to use the same tools, policies and procedures at the core of our data platform and extend this to our data in the public cloud.

NetApp’s SnapMirror provides us with a data movement engine so we can simply move data in and out of and between clouds. Replicating data in this way means that while our on-prem version can be the authoritative copy, it doesn’t have to be the only one, replicating a copy of our data to a location for a one off task, which once completed can then be destroyed, is a powerful capability and an important element of simplifying the extension of our platform into the cloud.

Summary

Throughout this series we have asked the question “do we have to use technology X to deliver this service?” the reality is of course no, but NetApp are a key element of our modern data platforms because of this cloud integration capability, the option to provide consistent data services across multiple locations is extremely powerful allowing us to take advantage of cloud while maintaining our enterprise controls.

While I’ve not seen any other data services provider coming close to what NetApp are doing in this space, the important thing in your design strategy, if it is to include public cloud, is ensure you have appropriate access to data services, integration, management and control, it’s crucial that you don’t put data at risk or diminish the capabilities of your data platform by using cloud.

This is part 6 in a series of posts on building a modern data platform, you can find the introduction and other parts of this series here.

Assessing the risk in public cloud – Darron Gibbard – Ep72

As the desire to integrate public cloud into our organisations IT continues to grow, the need to ensure we maintain control and security of our key assets is a challenge but one that we need to overcome if we are going to use cloud as a fundamental part of our future IT infrastructure.

The importance of security and reducing our vulnerabilities is not, of course, unique to using public cloud, it’s a key part of any organisations IT and data strategy. However, the move to public cloud does introduce some different challenges with many of our services and data now sitting well outside the protective walls of our datacentre. This means that if our risks and vulnerabilities go unidentified and unmanaged it can open us up to the potential of major and wide-reaching security breaches.

This weeks Tech Interviews is the second in our series looking at what organisations need to consider as they make the move to public cloud. In this episode we focus on risk, how to assess it, gain visibility into our systems regardless of location and how to mitigate the risks that our modern infrastructure may come across.

To help discuss the topic of risk management in the cloud, I’m joined by Darron Gibbard. Darron is the Managing Director for EMEA North and Chief Technology Security Officer for Qualys with 25 years’ experience in the enterprise security, risk and compliance industry, he is well placed too discuss the challenges of public cloud.

In this episode we look at the vulnerabilities that a move to cloud can create as our data and services are no longer the preserve of the data centre. We discuss whether the cloud is as high a risk as we may be led to believe and why a lack of visibility to risk and threats is more of a problem than any inherent risk in a cloud platform.

Darron shares some insight into building a risk-based approach to using cloud and how to assess risk and why understanding the impact of a vulnerability is just, if not more useful that working out the likelihood of a cloud based “event”.

We wrap up with a discussion around Qaulys’s 5 principles of security and their approach to transparent orchestration ensuring that all this additional information we can gather can be used effectively.

The challenges presented around vulnerability and risk management when we move to public cloud shouldn’t be ignored, but it was refreshing to hear Darron presenting a balanced view and discussing that the cloud is no riskier than any enterprise environment when managed correctly.

Qualys are an interesting company with a great portfolio of tools, including a number that are free to use and can assist companies of all sizes to reduce their risk exposure both on-prem and in the cloud, to find out more about Qualys you can visit www.qualys.com.

You can also contact Darron by email dgibbard@qualys.com or connect with him on LinkedIn.

Thanks for listening.

For the first show in this series then check out – Optimising the public cloud – Andrew Hillier – Ep71

Protecting 365 – a look at Veeam Backup for Office 365

Recently Veeam announced version 2.0 of their Backup for Office 365 product this extended the functionality of its predecessor with much needed support for SharePoint and OneDrive for business. While looking into the release and what’s new it prompted me to revisit the topic of protecting Office 365 data, especially the approach of building your own solution to do so.

Back in April I wrote a post for Gestalt IT (“How to protect Office 365 data”), the basics of which considered the broadly held misconception that Microsoft are taking care of your data on their SaaS platform. While Microsoft provide some protection via retention and compliance rules and a 30-day rolling backup of OneDrive, this is not a replacement for a robust enterprise level data protection solution.

The article examined this issue and compared two approaches for dealing with the challenge, either via SaaS (NetApp’s SaaS backup platform was used as an example) or doing it yourself with Veeam. The article wasn’t intended to cover either approach in detail but to discuss the premise of Office 365 data protection.

This Veeam release though seemed like a good opportunity to look in more detail into the DIY approach to protecting our Office 365 data.

Why flexibility is worth the work

One of the drivers for many in the shift to 365 is simplification, removing the complexity that can come with SharePoint and Exchange deployments. It then surely follows that if I wanted simplicity, I’d want the same with my data protection platform. Why would I want to worry about backup repositories, proxy and backup servers or any other element of infrastructure?

The reality however, is when it comes to data protection, simplification and limiting complexity may not be the answer. Simplicity of SaaS can come at a price of reducing our ability to be flexible enough to meet our requirements, for example limiting our options to;

  • Have data backed up where we want it.
  • Deal with hybrid infrastructure and protect on-prem services.
  • Have full flexibility with restore options.

These limitations can be a problem for some organisations and when we consider mitigation against provider “lock-in” and the pressures of more stringent compliance, then you can see how for some, flexibility quickly overrides the desire for simplicity.

It is this desire for flexibility that makes building your own platform an attractive proposition. We can see with Veeam’s model the broad flexibility this approach can provide;

Backup Repository

Data location is possibly the key deciding factor when deciding to build your own platform, Veeam provide the flexibility to store our data in our own datacentre, a co-lo facility, or even a public cloud repository. Giving the flexibility to meet the most stringent data protection needs.

Hybrid Support

The next most important driver for choosing to build your own solution, is protecting hybrid workloads. While many have embraced Office365 in its entirety, there are still organisations who, for numerous reasons, have maintained an on-premises element to their infrastructure. This hybrid deployment can be a stumbling block for SaaS providers, with an Office 365 focus only.

Veeam Backup for Office365 fully supports the protection of data both on-prem and in the cloud, all through one console and one infrastructure, under a single licence. This capability is hugely valuable, simplifying the data protection process for hybrid environments and removing any need to have multiple tools protecting the separate elements.

Recovery

It’s not just backup flexibility when building your own platform that has value, it is also the range of options this can bring to recovery. This flexibility to take data backed up in any location and restore it to multiple different locations is highly valuable and sometimes an absolute necessity for anything from practicality to regulatory reasons.

What’s the flexibility cost?

Installation

Does this extra flexibility come with a heavy price of complexity and cost? In Veeam’s case no, they are renowned for simplicity of deployment and Backup for Office 365 is no different. It requires just the usual components of backup server, proxy, backup repository and product explorers with the size of the protected infrastructure dictating the scale of the protection platform.

There are of course limitations (Backup for Office 365 System Requirements) one major consideration is bandwidth, it’s important to consider how much data you’ll be bringing into your backup repository both initially and for subsequent incremental updates. While most SaaS providers will have substantial connectivity into Microsoft’s platform for these operations, you may not.

Licencing

A major benefit of software as a service is the commercial model, paying by subscription can be very attractive and can be lost when deploying our own solution. This is not the case with Backup for Office 365 which is licenced on a subscription basis.

Do it Yourself V as a Service

The Gestalt IT article ended with a comparison of the “pro’s and Cons” of the two approaches.

Do It Yourself

As A Service

Pro’s

Cons

Pro’s

Cons

Flexibility Planning Simplicity Lack of control
Control Management Overhead Lower Management Overhead Inability to customise
Customisation Responsibility Ease of Deployment Cloud only workloads
Protect Hybrid Deployments Data Sovereignty

I think these points remain equally relevant and when deciding what approach is right for you, regardless of what we’ve discussed here with Veeam’s offering. If SaaS is the right approach, it remains so, but If you do take the DIY approach, then I hope this post gives you an indication of the flexibility and customisation that is possible and why this can be crucial as part of your data protection strategy.

If building your own platform is your chosen route then Veeam Backup for Office365 V2 is certainly worthy of your consideration, But regardless of approach remember the data sat in Office365 is your responsibility, make sure its protected.

If you want to know more, you can contact me on twitter @techstringy or check out Veeam’s website.

Getting your VeeamON!

Recently software vendor Veeam held its 2018 VeeamON conference in Chicago. VeeamON was one of my favourite conferences of last year, unfortunately I couldn’t make it out this time, but I did tune in for the keynote to listen to the new strategy messages that were shared.

The availability market is an interesting space at the minute, highlighted by the technical innovation and talent recruitment you can see companies like Veeam, Rubrik and others making. Similar to the storage industry of 5 years ago, the data protection industry is being forced to change its thinking with backup, replication and recovery no longer enough to meet modern demands. Availability is now the primary challenge, and not just of the data in our datacentre but also that sat in service providers, on SaaS platforms or with the big public hyperscalers, we need our availability strategy to cover all of these locations.

As with the storage industry when it was challenged by performance and the emergence of flash, two things are happening; New technology companies are emerging offering different approaches and thinking to take on modern challenges that traditional vendors are not addressing. But that challenge also inspires those established vendors, with experience, proven technologies, teams and budgets to react and find answers to these new challenges, well at least it encourages the smart ones.

This is where the availability industry currently sits and why the recent VeeamON conference was of interest. Veeam’s position is interesting, a few years ago they were the upstart with a new way of taking on the challenge presented by virtualisation. However, as our world continues to evolve so do the challenges, cloud, automation, security, governance and compliance just a few of the availability headaches many of us face and Veeam must react to.

One of the things I like about Veeam (and one of the reasons I was pleased to be asked to be a part of their Vanguard program this year) is that they are a very smart company, some of the talent acquisition is very impressive and the shift in how they see themselves and the problem they are trying to solve is intriguing.

VeeamON 2018 saw a further development of this message as Veeam introduced their 5 stages of intelligent data management which sees them continue to expand their focus beyond Veeam “The backup company”. The 5 stages provide the outline of a maturity model, something that can be used to measure progress towards a modern way of managing data.

Of these 5 stages, many of us are on the left-hand side of the graph with a robust policy-based backup approach as the extent of our data management. However, for many this is no longer appropriate as our infrastructures become more complex, changing more rapidly with data stored in a range of repositories and locations.

This is coupled with a need to better understand our data for management, privacy and compliance reasons, we can no longer operate an IT infrastructure without understanding at the very least where our data is located and what that means for its availability.

In my opinion, modern solutions must provide us with a level of intelligence and the ability to understand the behaviour of our systems and act accordingly. This is reflected on the right-hand side of Veeam’s strategy, that to meet this modern challenge will demand increasingly intelligent systems that can understand the criticality of a workload or what is being done to a dataset and act to protect it accordingly.

Although Veeam aren’t quite doing all of that yet, you can see steps moving them along the way, solutions such as Availability Orchestrator which takes the complexities of continuity and delivers automation to its execution, documentation and ongoing maintenance, are good examples.

It’s also important to note that Veeam understand they are not the answer to all of an organisations data management needs, they are a ultimately a company focussed on availability, but what they do realise is that availability is crucial and far beyond just recovering lost data, this is about making sure data is available “any data, any app, across any cloud” and they see the opportunity in becoming the integration engine in the data management stack.

Is all this relevant? Certainly, a major challenge for most businesses I chat with is how to build an appropriate data strategy, one that usually includes only having the data they need, to know how it’s been used and by who, where it is at any given time and having it in the right place when needed so they can extract “value” and make data driven decisions. This can only be achieved with a coherent strategy that ties together multiple repositories and systems, ensures that data is where it should be and maintains the management and control of that data across any platform that is required.

With that in mind Veeam’s direction makes perfect sense, with the 5 steps to intelligent data management model providing a framework upon which you can build a data management strategy, which is hugely beneficial to anyone who is tasked with developing their organisations data management platform.

In my opinion, Veeam’s direction is well thought out and I’ll be watching with interest in not only how it continues to develop, but importantly how they deliver tools and partnerships that allow those invested in their strategy to successfully execute it.

You can find more information on the announcements from VeeamON on Veeam’s website here www.veeam.com/veeamon/announcements