Skip to content

Why You Need Backup For Google Workspace

Could Bespoke Malware Target Your Organization?

How safe is your network from unauthorized actors? More and more, hackers are deploying bespoke malware and ephemeral injections to compromise networks in compliance-heavy sectors like financial services, government, healthcare, and technology.  

What is Bespoke Malware?

Bespoke malware is highly targeted and custom-designed malicious software that has been modified to evade traditional detection systems. So, while your typical malware is designed to target as many systems as possible, bespoke attacks are tailored to a specific target. You can think of bespoke attacks as “fishing with a spear” rather than “fishing with a net.” It’s precise, stealthy, and has a specific prey in mind.  

 Sometimes cybercriminals will design bespoke malware to target a specific enterprise. Other times they will go after a particular industry, leveraging the knowledge that organizations within sectors tend to use similar business systems.  

What Does Bespoke Malware Do?

Typically, custom-made malware is stripped of elements that would generally alert security teams to its presence. Cybersecurity teams know when an attacker has infiltrated the network due to something called Indicators of Compromise (IOCs).  

IOCs act as flags the security team uses to detect strange activity that is evidence of network compromise. Some common Indicators of Compromise include unusual outbound network traffic, geographical irregularities (say, login attempts from far away countries), and anomalies in privileged user account activity. But bespoke malware typically doesn’t contain these IOCs or others, making them nearly impossible to detect using traditional security systems.  

Many different types of bespoke malware exist, like bespoke ransomware and others, and the specific actions the malware takes will differ depending on the target. However, custom malware typically aims to access and exfiltrate data that can be sold to other bad actors.  

Ephemeral Malware on the Rise

In addition to bespoke malware, ephemeral malware is rising as the next big thing in advanced malware. Ephemeral malware exists solely in memory and disappears when the infected system is rebooted. But how does ghostly ephemeral malware differ from common malware? After all, doesn’t all malware start in memory? 

Well, while almost all malware starts in memory, this is typically a stepping stone. Most malware wants to establish a permanent foothold on the infected system, becoming persistent, so it can carry on working even after the computer is rebooted.  

By contrast, ephemeral malware, sometimes called Advanced Volatile Threats (AVTs), resides solely in the computer’s random access memory (RAM) or volatile memory but makes no strides to become permanent. This means it’s wiped clear once the system is rebooted, but the malware has a short window to steal data before that happens. And because the malware never becomes persistent, it stays below the radar of conventional anti-malware programs.  

Bespoke Malware in Action

Over recent years we’ve seen a spike in bespoke malware targeting specific industries.  

SamSam ransomware is one such example. SamSam predominantly targeted healthcare organizations and local government organizations within the US. And while SamSam wasn’t the first software that hijacks systems and then demands a ransom, the way it went about this was unique.  

Other ransomware like WannaCry and GandCrab would immediately lock files and demand a ransom after infecting the network. Instead, SamSam quietly infiltrates the network and monitors network and user activity so that it can penetrate deeper into the systems. Once it’s reached as much of the network as possible, it silently deletes or sabotages the backups before locking all the files. Crucially, because SamSam used cleanup files for standalone executables and some variants were fully executed in memory, detecting the malware was extremely challenging 

Another example is ProjectSauron, a sophisticated information exfiltration malware that spied on government computers for over five years. It was used to steal passwords, encryption keys, configuration files, and log stores 

 A more recent example comes from April 2022, when the Cybersecurity and Infrastructure Security Agency (CISA), Department of Energy (DOE), National Security Agency (NSA), and FBI released details of a custom malware designed to control a range of industrial control systems and supervisory control and data acquisition (SCADA) devices. As a result, the US security agencies urged companies in the energy sector to implement more stringent detection and mitigation processes to avoid falling victim to the bespoke malware2 

What Types of Organizations Are at Risk?

Bespoke malware and ephemeral infections are highly targeted, which means cybercriminals have put considerable thought into who to target. Or in other words, if they’re going to spend ample time and resources designing stealthy malware, it needs to be worth it.  

Typically, this means large organizations and organizations in compliance-heavy industries like healthcare, telecoms, technology, finance, and banking are the intended target. Organizations that fall into these categories have vast troves of sensitive data that can fetch a lucrative sum on the dark web or be leveraged for future cyber attacks. And in the case of custom ransomware attacks, companies in compliance-heavy sectors might be more likely to pay up to reduce disruption to critical systems (lives are at stake when a hospital’s files are locked, for example).  

In Conclusion

Bespoke malware is evasive by design, which makes mitigating it particularly challenging. However, that doesn’t mean you have to sit back and wait for an attack to happen. 

Leveraging modern and proven cybersecurity tools and techniques can go a long way to protecting you from this type of attack. For example, network segmentation, whereby intruders are contained within segments and can’t move laterally throughout the network, is a great idea. Similarly, deploying more stringent network access controls and Zero Trust models can be very effective. Advanced AI network monitoring can also help you detect anomalous behavior on the network, helping you stop an attack in its tracks.  

Bespoke attacks present an unprecedented risk to large organizations, and failing to respond to the dynamic nature of this threat can leave you vulnerable. Cybercriminals are investing in more advanced tools, so you should be too.  

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Portnox
Portnox provides simple-to-deploy, operate and maintain network access control, security and visibility solutions. Portnox software can be deployed on-premises, as a cloud-delivered service, or in hybrid mode. It is agentless and vendor-agnostic, allowing organizations to maximize their existing network and cybersecurity investments. Hundreds of enterprises around the world rely on Portnox for network visibility, cybersecurity policy enforcement and regulatory compliance. The company has been recognized for its innovations by Info Security Products Guide, Cyber Security Excellence Awards, IoT Innovator Awards, Computing Security Awards, Best of Interop ITX and Cyber Defense Magazine. Portnox has offices in the U.S., Europe and Asia. For information visit http://www.portnox.com, and follow us on Twitter and LinkedIn.。

“When it comes to ransomware attacks, it’s a matter of when, not if.”

Ransomware attacks are on the rise — in the first half of 2021, the average amount paid by organizations to perpetrators of was $570,000, an increase of 171% over the previous year. (1)

Last year also saw a 93% increase in the overall number of ransomware attacks (2) – a trend that is only likely to continue. While such attacks were once limited to outlandish movie plots, they’ve become an all-too-real problem for organizations of all sizes. In fact, when it comes to ransomware attacks, it’s more likely to be a question of when, not if.

Our concern at Keepit is that the regularity of ransomware attacks may lead to them eventually being dismissed as just a cost of doing business. But by choosing to pay the ransoms demanded, companies are powering a vicious cycle where the proceeds fuel increased cybercrime. (And paying a ransom does not guarantee getting your data back, as documented in ) the report ‘The Long Road Ahead to Ransomware Preparedness’ from ESG

It’s vital for the sake of commerce – and for society – that companies, governments, and law enforcement agencies come together to find long-term solutions to ransomware attacks.

In the short-term, we encourage companies to invest in a third-party backup and recovery service to minimize the threat posed by encrypted malware. The more secure your data is—and the quicker you’re able to recover it—the less worried you need to be about ransomware attacks.

At best, an attack won’t affect business continuity – it’ll just be a nuisance rather than a crisis. If you know your data is safe, you don’t have to pay the bad guys’ ransom. Problem solved.

How Keepit enables business continuity and disaster recovery by safeguarding SaaS data in 7 steps

 

Summing Up
The disruptive power of ransomware attacks in 2022
An increasingly common threat, ransomware attacks are forecast to cost victims around $265 billion annually by 2031. (3) With conventional data recovery times often taking weeks or even months, the disruption to companies can be catastrophic in terms of financial costs to your business. But the damage goes beyond the bottom line. Additional impacts of ransomware attacks in 2022 are likely to include:

  • Intellectual property cost – temporary or permanent loss of sensitive or proprietary information can be enormously damaging.
  • Business continuity – disruption is frustrating and costly as companies struggle to restore data and operations
  • Reputational cost – a ransomware attack can damage customer perception of the company and impact digital trust.

Why Keepit is the answer
Keepit backs up to an independent cloud, separate from your SaaS vendor’s environment, which means your data can be accessed completely independent from SaaS application availability. True backup—immutable and tamperproof on a separate logical infrastructure — is your answer to ransomware attacks.

For more details about Keepit’s dedicated SaaS data protection, read about our security on our website

References

  1. Research from Palo Alto suggests the average ransom in the first half of 2021 is $570,000 USD, an increase of 171% over the year prior; see Average Ransomware Payment Hits $570,000 in H1 2021 [Dark Reading] 
  2. Research from Check Point reports that ransomware incidents increased 93% year over year; see Ransomware attacks increase dramatically during 2021 [Computer Weekly] 
  3. https://cybersecurityventures.com/global-ransomware-damage-costs-predicted-to-reach-250-billion-usd-by-2031/

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Keepit
At Keepit, we believe in a digital future where all software is delivered as a service. Keepit’s mission is to protect data in the cloud Keepit is a software company specializing in Cloud-to-Cloud data backup and recovery. Deriving from +20 year experience in building best-in-class data protection and hosting services, Keepit is pioneering the way to secure and protect cloud data at scale.

Simple Certificate Enrollment Protocol (SCEP): What It Is & Why Should Network Engineers Care About It

There are several factors to consider when distributing certificates to managed devices, making it a massive undertaking. These include public key infrastructure (PKI), integration, gateway setup, configuration settings, certificate enrollment, device authentication, and more. 

Thanks to the Simple Certificate Enrollment Protocol (SCEP), administrators can quickly and easily enroll all managed devices for client certificates without any action from the end-user.

Here we will discuss what exactly the Simple Certificate Enrollment Protocol (SCEP) is and why network engineers should care about it. 

What Is The Simple Certificate Enrollment Protocol (SCEP)?

Digital certificate issuance in big enterprises is simplified, secured, and scalable with an open-source protocol called Simple Certificate Enrollment Protocol (SCEP).

SCEP servers utilize this protocol to give users a one-time password (OTP) through an asynchronous, out-of-band mechanism (OOB). After creating a key pair, the user submits the OTP and certificate signing request to the SCEP server for verification and signature. As soon as the certificate is ready, the user may request it from the SCEP server and then install it.

Digital certificate issuing was labor-intensive until the advent of SCEP and related protocols like Certificate Management Protocol and Certificate Management via CMS. SCEP is widely used in big organizations since it is supported by products from major vendors like Microsoft and Cisco.

After its creators left SCEP inactive in 2010, the project was dormant until it was revitalized in 2015. Apart from that, it is presently a draft that anybody may see as part of the work of the open-source community – the Internet Engineering Task Force (IETF).

Why Should Network Engineers Care About SCEP?

The public key infrastructure provides the most secure and user-friendly authentication and symmetric encryption solution for digital identities. Yet, the ambiguity and scale of certificate deployment for most businesses can challenge their already overworked network engineers.  

Manual deploying and maintaining certificates is tedious and error-prone. Whether an organization delivers a single certificate for a Wi-Fi router or holds several certifications across all networked devices and user identities, the whole process may take up to several hours. It leaves companies vulnerable to breaches, Man-in-the-Middle (MITM), and other forms of network disruption.

Certificates managed manually are more likely to be lost, overlooked, or expire without being replaced, putting businesses at high risk. Therefore, enterprises need the automated and well-organized certificate enrollment standard – the Simple Certificate Enrollment Protocol (SCEP) – due to the many risks associated with administering PKI certificates manually.

The significant benefits of the Simple Certificate Enrollment Protocol (SCEP) include:

  • Hassle-free certificate issuing.
  • Ensuring that certificates are correctly issued and configured across various devices.
  • A fully automated procedure for the issuance of certificates. As a consequence of this, it involves very little to no human participation.
  • A protocol that saves time, lowers operating expenses, and boosts productivity by enabling network engineers to concentrate on other duties rather than doing those chores themselves.

SCEP is a flexible solution that can meet all your network management requirements since it is compatible with most devices and server operating systems. These include Windows, Apple iOS, macOS, and Linux, as well as directory systems such as Active Directory.

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Portnox
Portnox provides simple-to-deploy, operate and maintain network access control, security and visibility solutions. Portnox software can be deployed on-premises, as a cloud-delivered service, or in hybrid mode. It is agentless and vendor-agnostic, allowing organizations to maximize their existing network and cybersecurity investments. Hundreds of enterprises around the world rely on Portnox for network visibility, cybersecurity policy enforcement and regulatory compliance. The company has been recognized for its innovations by Info Security Products Guide, Cyber Security Excellence Awards, IoT Innovator Awards, Computing Security Awards, Best of Interop ITX and Cyber Defense Magazine. Portnox has offices in the U.S., Europe and Asia. For information visit http://www.portnox.com, and follow us on Twitter and LinkedIn.。

What’s New Pandora FMS 763 RRR

Let’s check out together the new features and improvements included in the newest Pandora FMS release: Pandora FMS 763.

What’s new in the latest Pandora FMS release, Pandora FMS 763
NEW FEATURES AND IMPROVEMENTS


New service evaluation option
A new token has been added to be able to choose which services are recursively calculated and which ones are uniquely evaluated with a separate interval to improve the thread load of the service status calculation.

OS Quick Report
A new widget has been added to the dashboards, “OS Quick Report”.

Through this widget you may get a table that will perform counting by the different operating systems that you’re monitoring

Sunburst, new service representation

The possibility of a new service representation has been added.

This will be a centralized view where you may see the details of the tree node through a tooltip when hovering with the cursor.

New Azure Storage plugin – Enterprise plugin
With this plugin you may monitor a storage account and retrieve data from it such as the number of transactions, capacity or latency used by Azure to process a successful request.

It will retrieve data both at the general account level and from blobs, tables, files, and queues.

Link to the library.

New Azure SQL plugin – Enterprise plugin
With this plugin you can retrieve performance and connectivity data from any Azure SQL database. Such as the CPU used, failed or successful connections or firewall blocks.

Link to the library.

Nuevo plugin Gluster fs – Open
With this plugin you may get the statistics of each volume and the bricks of a gluster fs environment. Some of the data that you may see are the total disk, Ordblks or Smblks.

Link to the library.

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About PandoraFMS
Pandora FMS is a flexible monitoring system, capable of monitoring devices, infrastructures, applications, services and business processes.
Of course, one of the things that Pandora FMS can control is the hard disks of your computers.

GREYCORTEX Mendel 3.9 Now Available

June 20, 2022 – We have released a new version of GREYCORTEX Mendel. Version 3.9 is more interactive, safer and allows even deeper data analysis than ever before. We have increased the interoperability of Mendel with other tools and extended the hardware support.

More about GREYCORTEX Mendel 3.9

ESET Leader in G2 summer report for ESET PROTECT Advanced

Bratislava, June 23rd, 2022ESET, a global leader in digital security, has been awarded Leader status in G2’s report for summer 2022. The ESET PROTECT Advanced solution has consistently been highly ranked by G2 users and achieved substantial satisfaction and market presence scores in their Grid® Reports, which represent the democratic voice of real software users, offering benchmarks for product comparison and market trend analysis.

In the summer edition of the G2 Grid® Reports, ESET was a Leader in several of the reports, including for Antivirus Software, Endpoint Management Software, Endpoint Protection Suites, and Mobile Data Security Software. And in terms of the Mid-Market reports aimed at companies with up to 1,000 employees, ESET was a Leader in Mid-Market Endpoint Protection Suites, Mid-Market Antivirus Software, and Mid-Market Endpoint Management Software.

“We are delighted to be ranked as Leaders in the G2 reports. Our objective is to provide the digital security that keeps an organization’s systems working smoothly and securely, and being recognized for our efforts is always an honor,” comments Michal Jankech, VP for the SMB and MSP segment at ESET. “No modern business, large or small, can survive without an effective response in the face of an IT breach. We believe, that by employing ESET´s strong prevention, detection and response technologies delivered in the form our modular ESET PROTECT platform, businesses can benefit from the most densely multilayered and effective protection in the industry.”

For more than 30 years, ESET has continued to invest heavily in multiple layers of proprietary technology that prevent breaches of its customers’ endpoints and systems, by both known and never-before-seen threats. The ESET PROTECT platform has been designed with ESET’s customers in mind, with the main objective being in assisting IT admins to better manage the security risks in their environments.

As a privately owned, tech-focused company, ESET has always taken a science-based, security-first approach, with early adoption of machine learning and cloud computing power to develop its global threat intelligence systems. The company has continuously been named a top player and a leader in the industry for its business solutions.

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About ESET
For 30 years, ESET® has been developing industry-leading IT security software and services for businesses and consumers worldwide. With solutions ranging from endpoint security to encryption and two-factor authentication, ESET’s high-performing, easy-to-use products give individuals and businesses the peace of mind to enjoy the full potential of their technology. ESET unobtrusively protects and monitors 24/7, updating defenses in real time to keep users safe and businesses running without interruption. Evolving threats require an evolving IT security company. Backed by R&D facilities worldwide, ESET became the first IT security company to earn 100 Virus Bulletin VB100 awards, identifying every single “in-the-wild” malware without interruption since 2003.

NAC, SDP, ZTNA & The Future of Access Control

The Future of Access Control

For over a decade, Virtual Private Networks (VPNs) have been providing businesses with secure access for remote workers and end users. As cyberthreats become more rampant and dangerous, communication and network requirements are continuously changing in response. VPNs have begun to show their limited capabilities as they fail to address various security concerns in this ever-changing landscape.

The current VPN technology in use today may not be able to modernize and adapt to the evolving security threats of today. As a result, Software-Defined Perimeters (SDP), Zero Trust Network Access (ZTNA) and Network Access Control (NAC) come into play. Their collective purpose is like a VPN, but they serve as more comprehensive network security solutions.

Network Access Control (NAC)

NAC is a technology designed to provide endpoint authentication and network access controls to ensure only authorized devices are granted access to a network.

Research has demonstrated that NAC and SDP have become essential elements in the development of Zero Trust Networks as they enable monitoring, visibility, and control at the network access layer. Therefore, NAC is a critical foundation for cybersecurity resilience at the access layer.

Moreover, NAC is a technology that utilizes other core technologies in achieving ZTNA. As a result, NAC vendors should provide easy deployment options and software integration. Although network infrastructure evolvement is rapidly changing, many organizations keep hybrid or legacy environments. NAC can be deployed through both on-prem or virtual appliances.

Software-Defined Perimeter (SDP)

SDP solutions are designed to apply the principle of least privilege (POLP), and are software-centric rather than hardware based. It is made to conceal network infrastructure so as not to be seen by external parties and attackers. Once the device and the user are authenticated, SDP builds a one-to-one connection between the server and the device.

To understand how the SDP solution works, think of a door that is always kept locked. Because it is locked, no one can enter the door or even look inside unless the person inside verifies who the person is and what they are doing. When the visitor centers, the person in the house locks the door again.

Zero Trust Networks Access (ZTNA)

ZTNA is a security solution that provides secure remote access to data and services based on defined access control policies. ZTNA is different from VPN as it only grants authenticated users access to the resources needed to perform their job. and helps to eradicate gaps in other secure remote access methods and technologies. In this way, ZTNA works similarly to SDP as they utilize the concept of a dark cloud to stop users from seeing network resources that they are not authorized to access.

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Portnox
Portnox provides simple-to-deploy, operate and maintain network access control, security and visibility solutions. Portnox software can be deployed on-premises, as a cloud-delivered service, or in hybrid mode. It is agentless and vendor-agnostic, allowing organizations to maximize their existing network and cybersecurity investments. Hundreds of enterprises around the world rely on Portnox for network visibility, cybersecurity policy enforcement and regulatory compliance. The company has been recognized for its innovations by Info Security Products Guide, Cyber Security Excellence Awards, IoT Innovator Awards, Computing Security Awards, Best of Interop ITX and Cyber Defense Magazine. Portnox has offices in the U.S., Europe and Asia. For information visit http://www.portnox.com, and follow us on Twitter and LinkedIn.。

Surely nobody would write a web service in C++

A while back, one of my colleagues was hanging out in an online developer forum and some people there were starting up a good old-fashioned language war (the type of exchange where one person familiar with language A will announce its superiority over language B, with which the other person isn’t familiar – not really a productive use of time when you think about it, but a popular pastime nonetheless).

During this debate, one developer confidently proclaims that ‘surely nobody would ever write a web service in C++,’ to which my colleague responds, ‘well that’s exactly what we did here at Keepit.’ This prompted some questions, and this piece is an attempt to explain why we did what we did and to explain how this choice has been working out for us, given that this code base started life about 10 years ago.

To put things into perspective, it will be necessary to start with some minimal background information about this service we set out to build.

What is Keepit?

Keepit is a backup service in the cloud. We will store backup copies of your (cloud) data so that if—or when—your primary data is compromised for one reason or another (but most likely because of ransomware or account takeover via phishing), then you will still have complete daily copies of your data going back as many years as you want.

Years of data. This should make you think.

Several years ago, Microsoft claimed having 350 million seats on their 365 platform, which is one of the cloud platforms that we protect. Let’s say we get just 10% of that market (we should get much more because we are by far the best solution out there, but let’s be conservative for the sake of argument), that means we need to store all data for 35 million people (and that’s just on one of these platforms – we protect several other platforms as well).

It doesn’t end there: being backup, we copy all your changes, and we hold your old data, and that means when you clean up your primary storage and delete old documents, we keep the copy. Many customers want a year or three of retention, but we have customers who pay for 100 years of retention.

One hundred years. That means our great grandchildren will be fixing the bugs we put in our code today. This should make you think too.

Knowing the very high-level goals of our service, let’s talk about requirements for such a thing.

Core system: storage

We knew from the get-go that we would be implementing a storage solution which would need to store a very large amount of data (as everything is moving to the cloud, let’s say a few percent of all the world’s data) for a very long period of time (say a hundred years).

Now everyone in the storage business will talk about SSDs, NVMe, and other high-performance data storage technologies. None of this is relevant for large scale, affordable storage, however. Spinning disks is the name of the game and probably will be for at least another decade.

SSDs are getting the density, sure, but they are still not close to viable from a cost perspective. This means we will be writing all data to rotating magnetic media. When you write data to magnetic media, over the years, your media will demagnetize. That means, if we store a backup on a hard drive today, we probably can’t read it back just ten years from now.

That means we need to regularly move all this data from system to system to keep the data ‘fresh.’ Talking about performance, large capacity hard drives today rotate at 7200rpm, exactly the same speed as back in 1992. Access time is dominated by the rotational latency, which means that this is really an aspect of computers that has been almost at a standstill for 30 years while everything else has become faster in every way. We knew we had to deal with this.

I should probably note here that yes, we are talking about running our software on actual physical computers – no public cloud for us. If you want to go big, don’t do what the big players say you should do, do what the big players do. If public cloud was so great, Microsoft wouldn’t have built their own to run 365 – they would have run it on AWS which was very well established long before Microsoft thought about building 365. This doesn’t mean you can’t prototype on public cloud of course.

To solve our core storage need, we designed a filesystem—basically, an object storage system optimized for storing very large-scale backup data. Clearly, we expect the implementation of this storage system to have a significant lifespan.

We may want to create a better implementation one day in the future when hardware has evolved far beyond what we can imagine today, but it is worth pointing out that the storage systems we use today are very similar in architecture to what they would look like 30 years ago, and I would assume in 30 years from today. Clearly, the core code that manages all of your data is not something you want to re-write every few weeks.

So, to implement this system, we went out looking for which new experimental languages had been invented in the six months leading up to implementation start. No wait, we didn’t.

What we need from a language

There are really two types of languages:

1: Systems programming languages – those that have practically no runtime, where you can look at the code and have a high degree of confidence in understanding exactly what that leads to on your processor, the type of language you would write an operating system kernel in. This would be languages like C, C++, and who knows – maybe Rust or something else.

2: The higher-level languages, which often have significant runtimes. The good ones of these offer benefits that you cannot get in a language without a significant runtime. This would be a language like Common Lisp, but people more commonly talk about C# and Java even though I will argue they only do so because nobody taught them Lisp.

And then you have the other languages that fit various niche use cases. This could be Python, Haskell, JavaScript, and so forth. I don’t mean to talk them down, but they are not reasonable languages for software development of the type we are talking about here; and since what we’re talking about here isn’t actually so special, you could take my argument to mean that they are just not very reasonable languages for software development outside of niche uses, and that would be a fair interpretation of my opinion.

So, to be a little more concrete, what it is that we really need from a language is:

1: It must support the efficient implementation of algorithms and data structures; meaning we must have tight control over memory when we need it, our language must support the actual hardware data types like 64-bit integers on modern processors, etc. So, this rules out Python (not compiled), Ruby (not compiled) and JavaScript (JIT but doesn’t have integers or arrays).

2: When we write code today, the tool chain in 20 years’ time must still support our code with little or no changes. Because we simply can’t rewrite our code every few years. We will get nowhere if that’s what we do. That’s why large, important software systems today are still often written in C – because they started out life in the 80s or 90s and they are still the most significant operating system kernels or database management systems that exist to this day.

Steady evolution is the recipe, not rewrite from scratch every three years. This basically rules out any language that hasn’t been standardized and widely used for at least 10 years before we start the project. Meaning, since we started in 2012, that rules out any language that came out after 2002, so Go, Rust, and many other languages would have been out of the picture. C and C++ would work though.

3: We run on Linux. If you do anything significant with computers on a network, you probably run on Linux, too. We don’t want a language that is ‘ported’ to Linux as a curiosity – like C#. We need a language that is native on Linux with a significant and mature toolchain that is certain to receive significant investment for decades to come. Again, that’s really C and C++.

4: You need to design for failure. Everything from writing to a disk, to allocating the smallest piece of memory, can and will eventually fail. Relying on the developer to check error codes or return values at every single call to a nontrivial function (and too many trivial functions too) is rough. Yes, it can be done and there are impressive examples of this.

I am humbled by software such as the Postgres database or the Linux kernel which are very reliable pieces of software written in C which require such tedious checking. C++, in my experience, with RAII and exceptions, offers a much safer alternative. It is not free, of course – it avoids one set of problems and introduces another. In my experience however, it is less difficult to write reliable software using RAII and exceptions than to rely on developers not missing a single potential error return and correct recovery and cleanup. For this reason, I will prefer C++ over C and over both Rust and Go.

5: Obviously the language must offer sufficiently powerful functionality to make the implementation of a larger application bearable and maybe even enjoyable. In reality, however, if your language has functions, you can accomplish a lot; Fortran got functions in 1958 and since then most languages have had them.

Yes, generic programming is nice in C++. A real programmable language like Common Lisp would be preferable of course. Any other modern programming language will surely have some other feature which was added because it is potentially nice and potentially justifies the existence of the language.

But in reality, the hard part is getting your data structures right. Getting your algorithms right. Knowing what you’re trying to build and then building exactly that, nothing more and nothing less.

If we are honest, most languages would work. However, C++ is a nice compromise: it has some generic programming, the STL is incredibly useful, it offers basic OO concepts, and RAII (and structured error handling).

If we look at the criteria here, there really aren’t that many candidate languages to choose from, even if we compromise a bit here and there. Therefore, the question really isn’t ‘why’ we would write a web service in C++, the question really is ‘why wouldn’t we’ write a web service in C++. Realistically, what else would you use, given the scope of what we’re solving here?

Versatility

Performance matters. Don’t let anyone tell you otherwise. Anyone who says that ‘memory is cheap’ and uses that as an excuse should not be building your large-scale storage systems (or application servers or anything else that does interesting work on large amounts of data).

Donald Knuth said, ‘Premature optimization is the root of all evil’ and I absolutely believe that. However, ‘no optimization and elastic scaling is the root of all public cloud revenue’ is probably also true. Don’t go to extremes – don’t put yourself in a situation where you cannot, at the appropriate time, optimize your solution to be frugal with its resource use. When your solution is ‘elastically scaling’ for you in some public cloud on a credit card subscription, it is very hard to go back and fix your unit economics. Chances are you never will.

The typical computer configuration for a storage server in Keepit is 168 18 TB hard drives attached to a single-socket 32-core 3.4GHz 64-bit processor and 1TiB of RAM. It’s really important to note here that we use only one TiB of RAM for three PiB of raw disk: this is a 3000:1 ratio – it is not uncommon to see general purpose storage systems recommend a 30:1 ratio of disk to RAM (which would require us to run with 100TiB of RAM at which point memory most certainly isn’t cheap anymore). Through the magic of our storage software, this gives us about 2PiB of customer-usable storage in only 11U of rack space. This means we can provide a total of 8PiB of usable storage in a single 44U rack of systems, consuming less than 10kW of power. This matters.

If you run a business, you want to be able to make a profit. Your customers will want you to make a profit, especially if they bet on you having their data 100 years from now. If you want to grow your business with investments, your investors will think this matters. In Keepit, we have amazing unit economics – we got the largest series A round of investment for an IT company in the history of Denmark ever – and part of the reason for that was because of our unit economics. Basically, our storage technology, not least the implementation of it, enabled this.

The choice of C++ has allowed us to implement a CPU- and memory-efficient storage system reliably that uses the available hardware resources to their fullest extent. This ranges from careful layout of data structures in memory to an efficient HTTP stack that exposes the functionality and moves more than a GiB of data per second per server over a friendly RESTful HTTP API on the network. C++ enables and supports every layer of this software, and that is quite a feat.

Let me briefly digress with another note on versatility. I have this personal hobby project where I am developing a lab power supply for my basement lab (because every basement needs a lab). In order to adjust current and voltage limits, I want to use rotary encoders rather than potentiometers.

A rotary encoder is basically an axle that activates two small switches in a specific sequence and by looking at the sequence you can detect if the user is turning the axle in one direction or the other. The encoder signal gets fed to a 1MHz 8-bit processor with 1 kB of RAM and 8 kB of flash for my code.

To implement the code that detects the turning of these encoders, it makes sense to use a textbook, object-oriented approach. Create a class for an encoder. Define a couple of methods for reading the switches and for reading out the final turn data. Declare a bit of local state. Beautifully encapsulated in pure OO style. The main logic can then instantiate the two encoders and call the methods on these objects. I am implementing the software for this project in C++ as well – try to think about that for a moment: The same language that allows us to efficiently and fully utilize a 32-core 3.4GHz 64-bit processor with 1TiB of RAM and 3PiB of raw disk works ‘just as well’ on a 1-core 1MHz 8-bit processor with 1kiB of RAM and 8kiB of flash storage – and the code looks basically the same.

There are not many languages that can stretch this wide and not show the slightest sign of being close to its limit. This is truly something to behold.

The rest of the stack

The storage service exposes a simple RESTful API over HTTP using an HTTP stack we implemented from scratch in C++. Instantiating a web server in C++ is a single line of code – processing requests is as trivial as one could wish for.

I’ve heard plenty of arguments that doing HTTP or XML or other ‘web’ technology work would be simpler in Java or C# or other newer languages, but really, if you write your code well, why would this be difficult? Why would you spend more than a line of code to instantiate a web server? Why would parsing an XML document be difficult?

For XML, we implemented a validating parser using C++ metaprogramming; I have to be honest and say this was not fun all the way through and I couldn’t sit down and write another today without reading up on this significantly first. C++ metaprogramming is nothing like a proper macro system – but it can absolutely solve a lot of problems, including giving us an RNC-like schema syntax for declaring a validating XML parser and generating efficient code for exactly that parser.

This also means when we parse an XML document and we declare that one of the elements is an integer, then either it parses an integer successfully or it throws. If we declare a string, we get the string properly decoded so that we always work on the native data – we cannot ever forget to validate a value and we cannot ever forget to escape or un-escape data. By creating a proper XML parser using the language well, we have not only made our life simpler, we have also made it safer.

The entire software ecosystem at Keepit may revolve around our storage systems, but we have several other supporting systems that use our shared components for the HTTP and XML stack.

One other notable C++ system is our search engine. Like so many other companies, we found ourselves needing a search engine to assist us with providing an amazing end user experience when browsing their datasets. And like so many others we fired up a cluster of Elasticsearch servers and went to work.

Very quickly we got hit by this basic fact that Elastic is great at queries and not very good at updates – and we have many more updates than we have queries. We simply couldn’t get this to scale like we’re used to. What to do?

While struggling with Elastic, we started the ‘Plan-B’ project to create a simple search engine from scratch – this engine has been our only search engine for years now and to this day, the process is still called ‘bsearch.’

Our search engine offers a google-like matching so that you can find your documents even if you misspell them, and it is a piece of technology that we are quite actively developing both to improve matching capabilities across languages and to allow for more efficient processing of much larger datasets, which will open up for other uses in the future.

Of our backend code base, about 81% of our code is C++. Another 16% is Common Lisp. The remaining 3% is Java.

We use Common Lisp in two major areas: For ‘general purpose’ business functions such as messaging, resource accounting, billing, statistical data processing, etc. And we use it for backup dataset processing. These are two very different uses.

The first is a more classical application of the language where performance is maybe less of a concern but where the unparalleled power of the language allows for beautiful implementations of otherwise tedious programs.

The second use is a less traditional use case where enormous datasets are processed and where the majority of the memory is actually allocated and managed outside of the garbage collector – it is truly a high-performance Lisp system where we benefit from the power of the language to do interesting and efficient extractions of certain key data from the hundreds of petabytes of customer data that pass through our systems.

Many people don’t know Common Lisp and may propose that ‘Surely nobody would write a web service in Common Lisp.’ Well, as with all other languages you need to understand the language to offer useful criticism; and the really groundbreaking feature of Common Lisp is its macro system. It is what makes Common Lisp by far the most powerful language in existence by a large margin.

This is nothing like C pre-processor macros; the Common Lisp macro system allows you to use the full power of the language to generate code for the compiler. Effectively, this means the language is fully programmable. This is not something that is simple to understand since there is no meaningful way to do this using C-like language syntax, which is also why the Lisp dialects have a syntax that is fundamentally different from other languages.

In other words, if you do not understand the Lisp syntax, you are not equipped to comprehend what the macro system allows. This is not simple to wrap your head around, but, for example, I can mention that Common Lisp was the first general purpose programming language to get Object Orientation added to it, and this was done not with a change to the language and the compiler, but with a library that contained some macros. Imagine that.

Fortran allows you to implicitly declare the type of variables by using certain letters in the first character of the variable name – just for fun, I implemented that with a macro for Common Lisp. If I wanted to do that with C or C++ or any other language, I would need to extend the compiler.

The idea of using the first character in the name of the variable to implicitly declare its type is of course ridiculous, but there are many little syntactical shortcuts or constructs that can help you in daily life that you may wish was present in your language of choice which you can only hope the language steering committee may one day add to the standard.

With Common Lisp, this is everyday stuff – no need to wait. If you want a new type of control structure or declaration mechanism, just go ahead and build it. The power of this cannot be overstated. C++ metaprogramming (and go generics and everything else) pales in comparison, useful as it is.

Lessons learned

First of all, it really sucks to have multiple languages; you can’t expect everyone to be an expert in all, so by having more than one language, you decimate the effective size of your team. However, we picked Common Lisp to replace a sprawling forest of little scripts done in more languages than I could shake a stick at—meaning we are fortunate to have only two languages on our backend.

C++ and Common Lisp are so different that they complement each other well. Yes, we could have done everything in C++, but there are problems we solve in Common Lisp which would have been much less enjoyable to solve in C++. Now on the downside, we have two HTTP stacks, two XML stacks, two database libraries, two connection pools, and so on and so forth. There is no simple perfect solution here; the compromise we have arrived at is indeed working out very well for us.

We’ve been told many times that recruiting for C++ is hard because recruiting for ‘web technologies’ is so much simpler. Well guess what, finding good JavaScript developers is just as hard as finding good C++ developers in my experience. With Common Lisp it’s different again: it’s harder to find people, but the percentage of the candidates that are actually qualified is higher, so all in all, it’s actually fine. Recruitment is difficult across languages, period.

The best you can do is go to a conference, talk about your tech, and hope that some developers show up at your booth to talk about employment.

Old grumpy man’s advice for youngsters considering a career in software engineering

First of all, seriously consider a computer science education. There exist amazingly qualified people who do not have this and some of them work for us, but in my experience most really good developers have this. It certainly helps to get a foundation of mathematics, logic, and basic computer science. Knowing why things work will make learning new things infinitely simpler.

Learn multiple, properly different programming languages and write actual code in them. You need to experience (by failing) how functions are useful as abstractions and how terrible it is to work with ill-designed abstractions. You need to fail and spend serious time failing.

Make sure one of those languages is a compiled language with little or no runtime: C, C++, Rust, or even Fortran for that matter (not sure Fortran has much long-term perspective left in it though – it’s probably time to say goodbye). Now challenge yourself to write the most efficient implementation of some simple problem – maybe a matrix multiplication for example.

Disassemble the code and look at it. At least get some understanding of the processor instructions and why they are generated from the code you wrote. Learn how cache lines matter. Time your code and find out why your solution isn’t faster than it is. Then make it faster until you can prove to yourself that your instructions pipeline as much as they can, your cache misses are minimal, you don’t wait on register write delays and so on and so forth.

Also, make sure that one of those languages is Common Lisp. It should be a criminal offence for a university to not teach Common Lisp in their computer science curriculum. Read ‘The Structure and Interpretation of Computer Programs – SICP’ too. Even if you will never use Lisp again, knowing it will make you a better developer in any other language.

And finally, as much as I dislike JavaScript, you should learn that, too. The most beautiful backend code will too easily be ignored if you cannot beautifully present its results – and today this means doing something with JavaScript.

Aside from my previous criticisms, you can make working with JavaScript more bearable, for example, by creating your own framework rather than relying on the constantly changing megabyte sized atrocities that your common web projects rely on. However, this is probably a topic for future discussion.

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Keepit
At Keepit, we believe in a digital future where all software is delivered as a service. Keepit’s mission is to protect data in the cloud Keepit is a software company specializing in Cloud-to-Cloud data backup and recovery. Deriving from +20 year experience in building best-in-class data protection and hosting services, Keepit is pioneering the way to secure and protect cloud data at scale.

Is Litigation Hold a Reasonable Replacement for Backup in Microsoft 365?

We get asked this question often, and at face value, it’s easy to see how one could equate litigation hold with backup – both have something to do with ‘preserving’ data. However, the reality is that backup and litigation hold differ on many points, and any company that fails to understand the differences between them (and the utility of each) will eventually learn the repercussions the hard way. Let’s explore the key differences between litigation hold and backup.

What Is Litigation Hold?

The term ‘litigation hold’ comes from US case law (2003, Zubulake v. UBS Warburg) where the judge ruled: ‘once a party reasonably anticipates litigation, it must suspend its routine document retention/destruction policy and put in place a ‘litigation hold’ to ensure the preservation of relevant documents.’

In 2010, Microsoft introduced a litigation hold (sometimes referred to as legal hold) retention feature for Microsoft Exchange to support eDiscovery. The feature was intended primarily as a way of preserving data should there be a legal need to preserve it for access and viewing during a litigation. Think of it as being for documentation purposes, not as a way to restore data back in place to operating platforms like Microsoft 365.

Microsoft later added the ability to create what they call in-place holds, which are holds based on a query (such as “find all messages containing the phrase ‘Project Starburst’). The back-end implementation of litigation and in-place holds are slightly different; you can see more details in Microsoft’s documentation

Let me say it again, slightly differently: Litigation hold wasn’t designed with the intention of serving as a backup service. Yet, some still try to rely on it as a backup solution, particularly to make ends meet when not having a designated data security plan (including a third-party backup solution), with the reasoning that “some sort of data preservation is better than none, right?” 

However, there are many drawbacks and substantial risks associated with these types of setups that lead to a risky, false sense of data security. Some of the shortcomings and risks of relying on litigation hold as a backup are: 

  • Data storage quotas capped at only 110 GB 
  • Some eDiscovery features require additional-cost licenses; if you don’t buy the licenses, you can’t use the features 
  • User mailbox data is only kept while an Exchange Online license is assigned to the user. When a user leaves or becomes inactive, removing the license will eventually remove the data.   
  • Recovering data needs an administrator and is a time-consuming process 
  • The held data is not physically separate from the original copy  

The bottom line is that you can’t depend on litigation hold or in-place holds as mechanisms for general-purpose recovery from mistakes or disasters. That’s not what they’re meant for, and you run the risk of losing data if you try to use them for that purpose.

What Is Backup?

Backup, by definition, provides one or more additional copies of your data, stored in a location physically separate from that of your primary dataset. Physical separation is a fundamental facet of backup, since storing your backup data in the same location as the primary data represents a single point of failure.  Effectively, there is no data redundancy in these types of setups. 

With traditional on-premises backup, the physical separation rule meant having an off-premises backup stored in another building – so that in the event of a disaster, e.g. a fire in one building, would not destroy all your data. For cloud backup, it’s fair to ask ‘what cloud does my backup data go to?’ The answer is usually either ‘Microsoft Azure’ or ‘Amazon Web Services.’ Ideally, you want that data going to a cloud not operated by your SaaS application vendor (so, it wouldn’t be fair to put your Microsoft 365 data into Azure); otherwise, you’re violating the physical-separation rule.  

Any service that is not providing this separation of copies is not—and should not be—considered a true backup. 

At Keepit, we talk a lot about the ‘3 Ms’ that can cause data loss: mistakes made by people; mishaps at the SaaS application vendor; and malicious actions from inside or outside the organization.  

Following data protection best practices, a properly executed backup scheme provides against all three of the Ms if anything should happen to the primary (original) dataset: malicious action in the form of a ransomware attack or a disgruntled employee; mistakes where someone with legitimate access accidentally deletes important data (or needs to back out changes they didn’t want to keep); and mishaps, where the service provider has an outage or data loss. Litigation holds can’t protect you against all 3 of the Ms: there’s no physical separation, limited ability to do large-scale restores, and no real concept of version control.  

What to Look for In a Cloud SaaS Backup Solution

Besides the must-have features of data redundancy and availability, a worthy backup solution will offer a multitude of convenience and productivity-boosting tools and services, further distancing it from litigation hold. The first thing to look for is a solution that’s purpose-built for the cloud, not a refurbished or reskinned on-premises solution. Rather, a good, dedicated third-party backup solution.  

Here are some of the key benefits to look for in a dedicated third-party backup solution: 

  • Simple, quick restoration of the data you need, when and where you need it, in the format you need it 
  • Direct restore from live storage, with no waiting for offline or near-line storage 
  • An intuitive interface for quickly and easily finding and previewing specific files or messages before storing them 
  • Secure, immutable storage in an independent cloud 
  • Flexible geographic storage options to cover your data sovereignty requirements 
  • A predictable and transparent cost model, with no hidden surprise charges for data ingress, egress, or storage 

For more insight into data protection in the cloud era, get an in-depth look via the e-guide on Leading SaaS Data Security. Or, if you’d like to learn more about Keepit backup and recovery services for Microsoft 365, Salesforce, Google Workspace, and others, visit this page. 

About Version 2 Limited
Version 2 Limited is one of the most dynamic IT companies in Asia. The company develops and distributes IT products for Internet and IP-based networks, including communication systems, Internet software, security, network, and media products. Through an extensive network of channels, point of sales, resellers, and partnership companies, Version 2 Limited offers quality products and services which are highly acclaimed in the market. Its customers cover a wide spectrum which include Global 1000 enterprises, regional listed companies, public utilities, Government, a vast number of successful SMEs, and consumers in various Asian cities.

About Keepit
At Keepit, we believe in a digital future where all software is delivered as a service. Keepit’s mission is to protect data in the cloud Keepit is a software company specializing in Cloud-to-Cloud data backup and recovery. Deriving from +20 year experience in building best-in-class data protection and hosting services, Keepit is pioneering the way to secure and protect cloud data at scale.