If you had asked me in the 1990s whether most people in the tech world would be using a UNIX or Linux shell to manage their servers, I would have chuckled and said “No.” But two decades later we find ourselves in a world where 96% of the cloud is comprised of Linux servers running Web apps made with open source frameworks, and the majority of software development is done on UNIX (macOS) or Linux workstations. Nearly all network hardware, security appliances, smart appliances, cars, wearables and IoT devices also run UNIX or Linux, as do our smartphones and tablets. Want to secure your systems or network, or test your security using ethical hacking? You’ll need UNIX or Linux to do that as well.

In this blog post I’ll explain how we got to this point, as well as discuss the methods used to deploy UNIX and Linux today.

UNIX

UNIX was created by Dennis Ritchie and Ken Thompson of AT&T Bell Labs in 1969, and was rewritten in the C programming language (created by Dennis Ritchie) shortly thereafter. It is considered the “grandfather” of all operating systems, and led to the creation of nearly all other modern technologies, including the Internet.

When UNIX was created, AT&T was restricted by a federal court order preventing them from marketing it. As a result, AT&T sold the UNIX source code off to several companies, which developed their own flavours of UNIX, as well as gave free copies out to several universities including the University of California at Berkeley from which we have BSD (Berkeley Software Distribution) UNIX. These vendors and universities are largely responsible for the mainstream development of UNIX since the late 1970s. Some common flavours of UNIX that arose in the 1980s onwards include SunOS, Solaris, OpenSolaris, IBM AIX, Apple AUX, DEC Ultrix, True64 UNIX, SGI IRIX, SCO UNIX, Novell UnixWare (later SCO), NEC UNIX, Xenix, HP-UX, QNX, MINIX, FreeBSD, NetBSD, OpenBSD, and NeXTSTEP.

macOS

macOS (formerly called Mac OS X) is merely a UNIX flavour. It is based on NeXTSTEP UNIX, which Apple obtained in the late 1990s following their purchase of NeXT Computer. Similarly, the iOS and iPadOS variants of macOS are also UNIX flavours. So, if you use an iPhone, iPad or Mac computer, you’re using UNIX. The only UNIX flavours you’ll see today outside of the Apple ecosystem are the BSD ones (FreeBSD, NetBSD & OpenBSD), which are almost exclusively used as servers in the cloud (e.g. Netflix uses FreeBSD on all of their servers).

FSF, GNU, and Open Source

Linux finds its roots in the UNIX hacker culture (the term hacker now refers to those that use technology maliciously). In 1983, one famous hacker named Richard Stallman started the Free Software Foundation (FSF) to encourage computer science students and freelance software developers worldwide to create and improve free software for the UNIX platform. Richard had seen success as a result of collaboration between software developers in the Artificial Intelligence Labs at the Massachusetts Institute of Technology (MIT) earlier and believed that the sharing of ideas would result in faster software development.

During the mid 1980s, the collaboration seen by UNIX vendors was starting to diminish; thus the FSF quickly bloomed into the GNU Project. The GNU Project aimed to create a free UNIX-like operating system (GNU stands for GNUs Not UNIX). And while the GNU project never came to fruition, several great products came from the GNU community, including the GNU emacs editor. More importantly, the GNU Project led to the release of the GNU General Public License (GPL), which is often referred to as a copyleft license. If you develop software under the GPL, you must make the source code for the program freely accessible to anyone. Moreover, anyone else who modifies (and improves) your source code must also make that source code freely accessible to anyone. In short, the GPL ensures that source code is freely available, forever.

Today, there are many different software licenses that you can use to publish software under. Some are strict copyleft licenses like the GPL. Others allow others more flexibility with how the source code can be used, and are called permissive licenses as a result. Since both copyleft and permissive licenses allow anyone to obtain and modify the source code for a program, they are collectively called open source software licenses, and software created under these licenses are referred to as open source software.

Linux

In 1991, a Finnish student named Linus Torvalds created an improved version of MINIX UNIX called Linux that he released as open source software under the GPL. Since then, thousands of open source software developers around the globe have transformed Linux into a very sophisticated operating system. Linus and his team coordinate the core development of the Linux operating system kernel (the core component that operates the hardware and allocates resources to programs), while other open source developers worldwide create the software packages that provide for the remainder of the operating system (graphical desktop, compilers, editors, services, and so on.). You may obtain different distributions (or distros) of Linux as a result. All Linux distributions share the same kernel and core libraries, yet have different software packaged with the kernel.

There are hundreds of Linux distributions available; some common ones include Red Hat, Fedora, SuSE, Arch, Debian, Ubuntu and Alpine. Google’s Android operating system is also a Linux distribution, as are many other highly-customized Linux distributions used in embedded devices, routers, wearables and smart appliances. Due to its open source nature, Linux can be customized to run on nearly any hardware platform today. In 2018, Gartner estimated that over 90% of the world population was a Linux user in some way, and that Linux comprised 78% of embedded devices, 100% of supercomputers, 83% of smartphones (Android), 75% of tablets (Android), 48% of servers (96% of cloud servers, and 12% of on-premises servers), 93% of network appliances (routers, WAPs, next generation firewalls, etc.) and 7% of desktops (mostly developer workstations and Chromebooks). In short, Linux boasts the most interfaces and software for any computer or mobile device and provides the foundation for the largest systems and organizations in the world.

The Connection Between Linux, Open Source & The Cloud

The physical computer network we call the Internet started as the US government-funded ARPA/NSFnet. In the early 1990s, it renamed the Internet and sold to commercial organizations (called Internet Service Providers, or ISPs). ISPs sold access to this network so that people could to connect to Web servers around the world that hosted HTML webpages. The Internet was the physical network, and the Web server was the “killer app” that led to the popularization of the Internet. The worldwide collection of Web servers hosting webpages was called the World Wide Web (WWW), and you only needed an Internet connection to access it. The open source Linux operating system, alongside the open source Apache Web server allowed anyone with a computer to run a Web server, leading to an explosion of interest surrounding the Internet and WWW. Most believe that the rapid growth of the WWW would not have been possible without Linux or Apache. Can you imagine how many Web servers would have been installed in the 1990s if you had to purchase an expensive Windows NT Server license to run one?

Today, Web servers host complex Web apps instead of webpages, and the worldwide collection of servers hosting Web apps is called the cloud. Since the cloud is simply an evolution of the WWW, it should be no surprise that Linux alongside open source software and developer frameworks had a major part in its evolution and continues to guide its direction.

Of course, Web apps need to be written by developers and sent regularly to servers in the cloud for testing, before they are copied to production servers in the cloud. The software and processes used to get new Web app builds to the cloud is called a devops (developer operations) workflow. New versions of popular Web apps, such as Google, Facebook, Netflix, and Twitter are often pushed dozens of times a day to the cloud using a devops workflow to test new features or bug fixes before they are implemented. And nearly all devops workflows are comprised of software that runs exclusively on Linux or BSD UNIX.

Most security-related and emerging technologies also rely on open source software for rapid development, and these technologies are almost exclusively developed on the Linux platform. Once example of this is blockchain development, whose widely-adopted standard is HyperLedger from The Linux Foundation.

How are UNIX and Linux deployed today?

Nearly all UNIX systems today are macOS or iOS/iPadOS, and can only be deployed legally on Apple hardware. So to get macOS, you must buy an Apple product. If you want to reinstall the OS on that Apple product, Apple has options that allow you to do it easily across the Internet. For the BSD UNIX flavours and distributions of Linux, there are four main ways to deploy them:

1. Install Linux or BSD UNIX directly onto a computer (e.g. laptop, desktop, server).

This method is often called a bare metal installation. Developers who create Linux or BSD UNIX software often perform a bare metal installation on their laptop or desktop, and then install their developer tools on the operating system afterwards. Nearly all major developer tools, such as Visual Studio Code and Android Studio, have Linux versions. Both Lenovo and Dell sell laptops preinstalled with Linux to cater to developers (Dell even calls their XPS laptop line installed with Linux the “Developer Edition”). Many servers within an organization or in the cloud are also bare metal installations of Linux or BSD UNIX that host virtual machines or containers (discussed in 3. and 4.), and Systems Administrators are responsible for installing, managing and securing them.

2. Use Windows alongside the Windows Subsystem for Linux (WSL).

Windows 10 Pro and later allow you to run an entire Linux operating system (including a real Linux kernel) alongside Windows for developing and testing Linux apps. Several Linux distributions for WSL are available in the Microsoft Store, and the integration between Linux and Windows is seamless (even graphical Linux apps run with full graphics hardware acceleration). Developers can run either Windows or Linux developer tools to create Linux apps, and then test them directly on Windows using WSL. System Administrators can also use WSL to centrally manage their Linux servers within the organization or in the cloud, as well as perform ethical hacking and vulnerability assessments for the networks and systems they manage. Unfortunately, BSD UNIX flavours aren’t currently supported by WSL. To learn how to install and configure WSL, visit https://docs.microsoft.com/en-us/windows/wsl/install-win10.

3. Install Linux or BSD UNIX within a virtual machine.

Virtual machines allow you to run multiple operating systems simultaneously on a computer via hypervisor software that manages access to the underlying hardware resources. There are several hypervisors available, including Hyper-V, VMWare, Oracle Virtualbox, Parallels, and UTM. Many servers in the cloud are actually virtual machines running Linux or BSD UNIX on an underlying hypervisor within a datacenter. Software developers that wish to use Windows or macOS developer tools to create Linux or BSD UNIX apps often run the operating system in a virtual machine on their Windows or macOS workstation to test their apps. Similarly, Systems Administrators that wish to configure and test a new Linux or BSD UNIX technology before deploying it almost always do it within a Linux or BSD UNIX virtual machine running on their Windows or macOS workstation.

4. Run Linux or BSD UNIX as a container.

Containers are similar to virtual machines, but do not contain a full operating system, and lack an operating system kernel. As a result, containers must use the kernel of an underlying operating system. Linux containers only have enough filesystem, software libraries and supporting programs to run a particular Web app, which makes them very small and ideal for running in the cloud (cloud servers can run thousands of small containers). To run Linux or BSD UNIX in a container, you must install a container system such as Docker. Developers that create Linux cloud apps often install Docker Desktop on Windows or macOS to run Linux containers. On macOS, Docker Desktop uses a Linux virtual machine (with a Linux kernel) to run Linux containers. On Windows, Docker Desktop leverages the Linux kernel in WSL to run Linux containers. To learn how to install and configure Docker Desktop, visit https://www.docker.com/products/docker-desktop. System Administrators are responsible for deploying and managing containers that are created by developers within their organization or in the cloud. Moreover, many third-party software packages that System Administrators must deploy (e.g. ticketing systems, network & security monitoring) are often provided within a container today.