Memory protection

Latest modified: March 4, 2001
7. Operating systems security
As the operating system is the bottom layer of software supporting all the services used by applications its security is fundamental. A recent paper makes a case for the importance of the OS in a secure system [Los00].
Systems architecture
Most OSs are built using the multilayer pattern [Bus96]. The lowest level layer is the kernel, which normally includes process and memory management.
A security kernel overlaps most of the functions of the OS kernel but includes only security-related functions:
•Process creation, destruction, and domain switching
•Memory protection
•I/O functions
A security kernel is a combination of hardware and software that implements the concept of reference monitor [Ame83].
Utilities are usually the weakest parts of the OS. Most of the reported attacks happen there.
A good architectural concept to implement isolation is the use of virtual machines [Sum97]. Each VM runs in a virtual copy of the hardware and is separated from other VMs by the hardware. In this way each VM could run a different OS. A Virtual Machine Monitor(VMM) acts as kernel for the complete system and is the only one with access to the real hardware. Privileged instructions in the VMs are intercepted by the VMM, which interprets them according to the OS running in the corresponding VM.
User and system authentication
This function is normally part of the OS. Typically it is based on passwords, which are considered a relatively weak approach (See Pfleeger, Section 6.5, and [Sta99], Chapter 15). More secure approaches use biometrics (fingerprints, retina scan, face contour, face recognition), which is also a more expensive approach. Authentication between network nodes must use some protocol as discussed in the chapter on network security.
Process protection
This is performed normally in collaboration with the hardware and we discussed it in Chapter 6.
Memory protection
See Pfleeger, Section 6.
Access to objects or resources
See Pfleeger, Section 6.3
In AIX and Windows NT resources are associated with objects. They can have any relevant access type.
In standard Unix resources are reached through programs, embodied in executable files. The access types are only read/write/execute.
File protection
See Pfleeger, Section 6.4
There are two basic approaches:
•Map files to a single-level virtual address space (VAS). This has been used in Multics and IBM’s S/6000. It is the most elegant and secure approach because it requires only one mechanism to protect memory and files. However, it requires a very large address space, not easy to implement.
•Have a special file authorization system. This is done in most commercial OSs.
Unix implements a reduced form of the access matrix. It also allows only three types of access to files: read, write, and execute (these are interpreted differently for directories). Windows has a more elaborate access control structure and allows more types of access to files.
Unix files use descriptors, called inodes, to hold access permissions. The inode for a file is brought into memory when the file is opened. A PATH environment variable defines the order in which a requested file is found.
Unix uses a mixture of user and role to assign rights. Some system functions are special roles with their own rights, e.g., root or superuser, daemon, agent, gust, ftp, uucp [Gar96]. Audit
Audit is needed to verify that an attack happened, study its effect, and maybe prosecute somebody. See [Sum97].
Weaknesses of most commercial OSs
There are many vulnerabilities in commercial systems that have been exploited by hackers in most Internet attacks. Operating systems are very complex systems and most of the times they are built in an ad hoc manner, without using systematic secure software development processes; this makes the occurrence of flaws a likely event.
There are several products that look for vulnerabilities but they find only the most common ones. In a recent study of several of these tools the best one caught 14 out of 18 common vulnerabilities [For01].
A comparison of Unix and Windows security is given in [Vie00]. Their conclusion is that neither is clearly superior in this respect. Their study is rather superficial, however. An analysis of the security of Windows NT is in [Mud97], who consider it rather poor, and in [Hya], who considers ot a good improvement over “other systems”.
Some specific weaknesses:
Both Unix and Windows use passwords for authentication. Unix keeps passwords encrypted but the password file is readable by all users. This allows a user to make a copy and use dictionaries and parallel processing to guess passwords. The use of a “salt” field to encrypt passwords improves security [Fil86]. The file used to store passwords
(etc/password) includes also user information, readable through the finger command. This information is useful to hackers to make a better guess of passwords.
In these systems process protection is based mainly on the user/supervisor mode separation and kernel processes are not protected against each other. Even if hardware architectures offer further protection, e.g., descriptors and rings, commercial OSs do not use them in an effort to get more performance.
The concept of superuser, an almighty user, typically the systems administrator, is a poor security decision. When this account is compromised the hacker can reach anything in the system.
Inheritance of rights in forked processes is another flaw commonly exploited in attacks. If an attacker tricks a program in superuser mode to execute a Trojan Horse, this inherits the rights of that program and runs in superuser mode.
Transfer of rights between processes—In Unix every user has a unique id, UID. If a bit in a file permission (setuid) for a file containing an executable program is turned on, the program executing that program acquires the rights of the file owner. Windows has an impersonation token, that has a similar effect. This violates the principle of accountability.
Lack of conceptual model. The file permission structure doesn’t follow the access matrix or any other security model. The interpretation of rights for directories makes things even more muddled; for example, a user with search and write access to a directory can delete all the files in a directory without having direct rights on any of them.
Directory problems. An attacker can place his own file in the path of a writable directory and maybe get higher privileges when the file is invoked.
Most systems lack the concept of a trusted path [Los00]. A trusted path is a user connection to a part of the system that provides secure login, authentication, and rights.
Some systems do not have auditing facilities or the audit log is within reach of the superuser (and could be changed by a hacker acting as a superuser).
These systems have complex, poorly designed, and poorly tested utilities. Microsoft’s Outlook is a Swiss cheese. The Sendmail program in Unix is another source of trouble. Some flaws come from implementation languages, e.g., buffer overflow. Buffer overflow occurs when a variable in a procedure is filled with more values that it can hold.
The overflow can overwrite the return address and if the hacker put her code there her program could get superuser mode [Dil]. Correction requires bounds check and this takes time, which explains the reluctance of vendors to correct this problem. In [Vie00] the authors say that this is a C problem (no check for buffer boundaries) and not the fault of the OS. This is true, however, there is no reason for OS vendors not to use a better language or to modify the run-time system to perform these checks; Microsoft changed the Virtual Machine of Java to suit marketing purposes, why not change C? This problem was first reported in 1968.
Finally, configuration of these systems is complex and administrators make many mistakes. There are many demo programs and rarely used utilities which can be exploited by hackers. This is even more true for PCs where the users usually have no idea what they get in their software packages.
In summary, most systems have a variety of security flaws. Some are just implementation problems and are easily correctable, others are design flaws and much harder to improve. How an OS is attacked
Preparation stage—During this stage the attacker tries to find out what operating system a site is running, the hardware platform, the services offered [Hen00]. There are several tools available to help in this task.
Specific cases
A famous penetration of multiple Unix systems occurred in the San Francisco Bay area in 1986 [Rei87]. A computer at Stanford University used as a mail gateway had a guest account with a user id and a password of “guest”. This system had the flaw that a user having write permission to a system directory could store his own system command in the directory. When some system utility running as a superuser executed this command the special system command took over and ran with superuser rights. From there on the program used remote logins to get access to other systems and repeat the procedure.
The Internet Worm of 1988 used the following attacks (See Pfleeger, pp. 192-195 and Summers 95-97):
•Guessed and tried passwords with the use of a dictionary.
•Exploited bugs in the finger and sendmail programs, including buffer overflow.•Took advantage of the fact that some systems specify trusted computers from which remote logins are accepted without checking passwords.
•Used encryption and other ways to hide its existence (a polymorphic virus).
These are old examples, what is surprising is that recent attacks have used very much the same approaches.
Denial of service in an OS
This is a problem produced by a user or users hogging system resources. None of the known models for security can address this problem. Most OSs put limits on the number of threads a user may have and on the number of files that can be opened but most
systems do not count how many files a given user has created [Fil86,Mud97]. A bad-intentioned user can cause considerable trouble before being caught.
Later we’ll discuss the distributed form of this problem.
Secure (trusted) OSs
In the 1970’s there was considerable work on trusted OSs. These implemented multilevel models and their commercial versions were not successful. There is now renewed interest on them and several commercial versions have appeared that are more flexible than pure multilevel systems [Mil01].
Multilevel secure systems
A multilevel system can, by application of the *-property, control a Trojan horse trying to leak information out of the system. A text editor trying to make a copy of a user’s file in a lower level is stopped by this property [Ame83]. However, their policy is too restrictive for Internet applications and nobody seems to be using them for these purposes. Hardened versions of commercial systems
These try to improve security by reducing the known exposures of standard OSs.
For example: Administrative privileges can be reduced or divided into several groups or roles. Inheritance of rights in forked processes can be eliminated or reduced. Of the commercial systems a few have reached the E3/F-C2 classification: AIX 4.3 and Windows NT Server and Workstation 4.0 (maybe others). These are classifications of the Orange Book (See Pfleeger, 313-315). To reach a C classification the system must implement an access matrix DAC model.Several systems have extended or are extending their authorization systems to comply, e.g. Linux [Zav99], and some of the systems described below.
Windows NT [Hya]—As indicated earlier it uses objects as units of resource access. An object includes a security descriptor that contains the Owner SID, group SID, DACL (discretionary access), and SACL (used for auditing). Windows NT uses four levels of administrative privileges: standard, administrator, guest, and operator.
IBM’s AIX [Cam90]—It implements a TCB to support DAC (initially they intended to support also multilevel security). Some programs are installed with a high privilege and considered as trusted programs. Instead of read/write/execute rights AIX defines an Abstract Data Type (class), with higher-level operations, appropriate for the type of object such as copy, save, query, and set. These accesses define an access matrix implemented as Access Control Lists. The ACLs are set by the owners of files and by administrators. ACLs can be permissive or restrictive. AIX reduces the privileges of the system administrator by defining five partially-ordered roles
.
Virtual Vault [HP, Rub94]—This is a secure platform for Internet applications that includes a trusted HP-UX operating system (A Unix variant), and firewalls and other protection devices. It is part of a secure server system, the HP Praesidium family of products. It uses compartments based on the multilevel model to isolate portions of the OS. It also reduces the root privileges and controls inheritance of rights in forked threads.
Argus Pitbull [Arg]—This is a system based on:
•Compartmentalization based on a multilevel MAC model. It separates users and data by classifications and compartments.
•Least privilege applied to all processes, including superuser. The old superuser is implemented using three roles: Systems Security Officer, System Administrator, and System Operator.
•Kernel-level enforcement.
The system is applied as a layer on top of another OS, e.g., Solaris and AIX.
Trusted Linux [Dal01]—The NSA is sponsoring the development of a security-enhanced Linux [NSA], that uses a multilevel model for its kernel.
Other ideas
Some experimental systems have used reflection to provide adaptability and security [Son94]. This is an idea that you will see more in the future.
References
[Ame83] S.R.Ames, M. Gasser, and R.R.Schell, “Security kernel design and implementation: An introduction”, Computer, July 1983, 14-22.
[Arg] Argus Systems Group, “Trusted OS security: Principles and practice”,
/products/white_paper/pitbull
[Cam90] N.A.Camillone , D.H.Steves, and K.C.Witte, “AIX operating system: A trustworthy computing system”, in IBM RISC System/6000 Technology, SA23-2619, IBM Corp., 1990, 168-172.
[Dal01] C. Dalton and T.H. Choo, “An operating system approach to securing services”, Comm. of the ACM, vol. 44, No. 2, February 2001, 58-64.
[Dil] DilDog, “The Tao of Windows buffer overflow”, Cult of the Dead Cow,
/cDc_files/cDc-351/page1.html
[Fil86] A. Filipski and J. Hanko, “Making Unix secure”, Byte, April 1986, 113-128. [For01] J.Forristal and G.Shipley, “Vulnerability assessment scanners”, Network Computing, January 2001, 51-65,
[Gar96] S. Garfinkel and G. Spafford, Practical Unix and Internet security (2nd Edition), O’Reilly and Assocs., 1996.
[Hen00] S. Henry-Stocker, “OS identification”, Unix Insider, [HP] Hewlett Packard Corp., Virtual Vault, /security/products/virtualvault
[Hya] G. Hyatt, “Windows NT: How good is the security of Microsoft’s newest operating system? Comp. Sec. Journal, Vol. 9, No 2.
[Los00] P.A.Loscocco et al., “The inevitability of failure: The flawed assumption of security in moden computing environments”, NSA, /selinux/inevit-abs.html
[Mil01] ler, “The trusted OS makes a comeback”, Computer, February 2001, 16-19.
[Mud97] P. Mudge and Y. Benjamin, “Déjà vu, all over again”, Byte, November 1997, 81-86.
[NSA] National Security Agency, “Security-enhanced Linux”,
/selinux/index.htm
[Rei87] B. Reid, “Reflections on some recent widespread computer break-ins”, Comm of the ACM, vol. 30, No 2, February 1987, 103-105.
[Rub94] C. Rubin and D. Arnovitz, “Virtual Vault white paper”, Hewlett Packard, 1994-1999. /security/products/virtualvault/papers/whitepaper_3.1/
[Son94] S. Sonntag et al., “Adaptability using reflection”, Procs. of the 27th Annual Hawaii Int. Conf. On System Sciences, 1994, 363-392.
[Sta99] W. Stallings, Cryptography and network security: Principles and practice (2nd Edition), Prentice-Hall, 1999.
[Sum97] R.C.Summers, Secure Computing: Threats and Safeguards, McGraw-Hill, 1997.
[Vie00] J.Viega and J.Voas, “The pros and cons of Unix and Windows security policies”, IT Professional, Sept./October 2000, 40-45.
[Zav99] V. Zavadsky, “Linux Trustees (ACL) project”,
/linux/trustees.html。