Thursday, November 5, 2009

Restrict Logins with Likewise on Linux/Unix

I'm a pretty big fan of Likewise. I am currently using the likewise open version on many of my linux boxes so that I can authenticate against Active Directory. This makes my life easier in the fact that I don't have to deal with adding and removing users from these boxes.

Our standard build consists of a build that has sudo installed. Root cannot login over SSH and we have a default local account that the sysadmins know and can use if in a pinch. We do, however, use our own accounts so it can leave a sudo log trail.

Now, when we installed likewise-open on our default ubuntu image, we had one concern, any user with an Active Directory account can now login to our Ubuntu servers. While they will just be a dummy user with minimal privileges, what is to stop these users from using an exploit to escalate their privileges to root, or filling up their home directory and owning some of the boxes that don't keep those partitions separate. While I don't think any of my users would do anything like that, its always better to allow minimal privileges and lets face it, my sales, accounting, managers, and many others have absolutely no business in ever logging into any of the production linux/unix servers.

I did some strategic Googling query building and I was able to stumble on a thread (sorry I lost the original link) that pointed me to a specific line in a configuration file (/etc/security/pam_lwidentity.conf). By uncommenting this line and putting a group in there, I can restrict who can login to my servers. So I set it to the following and simply created a security group called 'unix admins' in Active Directory to hold the people who can admin these boxes. Piece of cake.

require_membership_of = GCA\unix^admins

Wednesday, November 4, 2009

Linux Kernel Vulnerability

Found the following article on slashdot and immediately jumped over to my linux boxes to ensure we aren't vulnerable.

http://linux.slashdot.org/story/09/11/04/0320254/Bug-In-Most-Linuxes-Can-Give-Untrusted-Users-Root?from=rss

More information can be found at securityfocus here:

http://www.securityfocus.com/bid/26831/info

From what I understand, you can fix this vulnerability by simply setting the vm.mmap_min_addr setting to a value of 4096 or greater ( in short, if this value is 0, you are vulnerable). I also read that if you install wine (winehq.com) that it will automatically set this value to 0 and make you vulnerable.

To check the value of this variable on your linux system to see if you are currently vulnerable, just run this command and ensure it is not set to a value of 0:

sysctl vm.mmap_min_addr

On debian based installations, check the following site for how to fix this for the current boot and how to make it persist a reboot.

http://wiki.debian.org/mmap_min_addr

For those that are really concerned right now, I checked my Ubuntu 8.04.3 LTS servers and they all are defaulted to 65535 for this value, so they should not be vulnerable to this bug. Keep in mind what I stated above though, if you install wine, you become vulnerable.

Tuesday, November 3, 2009

VMWare ESX Troubles

First a bit of background. We run some VDI (Virtual Desktop Infrastructure) here at GCA. We have our ESX 3.5 servers running and our Netapp SAN hooked up to it via NFS. On the ESX servers, along with our production 'server' vms, we also have a bunch of VDI vms. The VDI vms are just virtual machines running Windows XP for individual users to use. They access these vms using a few tools, such as Sun's Secure Global Desktop and Sunray Ultrathin clients. They can also be accessed with a regular RDP tool.

Let me also preface with the fact that we have nightly snapshots of the VM's and the developers check their code into subversion on a regular basis, so worst case scenario is a 20 min operation to copy the VM from last nights backup in the event one explodes.

Today I had an engineer who tried to reboot her Windows XP VDI and it would not reboot. I asked her if she had any open applications, to which she replied no, so I used VirtualCenter to send the VM a reboot command. That did not work, so I gave it a good amount of time, then got impatient and took the shotgun approach and tried the reset vm command instead. This still did not work, so I tried it again and I get an error that there is another operation in progress.

I did some googling and found information on the nuclear approach to the situation. First thing is to find the vm world id with the following command:

cat /proc/vmware/vm/*/names grep "vmname"

The result should start with vmid=####. With this number, you can browse over to /proc/vmware/vm/####/cpu and do:

less -S /proc/vmware/vm/###/cpu/status

Well, I had a problem. The cpu directory in this location was not present. I could not get the VM to stop at all. I also tried the other method of using ps -aux and grepping for the VM name, but it did not have a process either. I did not find a solution for this, but here is what I found in /var/log/vmkwarning log file. Look at the timestamps, it figured it out eventually and allowed me to do operations it, but it took well over an hour. See below for log output and please let me know if you know how I can fix this without waiting 90 minutes in the future:

Nov 3 14:54:50 gcaesx001 vmkernel: 40:00:33:25.587 cpu1:1042)WARNING: Swap: vm 1390: 7515: Swap sync read failed: status=195887193, retrying...
Nov 3 14:54:50 gcaesx001 vmkernel: 40:00:33:25.587 cpu0:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=1
Nov 3 14:54:50 gcaesx001 vmkernel: 40:00:33:25.638 cpu2:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=2
Nov 3 14:54:51 gcaesx001 vmkernel: 40:00:33:25.939 cpu2:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=4
Nov 3 14:54:54 gcaesx001 vmkernel: 40:00:33:29.144 cpu1:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=8
Nov 3 14:55:02 gcaesx001 vmkernel: 40:00:33:37.147 cpu2:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=16
Nov 3 14:55:18 gcaesx001 vmkernel: 40:00:33:53.155 cpu0:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=32
Nov 3 14:55:50 gcaesx001 vmkernel: 40:00:34:25.175 cpu3:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=64
Nov 3 14:56:54 gcaesx001 vmkernel: 40:00:35:29.224 cpu0:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=128
Nov 3 14:59:02 gcaesx001 vmkernel: 40:00:37:37.300 cpu0:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=256
Nov 3 15:03:18 gcaesx001 vmkernel: 40:00:41:53.473 cpu1:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=512
Nov 3 15:11:51 gcaesx001 vmkernel: 40:00:50:25.865 cpu2:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=1024
Nov 3 15:28:55 gcaesx001 vmkernel: 40:01:07:30.612 cpu3:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=2048
Nov 3 16:03:05 gcaesx001 vmkernel: 40:01:41:40.116 cpu1:1042)WARNING: Swap: vm 1390: 7528: Swap sync read retry failed: status=195887193, retry=4096
Nov 3 16:18:12 gcaesx001 vmkernel: 40:01:56:47.424 cpu3:1042)WARNING: Swap: vm 1390: 7550: Read failed: numPages = 1
Nov 3 16:18:12 gcaesx001 vmkernel: 40:01:56:47.424 cpu3:1042)WARNING: Alloc: vm 1390: 5226: unable to read from slot(0x102ac7c)
Nov 3 16:18:12 gcaesx001 vmkernel: 40:01:56:47.424 cpu3:1042)WARNING: World: vm 1390: 6870: vmm0:vdi-rlosaria:vmk: vcpu-0:Unable to read swapped out PPN(0xa1ef) from swap slot(0x102ac7c) for VM(1390)
Nov 3 16:18:12 gcaesx001 vmkernel: 40:01:56:47.426 cpu3:1042)WARNING: P2MCache: vm 1390: 450: Alloc_GetPhysMemRange failed for PPN 0xa1ef status = Permission denied
Nov 3 16:18:12 gcaesx001 vmkernel: 40:01:56:47.430 cpu2:1037)WARNING: World: vm 1390: 5160: Panic'd VMM world being reaped, but no core dumped.