At a client site we came upon a problem with Windows 2003 VM’s. They would get low network performance when we moved them to a newly formed ESX cluster consisting of HP 460c G6 blades. In some cases logging on to the server with a remote session took about 20 minutes.

As I mentioned this only occurred when we moved a VM to the new cluster, but also VM’s that where newly installed would get the same problem when running on the new cluster. As we are using Altiris to install and configure new VM’s a colleague decided to install a new VM by going through the steps manually which normally would be done by Altiris and found out that after the activation of a security template the performance dropped significantly.

(more…)

If you haven’t upgraded to VMware vSphere 4 by now, you should consider it and rethink your strategy. VMware has removed all but the most recent versions of their Virtual Infrastructure product binaries from their download page on June 17th. As of May 2010, the following Virtual infrastructure products have all reached end of general support according to the published support policy:

• ESX 3.5 versions 3.5 GA, Update 1, Update 2, Update 3, Update 4
• ESX 3.0 versions 3.0 GA, 3.01, 3.02, 3.03
• ESX 2.x versions 2.5.0 GA, 2.5.1, 2.5.2, 2.1.3, 2.5.3, 2.1.2, 2.5.4
• Virtual Center 2.5 GA, Update 1, Update 2, Update 3, Update 4, Update 5
• Virtual Center 2.0

(more…)

When we are selling, designing and/or building a virtual infrastructure, we brag about consolidation ratios we can accomplish.

But what’s a consolidation ratio of 8 to 1 or 10 to 1? In this case a picture says more than a thousand words ……..

On the right a part of the servers we collected during a recent VMware implementation.

28 servers, 4 more are placed on the left (out of sight), 7 are stored on an alternate location and 7 will be decommissioned when the WAN issues are solved.

A grand total of 46 servers which do NOT need power, cooling, support contracts, space, etc.

(more…)

VMware has given out a VMware Alert for customers who are using VMware View in combination with PCoIP and ESX 4.0 to NOT update with ESX Update 2 yet.

After upgrading ESX 4.0 hosts with Update 2 VMware View clients are unable to connect to the virtual desktops. VMware is currently investigating a workaround for customers that have installed ESX 4.0 Update 2 and are affected see for more information and update the following KB Article: KB Link

This Knowledge Base article will be updated as new information becomes available. If you have been affected by this, please read the KB.

Symptoms are:

• Connections to virtual desktops using PCoIP fail
• You are no longer able to connect to virtual desktops using PCoIP
• You have recently updated VMware Tools

For example, you have installed ESX 4.0 Update 2, which updates VMware Tools, and you are no longer able to connect to virtual desktops using PCoIP.

We recently received an update about the new stuff that is coming from Dell EqualLogic. Because a lot the information was under NDA so I can’t go into details or give you release dates.

What’s here already?

Last year Dell released the 10GB connections for the PS series. This 10GB connection speeds are mostly done because the marketing asked for it, because the customers asked for it. There are rarely used cases where the 10GB connections make sense but in 95% of all cases it doesn’t give you extra performance.

The bottleneck is often not the connection speed but the spindle speeds limits of the disk themselves. Because of the adaptive load balancing of the PS series the 1GB connections are used very efficiently. When used with the MPIO drivers from Dell EqualLogic you can gain a lot extra performance from the network connections.

So what is coming this year:

(more…)

A couple of weeks ago I had a problem where the connection between a snapshot and its parent disk was gone. The parent was a raw device mapping pointing to a LUN on a SAN. This all happened during a migration from one type of SAN to another. For the migration we had to remove the Raw Device Mappings from a virtual machine in order to move it with Storage VMotion. After the move the original RDM had to be re-added to the virtual machine. The final step was to copy the original data from the RDM to a new disk.

One of the steps in our migration plan was to make sure that there weren’t any snapshots on the virtual machine. This procedure was used during the whole migration. One of my colleagues and I were planned for the last batch of servers.

Everything went OK during the migration of this server. All steps were executed without errors. The server started like it always starts. Even the first check by the administrators looked good.

(more…)

Dell EqualLogic has a integration kit for integrating their storage solution with VMware and various Mirosoft products. In this Host Integration Kit you find AutoSnapshotManager/VMware Edition and the AutoSnapshotManager/Microsoft Edition.

While I was searching for information about when to use which edition of AutoSnapshotManager for EqualLogic I came across the website of Marc Malotke which I think is worth sharing.  If I did my googling right Marc Malotke works for Dell as a Enterprise Solution Specialist.

He had some very interesting posts and news items at his blog:

• EqualLogic: SCOM Management Pack Release
• Equallogic:  ASM editions
• Equallogic: Planning your IP addresses

Blog: http://www.marcmalotke.net

twitter: @MarcDELL

During the last month I have been very busy building a new infrastructure at a client site. I’m responsible for the overall technical solution and the basis, a VMware vSphere infrastructure build on five Dell PowerEdge R805′s, Dell EqualLogic PS5000 and 6000 storage and Cisco switches for LAN, DMZ and IP storage networking.

Just before the customer initiated their functional test period we discovered that the overall Windows network  performance was slow. We did several test like copying an 8 GB file from local vmdk to local vmdk and VM to VM and found that the storage performance was no issue but the network performance was very slow.

In the last few years that I have been working with virtualization I have always been a fan of a static network configuration. Meaning, when I configure ESX networking I like my network interfaces and physical switch ports to be configured at 1000MB full duplex if the switch/network interface combination allows it. The idea is that if you purchase gigabit network interfaces and switches you know the maximum speeds. So you configure it to run at it’s maximum capacity, eliminating overhead and using as much bandwidth as possible purely for data transfer.

So when we experienced slow network performance I had a colleague check the Cisco LAN switches for errors, drops, packet loss or any other flaw which might indicate a speed or duplex mismatch. None were found so I assumed that the network configuration was not the issue. But as we know by now, ‘Assumption is the mother of all fuck-ups!‘.

(more…)

Up until a week ago, I hadn’t had the pleasure to work a lot with vSphere. Now with a test environment at the customer site I was able to freely play a bit with vSphere. One of the features I was curious on trying was Fault Tolerance (FT). I just created a random VM and enabled FT on it which all went fine.

After FT was enabled on the VM we wanted to see how we could upgrade the ESX hosts in that cluster as the documentation states that FT only works on hosts with the same build number.

(more…)

While I was updating the Determining VMware vCenter and ESX Build Numbers post I thought I would semi-automate the updating of the post with new build numbers for new releases, while working on it I started too fill an excel sheet and ended up with an excel sheet with the following products and build numbers for easy reference:

VMware vSphere 4 Suite (combines several products)
VMware Infrastructure 3 Suite (combines several products)
Cisco Nexus v1000 (1.0)
VMware ESXi (4.0, 3.5, 3.0)
VMware ESX  (4.0, 3.5, 3.0, 2.5, 2.0, 1.5, 1.0)
VMware Server (2.0, 1.0)
VMware vCenter Server (4.0, 2.5, 2.0)
VMware vCenter Converter Standalone (4.0, 3.0)
VMware vCenter Lab Manager (4.0, 3.0, 2.5, 2.4)
VMware vCenter Site Recovery Manager (4.0, 1.0)
VMware vCenter Server Heartbeat (5.5)
VMware vCenter Lifecycle Manager (1.0)
VMware vCenter CapacityIQ (1.0)
VMware vCenter AppSpeed (1.0)
VMware vCenter Chargeback (1.0)
(more…)

Last week I have been struggling with  the installation of a vSphere 4 infrastructure on Dell hardware at a Belgium client site.

I have done many many many VMware installations and encountered my fair share of issues but apart from the HP USB sticks the hardware never gave me this much trouble.

It all started with a very difficult BIOS/firmware upgrade which, after various downloads and trials, ended with an old-school DOS boot USB and a DOS based BIOS update. Real 1980′s stuff.

With this fixed I installed all ESX hosts and left for the hotel, ready to start the configuration the next day. However, when I started with the first ESX host and wanted to configure the network, I noticed that I only had eight NICs when I should have had twelve. We use Dell PowerEdge R805 servers with two Intel quad port 82576 Gigabit Ethernet Adapters, the first card was already in the server, the second card we added just before the installation.

(more…)

When it comes to storage, the EqualLogic PS-series amazes me every time.

Last Tuesday I was at a client site investigating performance issues but when I opened SANHQ and the web interface on the PS 6000E one thing stood out. The performance issues were not caused by the storage device. The PS 6000E is hosting 12 virtualized Citrix XenApp servers en 8 supporting VMs and during heavy load the average latency is 1,2ms with a read/write ratio of 28%/72% and a max of 1100 IOPS.

But yesterday we had to swap SANs, a PS 5000XV for a PS 6000E and the ease with which this can be done amazes me every time. The hardest part was the process of endless firmware updates on the PS 5000XV to get it to a respectable level to get it to join the PS 6000 in the same storage group.

We had to upgrade from firmware version 3.3.0 to 4.3.4 but this is no simple task because you cannot directly apply 4.3.4. First we had to upgrade to 4.0.7, then to 4.1.7, to 4.3.4. Every time uploading the firmware using FTP, updating both controllers, restarting. This took us almost 2 hours to complete.

When this was done, all that was left was to join the PS 6000 in the existing storage group and evict the PS 5000 and this is the process that amazes me every time.

(more…)

About two weeks ago I attended a session at the VMware User Group meeting here in the Netherlands about Project VRC.  After the presentation I asked myself: ‘What is the value of this project?‘.

For you who don’t know what Project VRC is:

“Project Virtual Reality Check (VRC) is a joint venture of Log•in Consultants and PQR, who have researched the optimal configuration for the different available hypervisors (hardware virtualization layers). The project arises from the growing demand for a founded advice on how to virtualise Terminal Server and Virtual Desktop (VDI) workloads. Through a number of researches, Log•in Consultants and PQR show you the scaling possibilities for Terminal Server environments as well as Virtual Desktops.” http://www.virtualrealitycheck.net/

Don’t get me wrong: What they did was a very good initiative, it showed the performance differences between different hypervisors. Although the results were not that surprising it was good to see the validation numbers of the things we already knew.

I also think that the guys who did the project where totally surprised by the attention vendors and customers gave to the project. It was an outstanding (marketing) tool to show the value of virtualization and especially XenApp on a hypervisor. Because of this attention the whole project got out of hand. Although this was not the goal of the project, vendors and customers used it as a reference guide for vitalizing XenApp. That’s the point where I started to wonder what the real value of the project VRC was.

(more…)

Based on the real life results when virtualizing XenApp I thought it was about time to summarize some of the best practices for virtualizing XenApp servers.

Why we DO want to virtualize XenApp?

1. For server consolidation:  vSphere enables scale up XenApp deployments;
2. For mixing server editions: 32-bit and 64-bit XenApp VMs can coexist;
3. For management: Better management through flexibility & isolation think about Change Management and VMware DRS;
4. For high availability and disaster recovery: VMware HA and vCenter Site Recovery Manager;
5. For less costs for server hardware, maintenance contracts, power, cooling, floor and rackspace.

Virtualizing XenApp servers is very complex. There are a lot more layers involved, like the type of hardware, the capabilities of the processor, the performance of the shared storage, the hypervisor used, the specific settings per hypervisor, operating system settings in a virtual environment, the XenApp settings in a virtual environment, the Workspace management settings in a virtual environment etc, etc.

In the following sections I tried to summarize some of the best practices we use in our projects:

(more…)

I have just updated the list after receiving word the new version is out and the build numbers are known for vSphere 4 and vCenter Server. So I compiled a list of build numbers and updates from the patch notes for VMware ESX and VMware vCenter Server.

vCenter Server 4.0 Update 1 | 19 Nov 2009 | Build 208156

vCenter Server 4.0 | 05 May 2009 | Build 162902

VirtualCenter 2.5 Server Update 5 | 10 Jul 2009 | Build 174791 (English version)

VirtualCenter 2.5 Server Update 4 | 23 Feb 2009 | Build 147633 (English version)

VirtualCenter 2.5 Server Update 3 | 03 Oct 2008 | Build 119598 (English version)

VirtualCenter 2.5 Server Update 2 | 25 July 2008 | Build 104217 (English version)

VirtualCenter 2.5 Server Update 1 | 10 Apr 2008 | Build 84767 (English version)

VirtualCenter 2.5 Server | 10 Dec 2007 | Build 64201

(more…)

At the end of May of this year we wrote a article concerning Hypervisor comparisons and we got a lot of positive feedback on it. The downside to that is that people want an update as soon as one of the companies launches a new version of its product, and who can blame them. However the issue is that this takes a lot of research and because of that, a lot of time. And because two of us are ill and in bed wearing a sombrero and the other two are extremely busy, we simply don’t have that time right now.

However, Scott Lowe has written an excellent article on the feature comparison between VMware vSphere 4 and Microsoft’s Hyper-V R2 which is a must read for everybody who’s advising customers on hypervisors.

It’s not as extensive as the Enterprise hypervisor comparison we did earlier but it gives you a good image how both products relate to each other. To extend the picture I added a list of supported operating systems.

(more…)

Last night VMware has released Update 1 for ESXi 4, ESX 4, and vCenter server 4.

According to the release notes Update 1 includes the following improvements:

• Support for Windows 7 and Windows Server 2008 R2;
• Support for View 4;
• Support for 160 VMs per host in a HA cluster with 8 hosts or less;
• Increased vCPUs per core limit from 20 to 25;
• Paravirtualized SCSI support has been extended to Windows 2003 and 2008 boot drives;
• vDS performance improvements;
• Support for DB2 database;
• Improved support for Microsoft Clustering.

Besides this vCenter server now includes a pre-upgrade checker tool which enables you to proactively check ESX hosts for any potential issues that you might encounter while upgrading vCenter agents on these hosts as part of the vCenter Server upgrade process. You can run this tool independently prior to upgrading an existing vCenter Server instance. The tool can help identify any configuration, networking, disk space or other ESX host-related issues that could prevent ESX hosts from being managed by vCenter Server after a successful vCenter Server upgrade.

The release note can be found here.

vSphere 4 Update 1 can be downloaded here.

Since last week I’ve got a new assignment which is a long way from home. Next to the downsides of traveling 5 hours a day and staying in hotels, it certainly has advantages.

During my 5 hour drive or when I’m staying in a hotel i have time to listen to the VMware Community Roundtable podcasts I put on my iPhone. For those of you who don’t know this. VMware Community Roundtable is a weekly conference call/chat for VMware enthusiasts. Every week there’s a different topic and this is your change to discuss these topics with the experts in the community. For those of you who don’t have time to attend, there’s the possibility to listen online or download the sessions as podcasts.

Last week I listened to a great topic which was called the ‘#66 – iSCSI Super Friends with EMC, NetApp, Dell, HP, VMware‘. This podcast was from September 30 so it was quite an old podcast but this was such a great session that I don’t want to keep it from you. This session discusses the multi vendor blog post by VMware (Andy Banta), EMC (Chad Sakac), NetApp (Vaughn Stewart), Dell/EqualLogic( Eric Schott), HP/Lefthand Networks (Adam Carter) on how to implement iSCSI with VMware VI3.5 and vSphere 4.

So for those of you who already heard it, sorry for the ‘old’ intel, for those of you who haven’t, this is a must read/hear.

The mult vendor blog post discusses can be found here:

• Using iSCSI with VMware VI 3.x
• Using iSCSI with VMware vSphere 4
  

I really liked the conclusion that you can build iSCSI storage solutions which are as fast as fiber storage solutions and that tier 1, tier 2 and tier 3 storage definitions should not be based on the technique used but on the service level agreement with your end user.

Personally I wonder what the future will bring. Will fiber channel disappear in favor of iSCSI? As discussed during this podcast, ethernet is ideal for storage connections, the questions is which protocol to use, NFS, iSCSI or FCoE. We will see ……

This week a colleague called me with a VMware ESX problem. He’s doing a Exchange 2007 implementation and he configured a virtual machine which should function as the Mailbox server. He was complaining that the virtual machine took 13 to 15 minutes just to boot and he suspected a storage issue as they had been struggling with the storage for a few days.

I asked him to check the virtual machine performance tab and asked him if he saw anything strange. Nothing! The server had plenty of memory left and was not using very much CPU.

When I asked him how many resources he assigned to the virtual machine his response alerted me. He configured the virtual machine with 16GB of memory and 4 vCPUs!

When I asked him why he gave the virtual machine so much resources his response indicated that he hadn’t read the Virtual Infrastructure best practices and applied the ‘MS, I might use those resources just once during my lifecycle‘ sizing. When we checked the%CPU ready performance statistics the values were astronomical.

So I explained the behavior of the CPU scheduler to him and asked him to reduce the number of vCPUs to 1 or 2 and try again. When I called him the next day he told me that the virtual machine now booted within a minute.

So bottom line: Give a virtual machine the resources it needs for normal every day operation and be very modest. Monitor the virtual machine and adjust the resource settings according to your findings. In this case, less definitely is more and it’s very easy, especially in vSphere 4, to (hot) add resources. And if you do need 4vCPUs, it can be more effective to deploy two 2vCPU virtual machines instead of one 4 vCPU virtual machine.