This Article is about the tool called Hyper-V Performance Monitor Tool (PowerShell)
you can download it from the tech net article down the page, or use the link above.
Over the last few years, we have made rapid changes from Physical Host Machines for production work loads, to these Virtual Environments, that now host our whole company.
Along with this change, you may recall that Early Hyper-V documentation has gently let us know that monitoring inside the Virtual Machine was not going to give us the parity results to the Physical counters, depending on configuration. This is for a few reasons, which are beyond this articles scope. However, I would like to shine light on this, so we can think differently about their Virtual performance.
The most common measure of how well a server is performing is Latency in Milliseconds. Every one is most concerned with how much latency is in the storage system. Perhaps with good reason. The San storage vendors can perform so fast now days, that you can throw the empire state building at a server, and the Latency is less then 10 Milliseconds (MS).
To be clear, we are interested more in Latency then throughput. Latency should be minimized, and throughput will generally increase.
Can I make a case that Counters are not reliable?
Well let me tell your actual Latency is not as easily attained as you would think. If I told you that your tests are lying to you, would you believe me? Lets say your in shock. Without knowing your design, my generalized answer has a very high chance of being correct. The issue is that you have a SAN and you are tying to get latency by measuring responses to a file, that go though 4 different filters, and then has to wait until it gets queued to a disk subsystem, that is always expanding. Its this type of activity that makes a virtual machine difficult to match up to a physical machine.
FYI – latency and throughput for Further reading.
Storage Latency of VM guests
There are many problems with calculating storage latency, but disk is the model we are going to use to illustrate how tricky it can be to find out how your VM is performing.
The most common approach to gaining latency information is to use a command line tool like disk speed. Normally the tool will work fine. The model breaks down when the Disk itself is changing, along with the Ram and processor availability. The bottom line is a Virtual machine may lie to you about resource numbers at any given time. Its not lying, but its changing. This means your only seeing a narrow view, and a calculation made from the narrow snapshot. Add to the mix, that the clock cycle is a weakness in any virtualization platform. That means that the calculation of time itself can result In poor results based on good math with bad numbers.
There are a crowd of you who say that is bull. Well all I can say is; don’t read this and good luck solving your latency issues.
Let me try to list some areas where the numbers may go awry. I am just making a one line explanation with a link, so you can read more. I don’t want this to be about the problems. Below, I talk more about the solution. Read more if you have a specific issue:
- 4K sector drivers and problems with VMs, Sans, Storage spaces etc.…
- Underlying Disk Geometry differences account for significantly different performance numbers (VHD vs. VHDX)
- SQL pros don’t trust some Perf Counters
- Ah someone at Microsoft agrees with me!
- Perfmon Adds 4 Disk Queues to the total latency(example)
- From Windows, Only Storport can really see reliable latency (have you heard that before)
- Now is a VHDX even a Disk?
- The Block size of your Disk may affect storage speed
I could keep going. Do you get the feeling there are a ton of variables that change how storage latency should be calculated?
From my experience, I have found that every set of servers are their own data set of network behavior. There are some basic assumptions I found to pass along to Admins who want to find out latency of Virtual machines.
Guidelines for VM latency Study
- 1. Do not use hardware tools. they don’t calculate the correct latency ever. Not all MS counters are correct either!
- SQLIO,Disk Speed, and others will not work properly if the disk is not fixed.
- 2. For Dynamic Disks, you are limited to counters of the VM itself.
- 3. The Goal of performance tuning is to find the average traffic that works well to a VM
- many Admins run commands that are more reflective of maximum.
- In other words the commands bring the VM to its knees, instead of finding what works, they find the server does not work.
- The answer? make your test a reasonable workload, where the machine handles things well. These are called baselines. They are not maximum commands that kill the server.
- Even using Windows counters, you may fail to collect all data. If windows cant collect all the data, then how well is outside the VM going to capture that data?
- Use Hyper-V counters to calculate storage latency, not external tool calculations
- Disk QUEUE Length calculation
- Example of Lying data. This links shows how data is wrong on the VM itself. Best to follow instructions and just accept what data you know is true.
You cant say its slow if you dont have a baseline
So again, the basic message is that the calculation of latency’s is totally based on the sum of the deployment factors. In one data center you may find under reporting, and the other Over Reporting. Support agents do not have the Onus to prove why one is slower then the other. We will have to look at your Design and deploy and try to make a story of things we can identify. it is not likely we will find that moment where the Deployment deviated from your Baseline storage latency measurements. We offer Best effort, but encourage you to strip down your deployment to make a core Baseline latency for a Dynamically expanding VM. All Vms will compare to that one. We go from there.
Solutions for Monitoring Virtual Machines
SO it was just a few years ago, I remember having a case where I knew the MS counters looked really wrong, but I did not realize they could be wrong. So I had to just Defend what i assumed was right. given that customers just want to run Disk Speed or SQLIO, and get an output to look a, this ends up being more work. but if you want accurate numbers, Please see below. There is a script out there. The link is at the Tech Net Gallery:
Hyper-V Performance Monitor Tool (PowerShell)
Below is the walk through of the basic performance collection.
you Just run the Script from an Admin PowerShell. There are a few ways to run it:
### export data to csv via GUI, defaults to current dir
### retrieve data as PSobjects, great for parsing and logging, -name parameter is optional, defaults to automatic discovery
### specify host and interval/samples manually
.\Monitor-HyperVGuestPerformance.ps1 -Name host1,host2 -PSobjects -Interval 2 -MaxSamples 5
### accepts pipeline input
‘Host1′,’Host2’ | .\Monitor-HyperVGuestPerformance.ps1 -PSobjects
### Log to SQL server with Write-ObjectToSQL , this example uses SQL auth
.\Monitor-HyperVGuestPerformance.ps1 –PSobjects | Write-ObjectToSQL –TableName table –Database db -Server server –credential (get-credential)
If the domain connection fails, it tries for a Local connection:
In my case, I ran the tool on the Host, and this GUI below popped up. All I did was hit monitor, and I got an export vm_perfmon_stats file. This file can be used to find your latency.
While this method may not be pretty, It does follow the rules for Hyper-V guest. The main purpose for this tool would be to use instead of SQLIO or DISK SPEED. tools like these should be used for hardware testing. A Hyper-V Server, running on ISCSI shared storage, with two VHDX files attached, is likely going to come back with Erroneous Latencies. This may not be perfect, but I do believe you will see a consistent result that is not a totally unbelievable number.
See I changed the Sample and interval:
And I get a time-frame to wait for the test results:
Find the Link at the Microsoft TechNet Center. Thank you for taking the time to Read about Storage Performance for Hyper-V virtual Machines
I hope this helps in your Baseline Studies.
The result is a nice little Excel Display of the data, that I cleaned up a little with colors, to the Excel Fields.