Using Powershell and Excel to compare Microsoft Hotfixs on Servers and Virtual Machines.

Hello All,

 

Two Types of Comparisons Devised

I have been working several Clustering and Hyper-V cases Recently.  I have come to a point where I notice the Virtual Machines rarely have the same level of  Microsoft Updates that the Cluster Node does. I finally had a few minutes to look at this issue, and I even made an Excel Spreadsheet to compare updates to other machines.

Fortunately, When I got serious about making a Spread Sheet Script, I noticed there were some pretty good work already available. In fact, I found one that will likely serve to compare updates on all the Nodes, all the VMs and more. I am sure I can do cross checks using a Host as the standard, and a bunch of VM’S as the comparisons.

So we are basically showing off two different ways of doing the same thing.  The Gallery script vs. MyExcelHotfixCompare. These are different things for different reasons. But its good to have them in the same place, Because they both have their own purpose.

 

 

First thing you want to do have is a copy of the spread sheet, MYexcelHotfixCompare , and a copy of the The Gallery script

 

Why Two Comparisons

So I made one up, and I found the other by accident! That is true. However, I discovered these both served different purposes.

The First method came from Stane Monsik. What this count does, is aggregate the KB’s for every server in the survey. So your Final report answers the question “when you updated Server Group X with KB Group Summation(A-Z), did you do that same thing to all the other servers? This is what I started out to have answered! If the answer to this is no, then I always tell my customer to finish his updates right away.

However, another question I was often asking, was not being answered by this script. The Questions was, When we know that there are X updates packages for Technology Y, how many of those updates did you apply to your server? Then Question #2,  For X and Y, did you apply these updates to just the server which had problems? The answer to this question should be that you updated all the servers which contained technology Y, with every Update Package X. I found this question was often met with Guesstimates more than actual proof; this was an easy question to get a good guess on. It’s also an easy question to just accept the guess and move on.  In today’s complex world, that is no longer acceptable. So I set out to easily answer this kind of question.

 

Method1 Script

This script will take “a list of Machines” and show you what updates the entire list has and does not have. Every KB gets added to the list, no matter what server it was found on.

The Script, located at Gallery.Tech.net.Microsoft.Com , Goes to every server and pulls the list of hot fixes. Then it goes through the hot fix list and checks every computer, to see if the hot fix has been applied to the computer.

To make GetHotfixCompare_Gallery work, all you have to do is open the .PS1 file with notepad and add your list of computer names. You can add as many computers as you like, but realize it takes longer and longer and longer to complete. The text you are editing looks like this in the file:  See Figure 1. :

# —————————————————————————————————-

$computers = “ComputerName1”, “ComputerName2”, “ComputerName3”, “ComputerName4”, “ComputerName5”

# —————————————————————————————————-

The Result is a very nice HTML report:

 

Figure 1.

1comparetable

 

Once you populate the script with edits on the names of computers, you run the PS1 file by right Clicking the file and choosing Run With Power Shell. The Script runs (longer the more computers you select), then generates an HTML file of your KB’s. There is an orderly chart of stars telling you which KB’s have stars for the machine, and which do not have stars.

 

All in all, A  very good method. But in order for me to answer the technology related questions, I needed to use something different.

 

Method 2 Spread Sheet comparison (MY_excel_GetHotfixCompare)

On the Excel Help article, called “Compare two lists and highlight matches and differences” I use Example 4 of that section. Example 4 runs a comparison from column one (the update list of a server (get-hotfix)  against column two (the list of updates for the specific technology) .  The comparison Colorizes the two list columns to highlight where the updates match or are unique.

MY_ExcelGetHotfixCompare Is the name of the spread, and you just need to right click and save as to save the file as an XLSX file.

This method will always work, as long as you can find a list of key updates, Microsoft says is relevant. This list below is not unabridged. This is just areas where I work most often. Below are some Example Technology Patch lists, that will power column two, on your spread sheets.

Server 2012 and Exchange make you dig a little bit to get the actual KB list, but you get the Idea. The point is to get the list of “important hotfixes” and be able to use them to “audit” you current situation. So let’s get to the formulas and make a spread sheet!

 

Get Hot-fix Compare Spread Sheet

The Compare column Colorizes numbers in column A , that are also in column B. And Vice Versa. So there is only one caveat to this method. If the KB shows up twice in the same column, then the cell becomes colored, denoting that the patch has a match. But, This is an impossibility of the get-hot fix command. As long as your pasting in get-Hotfix results from only one machine per column, you will get a valid result. So the limitation of this, or any tool reviewing hot-fixes; don’t try to do two machines in one spread.

Steps to create:

  1. Open Excel and put your Values for Get-Hotfix (just the KB) into column 1.
  2. Copy the Hot-fix numbers for the relevant Microsoft KB numbers from your articles like above, into Column 2.
  3. With Excel at the Home Tab, Choose “Conditional Formatting” and Manage Rules . You will see a pop up window, New Formatting rule: (figure Below)

 

2spread

 

  1. Choose New Rule and “Format only unique or duplicate values”
  2. There is a little Drop down that says Duplicate or Unique- You will make that choice here

3spread

 

6.Then you see where it says no format set? (above) This is where you choose your color.

7. Hit the Format button. Then choose the Fill tab.

8. Now see the Fill tab below Choose your color and hit Ok.

3color

 

9 Then Choose OK Below

5

 

10. Now Comes the important part. You simply highlight all values in both columns and there is a wizard that will form the formula for you based on how many rows each column has in it. I am showing the screenshot below, after I clicked into the “applies to field” and held down the left click of the mouse while I selected both columns and went down enough rows to cover all values in both columns. I also found that if you pre-select both columns, before clicking conditional format, then these vales will be there automatically

6

11.  The only thing you can’t see in the picture above is the dashed line that is now hovering over both columns. Once I hit OK or apply, it all goes away and my formula is:

=$A$3:$B$13

Pre-Select your columns before you start with Conditional formatting

Now I found out some weirdness about the formula. If you have to go back and change the length of the column, you have to make the formula a little differently.  You click into the applies to field

and highlight Column one, then you type a comma, and then you highlight column 2. So the formula looks like this

=$A$3:$A$13,$B$3:$B$13

The more general solution is coming up below:

=$A:$A,$B:$B

I like the second formula better, because you can see the start and finish of each column. But the more general one covers all KB in both Columns. cant beat that. I am editing this sentence,

after the fact, and I found that if you select the columns, by Highlighting the columns entirely, you don’t have to deal with touching any formula aspect at  all.

Here the download again for My Excel HotfixCompare , You can also watch the video at the top and make your own spread.

Believe it or not, that is it! When you hit OK, your values immediately calculate out:

7

 

The next issue I tried to solve was the length of the column. It looks to me that this is the master formula, no matter how many KB s you have  (=$A:$A,$B:$B). This is for column A and B.

I do believe you could replace these values with columns a and c or b and E or whatever. This can help you move columns for your ease of use and you list can be as long or short as you want.

So use this in your Duplicate Values formula:

=$A:$A,$B:$B

Again, I am coming back after the fact, and just saying to Highlight the columns you want to use, before you start, and you don’t have to deal with the formulas at all. I have a second Excel

Method for comparing up to 6 servers, but I will let you just download the Excel File and see how it works!!! Watch the video, I performed that in about 1 minute in the video. also

after the fact, and I found that if you select the columns, by Highlighting the columns entirely, you don’t have to deal with touching any formula aspect at  all.

 

How do I get my Hot-fix list = Get-Hotfix

So this is all we have left to cover. Below these are the steps I use to get that list of KB numbers isolated from the rest of the information you find when you run the Get-Hotfix command.

 

1. Open PowerShell as Administrator
2. You may run Get-Hotfix > c:\myhotfix.txt
3. Open Notepad ++ and Open myhotfix.txt
4. Hold Control and Alt Key on your keyboard at the same time
5. While (4) use the mouse to select just the KB column
6. once the KBs are highlighted Right click and choose copy.
7. Paste you KBs into notepad for transport to your work station.

That’s It! I hope you have enjoyed this article. In closing I have the original link for the script in the initial discussion. I also have a method to update your Hyper-V Cluster automatically with a script. That can be pretty handy.

 

Advertisements

Why Network Design is so Important for Hyper-V CSV Clusters. And Some Common Items to Check

I realize I have written several Hyper-V articles lately. They all come for the unique perspective of technical  support. I now see, I have been trying to put together some material to help understand how these new 2016 Clusters are from the 2012 Clusters from the 2008 Clusters.

So I want to express three clear goals in this paper. I want to define a list of items you may want to read, to make your Cluster as supportable as possible. Second, I simply want to speak to the Importance of your NIC hardware purchases, as it relates to the past and current Stance of Microsoft, with respect to Network requirements, of  Hyper-V cluster setups. Finally, I just summarize some of the command line setting you may look at if you don’t have the optimal NIC setup for a Cluster.

 

Disclaimers

 

      • Disclaimer (agent), While there are items in this discussion, that may not be good advice, for your particular Infrastructure, or your particular situation, I am writing this from a familiar perspective, that Admins, and Designers are approaching me from. Namely this is an “I want it all, and I want it all to work now!” type of design steps.
      •  Translation: (customer needs)Give me the settings that make the fullest use of my server, give me the most VMS, with the most possible resources, and I want to Live Migrate them all day long. Furthermore, I want to be able to host conference in an RDP session, on any one of my Cluster Nodes, and not have any problems.

 

Facts and Common issues in today’s and yesterday’s Hyper-V CSV Clusters

 

With this disclaimer in mind let us proceed. First things first. I am providing this list based on my 10 years with Hyper-V and Clustering, along with the reading and video information I have come across.:

 

      1. Never Use the (Hyper-V) shared Network Adapter as a NIC in the Host Server.
      2. Never Software Team with the NDIS driver Installed from your NIC vendor.
      3. Software teaming is fine for most workloads, unless you’re having latency problems. The answer is Hardware Teaming, or Vice Versa
      4. Don’t put SQL servers on your Hyper-V host.
      5. VM QUEUING can be a problem. Try your workloads with and without VM Queuing and see which works best for your situation.
      6. TCP Offload is not supported for Server 2012 Cluster Teams. Check the other settings here
      7. The Preferred software Team setting is Hyper-V PORT virtualized, switch Independent teaming is best. This is where we are at today. Remember, you had access to these statements in current documentation.
      8. If you Use the Multiplexer Driver as the Virtual Machine NIC, do not turn around and share that NIC with the Hyper-V Host. This is not pretty.
      9. Use Jumbo Frames, QOS etc., where you’re supposed to, according to the current guidelines
      10. Piggy Back off of #8 is that today’s Clusters with Hyper-V, are a balance of Isolation and Bandwidth networks. There is no hard fast rule on how many Network adapters you need.
      11. You cannot just say a node is too slow or fast. When you first install the server, you need to perform clearly laid out, baseline testing, where similar results can be obtained for your server, in pristine condition, with no other workloads. The Same is true for Virtual Machines.
      12. You cannot run a Hyper-V CSV cluster, with all your NICS in the TEAM. You need at least 3 networks in any version of Hyper-V. This is
        1. Cluster Only (Cluster Communication)
        2. Cluster and Client (Management etc.…)
        3. No Cluster Communication. (ISCSI)
      13. Run Cluster Validation- If your updates don’t match across your cluster, you need to get all your nodes to match, before the cluster will work properly.
      14. Clustering only recognizes one NIC per sub-net, when you add multiple NICS to cluster
      15. Back up applications and Antivirus, may have compatibility issues, disable both, and see if the issues disappear.
      16. Network Considerations
      • The Binding Order and DNS must match your current MS documentation. DO not miss this.
      • Cluster Setup now adds rules to the firewall automatically. If you are using Symantec Endpoint. These Firewall rules can serve as your port list to add to Symantec firewall.
      • You can Now Sysprep with Cluster Role installed now for Server 2012.
      • Your NETFT is enabled at the physical NIC, Where you find your IPV4 properties.  Do not Disable it.
      •  So if you are setting up converged clusters, you now have to rely on cluster validation to tell you, if you have enough networks to effectively set up your cluster. Resolve any of the network issues here in validation
      • CSV traffic includes Metadata Updates and Live migration data, as well as failure recovery (IE no storage connectivity) you cannot break this traffic into isolated streams.
      • CSV needs NTLM and SMB- don’t disable either.
      • ISCI Teams now work with MPIO and Jumbo Frames. Jumbo Needed for ISCSI
      • Using multiple Nic Brands Is now preferred.
      1. This series of Articles covers topics I did not go into In Depth. Topics Include:
        1. Mapping the OSI model
        2. VLANS
        3. IP routing
        4. Link Aggregation and Teaming
        5. DNS
        6. Ports, Sockets and Applications
        7. Bindings
        8. Load Balancing, Algorithms

 

 

Cluster nodes are multi-homed systems.  Network priority affects DNS Client for outbound network connectivity.  Network adapters used for client communication should be at the top in the binding order.  Non-routed networks can be placed at lower priority.  In Windows Server 2012/2012R2, the Cluster Network Driver (NETFT.SYS) adapter is automatically placed at the bottom in the binding order list

 

Network Evolution and common sense Network needs

This section is really addressing how we build clusters today. For example, See recently how I wrote a paper on using the old Isolation rules for a simple 2016 cluster, based off the old method of deployment. This method is elegant and works well, will little maintenance needed.

For 2012, and forward, we have the new design which is detailed in the Tech net article, “Network Recommendations for a Hyper-V Cluster in Windows Server 2012”. In this paper, it  Includes the modern setup, using the a software team, and scripted Network Isolation

This paper interleaves these two philosophies, at least that was the intent or message, you are always using one or the other as a guiding principle. Insofar as you have the technical reasoning to do so. what I mean is, if you have 10GB nics, you may fully move to the 2012 method. If you have like 3 1GB nics, you are leaning on the 2008 article to explain to the customer why live migration would not work properly.

Get logging information for Hyper-V and clustering from this article

The quick history of the CSV cluster as follows:

2008

Heartbeats/Intra Cluster Communications -in some documentation  (1GB)

CSV I/O Redirection  (1GB)

VM Network (1GB)

Cluster Network (1GB)

Management Network (1GB)

ISCSI Network – (1GB)

 

2012 and 2016

HeartbeatsNetwork Health monitoring in some documentation (QOS IMPORTANT)  (10GB)

Intra Cluster Communications (QOS IMPORTANT)  (10GB)

CSV I/O Redirection (Bandwidth Important)  (10GB)

ISCSI Network – Not registered in DNS (10GB)

 

This is where you can clearly see how new clusters in 2016, just don’t have the same specifications. The recommendation here is to adjust the Cluster Networks, by the number of network adapters, and what the throughput is. If the NIC setup looks like the 2008 cluster, then apply 2008 network setup guidelines. If the cluster has 2 or more 10GB nics, then treat it as the newer 2016 logic. This has worked well for me for some time now. This will ensure that you get the best Isolation and throughput for your customer.

So as you can see, the Number of NICS is going down, but the NIC SPEED is going up. To make matters more difficult, Microsoft Now states that to be optimized, a CSV cluster will have a combination of Isolation and Bandwidth. They are no longer able to lean on the hard 5 to 7 NIC requirement that once was the norm. For proof of this, you will need to watch this video entitled. ” Fail over Cluster Networking essentials. ”

So really, Support may not be giving you a great explanation as to why your CSV cluster is slow. It is really closely related to the Network Design. Does your Network look more like a 2008 cluster, or a 2012 or 16 cluster? This will give you justification as to why a cluster would be slow or fast.

Server 2012 Requirements are here, along with a basic script for Embedded Teams

In addition to the script above, you also have control over the heartbeat, and other things like priority of the various Cluster NICS and timeouts.

 

Settings you may look at to change if needed

The rest of this article, just shows you some config settings, if you find you have to make a manual change. With a 2016 cluster, they are saying its all automatic, and should not be changed.

While you can make changes to the following. The recommendation is to leave the settings alone. The automatic settings should adjust to the proper, situational network changes:

 

Configure Cluster Heart Beating

 

(Get-Cluster). SameSubnetDelay = 2

 

The Above command Is an example of how you set the following variables. They are posted below with their Default values

 

  • SameSubnetDelay (1 Second)
  • SameSubnetThreshold (5 heartbeats)
  • CrossSubnetDelay  (1 Second)
  • CrossSubnetThreshold (5 heartbeats)

The above setting Is for regular clustering. For Hyper-V clustering, consider the following defaults

  • SameSubnetThreshold (10 heartbeats)
  • CrossSubnetThreshold (20 heartbeats)

If you go more than 10 to 20 on these two settings, the documentations says the overhead starts to interfere, more than the benefit. FYI.

This Step Below is only for allowing the creation of the cluster on a Slow network. Set the value of SetHeartbeatThresholdOnClusterCreate  to 10, for a value of 10 seconds.

HKLM\SYSTEM\CurrentControlSet\Services\ClusSvc\Parameters
add DWORD value SetHeartbeatThresholdOnClusterCreate

 

Configure Full Mesh HeartBeat

(Get-Cluster). PlumbAllCrossSubnetRoutes = 1

 

Other Important changes to change Cluster Setup Parameters

Please Be advised, All the following syntax, has been duplicated from this publicly available Microsoft Article:

Change Cluster Network Roles ( 0=no cluster, 1=Only cluster communication, 3=Client and Cluster Communication)

  • (Get-ClusterNetwork “Cluster Network 1”). Role =3
  • Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role
  • ( Get-ClusterNetwork “Cluster Network 1” ).Metric = 900
  • ( Get-ClusterNetwork “Cluster Network 1” ).AutoMetric = $true

Set Quality of Service Policies (values 0-6) ( Must be enabled on all the nodes in the cluster and the physical network switch)

  • New-NetQosPolicy “Cluster”-Cluster –Priority 6
  • New-NetQosPolicy “SMB” –SMB –Priority 5
  • New-NetQosPolicy “Live Migration” –LiveMigration –Priority 3

 

Set Bandwidth policy (relative minimum bandwidth policy) (It is recommended to configure Relative Minimum Bandwidth SMB policy on CSV deployments)

  • New-NetQosPolicy “Cluster” –Cluster –MinBandwidthWeightAction 30
  • New-NetQosPolicy “Live Migration” –LiveMigration –MinBandwidthWeightAction 20
  • New-NetQosPolicy “SMB” –SMB –MinBandwidthWeightAction 50

 

If you need to add a Hyper-V replica

  • Add-VMNetworkAdapter –ManagementOS –Name “Replica” –SwitchName “TeamSwitch”
    Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName “Replica” –Access –VlanId 17
    Set-VMNetworkAdapter -ManagementOS -Name “Replica” -VmqWeight 80 -MinimumBandwidthWeight 10
    # If the host is clustered – configure the cluster name and role
    * (Get-ClusterNetwork | Where-Object {$_.Address -eq “10.0.17.0”}).Name = “Replica”
    *(Get-ClusterNetwork -Name “Replica”).Role = 3

From <https://technet.microsoft.com/en-us/library/dn550728(v=ws.11).aspx>

Configure Live Migration Network

  • # Configure the live migration network
    Get-ClusterResourceType -Name “Virtual Machine” | Set-ClusterParameter -Name MigrationExcludeNetworks -Value ([String]::Join(“;”,(Get-ClusterNetwork | Where-Object {$_.Name -ne “Migration_Network”}).ID))

From <https://technet.microsoft.com/en-us/library/dn550728(v=ws.11).aspx>

 

Other Commands

  • Enable VM team Set-VMNetworkAdapter -VMName <VMname> -AllowTeaming On
  • Restrict SMB – New-SmbMultichannelConstraint -ServerName “FileServer1” -InterfaceAlias “SMB1”, “SMB2”, “SMB3”, “SMB4”

 

 

The Fallacy of Performance or; Are you bringing your Support Agent Apples or Oranges? VM Virtualization Performance with DiskSpd.

The Fallacy of performance.

I don’t think you don’t already know this. My experience does tell me that we all group things together naturally and sometimes the performance issues we find, are really assertions made, with one piece of evidence. This performance claim is generally hard for a support agent to frame. Not that your case wont be worked, on. It will. It just it may take support teams hours or weeks to get to the truth of your statements.

I have to write this because it is so prevalent. When someone calls me and they want to open a support case, I generally try to standardize the case to some truthful statements, which I can prove, disprove, or alter.

However one such case type, that does not fit into such neat lines is the Virtualization of Performance case. Rather than describe in computer terms, let’s use American Auto Makers Ford and Chevy.

Baselines Matter

I Studied Ford and Chevy Specifications for months. I know the performance characteristics of each very well. Let’s even say I own a Chevy, and I am now looking for a second car of equal specifications. Let’s even say I own a Ford and am looking for a second ford. Then I purchase that second car, and on the way home, I find that the one car does not seem to meet the specifications of the first!

Of course I must call the car company and make the complaint that one car is not as fast as the other, or not as quick to brake, or some other specification. How about this; the air is not as cold in one as it is in the other!! This is what happens in Performance calls.

Just so you know we technically should not even entertain these types of questions. But in support we do, to some degree, because we want to help, and were not sure what your showing us yet. We don’t see the pieces for several calls.  You can’t force it, it just takes time.  This is because you are asking us to form a relationship to ideas that are not related. Two cars are not related, two computers are not related.

 

How about taking a cross country trip? One car crossed the country in three days. The other took 4. At this point you may be seeing my point. Trying to get down to difference in different items, of any type can be like comparing apples to oranges. What’s even worse is when one of the items is off limits. So My Ford is definitely slower, but my wife takes the other ford to work so we really can’t use that one for testing! Now what do we do?

Basic Performance assumptions

So there are some simple rules that you should apply to any performance problem:

 

  1. The performance should be documented and repeatable.
  2. More than one test should be run, and simple is usually more realistic.
  3. Tests should be standardized, down to a science, so that if applied to another matching scenario, you would expect similar results.
  4. Keep the time down to a short test. The longer the test, the more variables can be introduced.
  5. Do not focus on two separate car models not functioning the same, find a way to introduce a baseline into what a reasonable car will perform like. Then prove or disprove your baseline.

 

Now obviously, the complexity of computers can result in more rules, but if you follow these basics, you can at least find some sanity in your test results. In fact Support has an absolute need, that this happens. It is very possible, nothing is really, wrong, if we don’t get down to brass tacks.

So real world

Call into support and report when you run this command on one machine, things are fine. When you run on the other environment, things failed. This is the Disk Speed command. This is the replacement for SQLIO. I really like this tool.

 

  • diskspd –b8K –d30 –o4 –t8 –h –r –w25 –L –Z1G –c20G  D:\iotest.dat > DiskSpeedResults.txt

 

However, what is hiding in this statement, violates all the 5 rules above. This is an assertion, based on one command. Furthermore, you ran this command, in the test location,  over and over, while other VMS are also running, randomly, creating a random pattern of storage fragmentation, while the Production environment was only one once, in a very controlled situation. These commands were not run in a scientific fashion.

It literally took me a days to think of a way to baseline this situation and to  test this correctly. This is where the 5 rules came from. I think they are solid rules for support to go by. So here is how you test to make your case to Support:

User Guide and Product here

Introduce a Baseline. Anything is better then nothing

The Above diskspd command is complex and long. Come up with some simple tests and run more of them, over time. Second, test your commands, on a laptop, or desktop, with a specific Ram, Storage, and Processor Profile. Once you record all the results on the client machine, duplicate the test in the Virtual Machine. Make sure it’s the only Virtual Machine Running. Make sure nothing is running on the Host but this one VM, with specific resources.

Now below I am not giving you results. I am just giving you the commands, Along with some Instructions on how to use DSKSPD. I am also leaving you with Articles that VMware and Microsoft Hyper-V use, when asking for baseline testing. Notice, how many little requirements they have Seem familiar? There is a reason for this! We are all trying to be scientific.

Tests to establish a Baseline.

  • .\diskspd -c100M -d20 c:\test1 d:\test2
  • .\diskspd -c2G -b4K -F8 -r -o32 -W30 -d30 -Sh d:\testfile.dat
  • .\diskspd -t1 -o1 -s8k -b8k -Sh -w100 -g80 c:\test1.dat d:\test2.dat
  • .\diskspd.exe -c5G -d60 -r -w90 -t8 -o8 -b8K -h -L
  • .\diskspd.exe -c10G -d10 -r -w0 -t8 -o8 -b8K -h -L d:\testfile.dat
  • .\scriptname.ps1
  • Same as above- second location
  • .\Diskspd -b8k -d30 -Sh  -o8 -t8  -w20 -c2G d:\iotest.dat

 

This list will generate about 15 unique results. Any of these will run on a laptop or a server. Just make sure you read the text character decoder sheet available with the product.

So the instructions are very simple. The specs on the Hyper-V or VMware VM, must be the same as the laptop. My laptop has 16GB of ram, and 8 Processors.

The VM must be the only one running, and the OS should be a fresh Install. Now if the results of testing are in the ballpark of your comparison Client, then you are not having a performance issue.

The moral of the story is test from different perspectives, and use the Scientific method, as much as you are able to.

I hope this is helpful in your troubleshooting.

 

A few other Details

Here is a way to manually pre-create the files if disired

  • fsutil file createnew d:\iotest.dat 20000000
  • fsutil file createnew d:\iotest.dat 2000000000
  • fsutil file createnew d:\iotest.dat 20000000000

Here is all of the best articles on storage, and IO online right now. I was surprised that so many of  Storage Performance Needs are all in one place.

This could be an important point. If you came to this site because your numbers are not matching reality, your Monitoring tools, may not be collecting the right perfmon numbers, then you may need the Hyper-V performance script to use to see your actual VM numbers. try using this tool Run this tool on the host, while using diskspd on your VM.

DO not run more then one instance of Diskspd at once!! This will invalidate your tests!

 

Finally, as promised, here is how  VMware or Microsoft  handle these issues:

 

Louis

Deploy Remote Desktop Services in a Workgroup or Domain Easily!

image

Update 10/9/2017

https://support.microsoft.com/en-us/help/2833839/guidelines-for-installing-the-remote-desktop-session-host-role-service

This article was the underlying document the post came from. Notice at the bottom it is valid for 2016 as well!

Applies to

Windows Server 2016 Datacenter, Windows Server 2016 Standard,
Windows Server 2012 R2 Datacenter, Windows Server 2012 R2 Standard,
Windows Server 2012 Datacenter, Windows Server 2012 Foundation

—————————————————————————-

 

For all you Terminal Server or Remote Desktop Services or RDP Geeks out there, Let me spend a minute to clarify a call driver that continues to be popular.

The Scenario is deploying Remote Desktop Services in a work group. Call this a corner case, or call it what you will. The reason this is a popular support call is due to the fact that there are two articles needed to complete the setup.

Oh sure Microsoft does tell you to add a policy after your setup, but they specify you use GPEDIT. not much help there… Until Today!

First you need to Deploy the roles correctly. The Specific KB I chose for this article, is the one you would use for the simplest setup. One that keeps you clear of very common mis-steps of walking through the setup in the Server Manager. If you did your deployment correctly, you didn’t even need Server Manager.

So begin by reading this:

Guidelines for installing the Remote Desktop Session Host role service on a computer running Windows Server 2012 without the Remote Desktop Connection Broker role service

“”RD Session Host without the Connection Broker

To be clear about this deployment:

  • No Domain Controller
  • No Hyper-V
  • Install Remote Desktop Services.
  • You won’t get to select more until the end. You may install the following
      1. Remote Desktop Licensing and
      2. Remote Desktop Session Host
      3. RDWEB
      4. RD Gateway
  • For a work group you do not have to add the Terminal Server to the Terminal Server License server group. Why? With No DC, you don’t have to report to anyone.
  • Activate the Terminal Server using one of the three methods.. This is most common
  • Activate your RD Cal Licenses with KB2833839 or cc725890

So Far, so good. Now this is where we start to diverge from some existing Documentation

  • If you are in a work-group, go to “edit local users and groups”
  • Find the group folder and create a group for your RDP users and add your users to this group.
  • Alternatively, you may add your users to the RDP users group already there
  • Remember the group you are using. It becomes important
  • Now you are going to edit the Local Policy by doing the following:
      1. Start and Run GPEDIT.MSC
      2. Navigate to the following:

Local Computer Policy ->Computer Configuration-> Administrative Templates-> Windows Components-> Remote Desktop Services-> Remote Desktop Session Host-> Licensing

Figure 1. GPEDIT.MSCUntitled

You are going to see now, the two LSO (Local Security Object) you will be enabling.

Use the Specified Remote Desktop License Servers –Value- IP address of RD License Server.

Set the Remote Desktop Licensing Mode- Value – 2 or 4. 2 is for Device CAL and 4 is user Cal.

Figure 2. Local Policy. Untitled

Now there is another Policy to set. For this, you want to just go back to the top. Start out at Local Security policy like before (GPEDIT.MSC).See figure 3.

 

Figure 3 GPEDIT.MSCimage

 

Expand Computer –>Configuration,–> expand Windows Settings, –> expand Security Settings,–> expand Local Policies and then click User Rights Assignment.

image

 

image

Enable this policy and add the group you used earlier. It is highlighted in this article above. Add this group to this policy. In addition, add the Remote Desktop Users group to this policy if desired. Don’t and your administrator name here. The Admin already has access. If you ad your admin name, it will lock you out. So best to to stick to adding the Remote Desktop users group.

image

 

Notice it says administrators (Plural), that is fine, but the single administrator should not be in this list. there is a well known break here if you do that.  When you are finished, you just need to add the users you want to give access to, to the group we just added to the Policy (likely the remote desktop users group)

Step 8 comes right out of the KB2833839

  • Open an elevated Windows PowerShell prompt
  • Type the following command on the PS prompt and press Enter:
    $obj = gwmi -namespace “Root/CIMV2/TerminalServices” Win32_TerminalServiceSetting
  • Run the following command to set the licensing mode:
    Note: Value = 2 for Per device, Value = 4 for Per User
    $obj.ChangeMode(value)
  • Run the following command to replace the machine name with License Server:
    $obj.SetSpecifiedLicenseServerList(“LicServer”)
  • Run the following command to verify the settings that are configured using above mentioned steps:
    $obj.GetSpecifiedLicenseServerList()
    You should see the server name in the output.

You have now covered all your bases, and  your RDP should be happy!! It will be happy because you paid attention to all the rights things!

Now I did find an interesting article to which I cant really comment on. However, it is an interesting article. IT deals with some issues, you could run across.

Based on a comment to this article, I want to address this KB    “Best practices for setting up Remote Desktop Licensing (Terminal Server Licensing) across Active Directory Domains/Forests or Work group”

This is where it tells you not to use a work group server. So Please consider this against this KB_

Guidelines for installing the Remote Desktop Session Host role service on a computer running Windows Server 2012 without the Remote Desktop Connection Broker role service

One article says don’t use a work-group. The other says “This server can be part of a work-group or may be configured as a DC.” Which do you believe? They are both true. Microsoft does not want you to use it in a work-group, but you can because they provided a way for you to do it. The cost? Printer Redirection issues, and a few other small things, but you can do it if you need to. Bottom line is Install your Terminal Server in a Domain. But, If you have no other choice then use a work-group if you have to.

 

Well That is it. I hope this has been helpful for work groups or non-work groups. this basically  can be set up on either.

 

 

Getting Accurate Latency from Dynamically expanding Hyper-V virtual Machine Disks

 

This Article is about the tool called Hyper-V Performance Monitor Tool (PowerShell)

you can download it from the tech net article down the page, or use the link above.

So Hyper-V gets thrown around loosely these days, when you talk about Virtualization or Performance Tuning, or Planning or any other aspect of the product life-cycle of a new Host Deployment.

Over the last few years, we have made rapid changes from Physical Host Machines for production work loads, to these Virtual monstrosities, that now host our whole company.

Along with this change, you may recall that Early Hyper-V documentation has gently let us know that monitoring inside the Virtual Machine was not going to give us the parity results to the Physical counters, depending on configuration.  This is for a few reasons, which are beyond this articles scope. However, I would like to shine light on this, so more people can think differently about their Virtual performance.

The most common measure of how well a server is performing is Latency in Milliseconds. Every one is most concerned with how much latency is in the storage system. Perhaps with good reason. The San storage vendors can perform so fast now days, that you can throw the empire state building at a server, and the Latency is less then 10 Milliseconds (MS). Or is it throughput?

To be clear, we are interested more in Latency then throughput. Latency should be minimized, and throughput will generally increase.

Can I make a case that Counters are not reliable?

Well let me tell your actual Latency is not as easily attained as you would think. If I told you that your tests are lying to you, would you believe me? Lets say your in shock. Without knowing your design, my generalized answer has a very high chance of being correct.  The issue is that you have a SAN and you are tying to get latency by measuring responses to a file, that go though 4 different filters, and then has to wait until it gets queued to a disk subsystem, that is always expanding. I promise you; your numbers are incorrect.

If you cannot quantify how latency and throughput are different, but related, then I would say you should not stop reading.

Storage Latency of VM guests

There are many problems with calculating storage latency, but disk is the model we are going to use to illustrate how tricky it can be to find out how your VM is performing.

The most common approach to gaining latency information is to use a command line tool. Normally the tool will work fine. The model breaks down when the Disk itself is changing, along with the Ram and processor availability. The bottom line is a Virtual machine may lie to you about resource numbers at any given time.  Add to the mix, that the clock cycle is a weakness in any virtualization platform. That means that the calculation of time itself can result In poor results based on good math with bad numbers.

There are a crowd of you who say that is bull. Well all I can say is; don’t read this and good luck solving your latency issues.

Let me try to list some areas where the numbers may go awry. I am just making a one line explanation with a link, so you can read more. I don’t want this to be about the problems. Below, I talk more about the solution. Read more if you have a specific issue:

 

I could keep going. Do you get the feeling there are a ton of variables that change how storage latencies should be calculated?

From my experience, I have found that every set of servers are their own data set of network behavior. There are some basic assumptions I found to pass along to Admins who want to find out latency of Virtual machines.

Guidelines for VM latency Study

Who to Blame

So again, the basic message is that the calculation of latencies is totally based on the sum of the deployment factors. In one data center you may find under reporting, and the other Over Reporting. Support agents do not have the Onus to prove why one is slower then the other. We will have to look at your Design and deploy and try to make a story of things we can identify. it is not likely we will find that moment where the Deployment deviated from your Baseline storage latency measurements. We offer Best effort, but encourage you to strip down your deployment to make a core Baseline latency for a Dynamically expanding VM. All Vms will compare to that one. We go from there.

Using the Stop Gap solutions for Monitoring Virtual Machines

SO it was just a few years ago, this issue with VM monitoring was not easily remedied. You could certainly use the Perfmon counters to get VM stats. But Customers just want to run Disk Speed or SQLIO, and get an output to look at. This did not exist for quite some time. Thankfully there is a script out there, that will now carve out some parity to those tools. the link is at the TechNet Gallery:

Gallery.TechNet.Com

Hyper-V Performance Monitor Tool (PowerShell)

Below is the walk through of the basic performance collection.

you Just run the Script from an Admin PowerShell. There are a few ways to run it:

.\Monitor-HyperVGuestPerformance.ps1

### export data to csv via GUI, defaults to current dir
.\Monitor-HyperVGuestPerformance.ps1 -ExportToCsv

### retrieve data as PSobjects, great for parsing and logging, -name parameter is optional, defaults to automatic discovery
.\Monitor-HyperVGuestPerformance.ps1 -PSobjects

### specify host and interval/samples manually
.\Monitor-HyperVGuestPerformance.ps1  -Name host1,host2 -PSobjects -Interval 2 -MaxSamples 5
### accepts pipeline input
‘Host1′,’Host2’
| .\Monitor-HyperVGuestPerformance.ps1 -PSobjects

### Log to SQL server with Write-ObjectToSQL , this example uses SQL auth
.\Monitor-HyperVGuestPerformance.ps1 –PSobjects  |  Write-ObjectToSQL –TableName table –Database db -Server server –credential (get-credential)

 

image

If the domain connection fails, it tries for a Local connection:

 

image

In my case, I ran the tool on the Host, and this GUI below popped up. All I did was hit monitor, and I got an export vm_perfmon_stats file. This file can be used to find your latency.

Untitled

While this method may not be pretty, It does follow the rules for Hyper-V guest. The main purpose for this tool would be to use instead of SQLIO or DISK SPEED. tools like these should be used for hardware testing. A Hyper-V Server, running on ISCSI shared storage, with two VHDX files attached, is likely going to come back with Erroneous Latencies. This may not be perfect, but I do believe you will see a consistent result that is not a totally unbelievable number.

See I changed the Sample and interval:

Untitled

And I get a time-frame to wait for the test results:

image

Find the Link at the Microsoft TechNet Center. Thank you for taking the time to Read about Storage Performance for Hyper-V virtual Machines

I hope this helps in your Baseline Studies.

The result is a nice little Excel Display of the data, that I cleaned up a little with colors, to the Excel Fields.

 

image

Louis

How to activate a license server and install CALs

Glennopedia

This is the 3rd article in a series of articles on RDS (Remote Desktop Services)  In this installment I will cover how to activate a license server and installing RDS CALs (Client Access Licenses)

To activate the license server:

  1. Open the RD Licensing Manager.  (Right-click Stat button, Run, licmgr.exe)
  2. Expand All servers.
  3. Right-click the server name and choose Activate Server.
    Activate Server1
  4. Click Next on the Welcome screen.
  5. Click Next on the Connection Method screen.
  6. Enter the company information.  This should match the information given when the RDS CALs were purchased.  Click Next when complete.
  7. This screen is optional.  You can enter the information if you choose.  Click Next when complete.
  8. The Completing the Activate Server Wizard screen should come up and indicate the server was activated successfully.  If the CALs are available to install, Click Next.  Otherwise uncheck the “Start Install Licenses Wizard now” option and click Finish.

To Install RDS…

View original post 139 more words

For That Once in a Blue Moon when I Need That Program Called RoboCopy!

Hello everyone,

 

Untitled

Happy Friday! Let me start by saying I know many of you may not feel the need to keep this article. However, In that one Blue Moon, I guarantee you will need the information, because Robocopy has some odd scenarios.

I only call light to the thing we do with Robocopy: copy files. There are two scenarios for this. The target existed previously, or the target did not. Expressed another way, you are copying files or you are syncing files. It is a fine line indeed. So I tried to supply the basic commands below. If you are creating the target, then run the Create Target commands. If you are Syncing, then run the Sync Target Commands.

Create Target Commands

I know you likely already know how to use robocopy. But this little application can go wrong, if you make a mistake in your syntax. I decided to just share my basic commands with you. I generally just want a whole drive or a whole folder moved, with thread control.

The scope of my command is narrow, but it gets the job done in the fastest time, without a mistake.

the general form is

robocopy (source)(destination)(controls)

Here are solid examples of copying when the target does not yet exist.

In the above commands, the source is copied to the destination. You may add /SEC to any of these commands. This will copy and keep the security permissions. On a first time copy, the command will perform as expected.

Sync Target Commands
Now I am not getting into the insanity when the target exists. we all have too many things to do. If the Files have been copied before, to the destination, then you have to worry about files that have not changed from source to destination. My vote would be to delete all the target folders and recopy. If that does not work for you, then read on.

If you have an existing source and old target, meaning you are really syncing data, and not copying, there is another set of commands I will lay on you:

  • ROBOCOPY /Mir <Source> <Target>
  • ROBOCOPY /E /Copy:S /IS /IT <Source> <Target>

You have to run both commands! This is a work around, but it works. If you want to spend the hours of spinning up on robocopy, go ahead. I plan to just keep this article, with the two basic sets of commands, to get the work done.
I sourced my information from the following blog, which you can read and understand if you need detail:

Robocopy Mirror Switch Mirroring File Permissions

I hope this is helpful. I know we don’t do this too often, but I wanted to make sure I pointed to the Correct Article for this issue. This is likely going to remain a work around, based on the behavior of the command.

 

Have a great Early Spring weekend!

 

Louis Reeves