Why Network Design is so Important for Hyper-V CSV Clusters. And Some Common Items to Check

I realize I have written several Hyper-V articles lately. They all come for the unique perspective of technical  support. I now see, I have been trying to put together some material to help understand how these new 2016 Clusters are from the 2012 Clusters from the 2008 Clusters.

So I want to express three clear goals in this paper. I want to define a list of items you may want to read, to make your Cluster as supportable as possible. Second, I simply want to speak to the Importance of your NIC hardware purchases, as it relates to the past and current Stance of Microsoft, with respect to Network requirements, of  Hyper-V cluster setups. Finally, I just summarize some of the command line setting you may look at if you don’t have the optimal NIC setup for a Cluster.

 

Disclaimers

 

      • Disclaimer (agent), While there are items in this discussion, that may not be good advice, for your particular Infrastructure, or your particular situation, I am writing this from a familiar perspective, that Admins, and Designers are approaching me from. Namely this is an “I want it all, and I want it all to work now!” type of design steps.
      •  Translation: (customer needs)Give me the settings that make the fullest use of my server, give me the most VMS, with the most possible resources, and I want to Live Migrate them all day long. Furthermore, I want to be able to host conference in an RDP session, on any one of my Cluster Nodes, and not have any problems.

 

Facts and Common issues in today’s and yesterday’s Hyper-V CSV Clusters

 

With this disclaimer in mind let us proceed. First things first. I am providing this list based on my 10 years with Hyper-V and Clustering, along with the reading and video information I have come across.:

 

      1. Never Use the (Hyper-V) shared Network Adapter as a NIC in the Host Server.
      2. Never Software Team with the NDIS driver Installed from your NIC vendor.
      3. Software teaming is fine for most workloads, unless you’re having latency problems. The answer is Hardware Teaming, or Vice Versa
      4. Don’t put SQL servers on your Hyper-V host.
      5. VM QUEUING can be a problem. Try your workloads with and without VM Queuing and see which works best for your situation.
      6. TCP Offload is not supported for Server 2012 Cluster Teams. Check the other settings here
      7. The Preferred software Team setting is Hyper-V PORT virtualized, switch Independent teaming is best. This is where we are at today. Remember, you had access to these statements in current documentation.
      8. If you Use the Multiplexer Driver as the Virtual Machine NIC, do not turn around and share that NIC with the Hyper-V Host. This is not pretty.
      9. Use Jumbo Frames, QOS etc., where you’re supposed to, according to the current guidelines
      10. Piggy Back off of #8 is that today’s Clusters with Hyper-V, are a balance of Isolation and Bandwidth networks. There is no hard fast rule on how many Network adapters you need.
      11. You cannot just say a node is too slow or fast. When you first install the server, you need to perform clearly laid out, baseline testing, where similar results can be obtained for your server, in pristine condition, with no other workloads. The Same is true for Virtual Machines.
      12. You cannot run a Hyper-V CSV cluster, with all your NICS in the TEAM. You need at least 3 networks in any version of Hyper-V. This is
        1. Cluster Only (Cluster Communication)
        2. Cluster and Client (Management etc.…)
        3. No Cluster Communication. (ISCSI)
      13. Run Cluster Validation- If your updates don’t match across your cluster, you need to get all your nodes to match, before the cluster will work properly.
      14. Clustering only recognizes one NIC per sub-net, when you add multiple NICS to cluster
      15. Back up applications and Antivirus, may have compatibility issues, disable both, and see if the issues disappear.
      16. Network Considerations
      • The Binding Order and DNS must match your current MS documentation. DO not miss this.
      • Cluster Setup now adds rules to the firewall automatically. If you are using Symantec Endpoint. These Firewall rules can serve as your port list to add to Symantec firewall.
      • You can Now Sysprep with Cluster Role installed now for Server 2012.
      • Your NETFT is enabled at the physical NIC, Where you find your IPV4 properties.  Do not Disable it.
      •  So if you are setting up converged clusters, you now have to rely on cluster validation to tell you, if you have enough networks to effectively set up your cluster. Resolve any of the network issues here in validation
      • CSV traffic includes Metadata Updates and Live migration data, as well as failure recovery (IE no storage connectivity) you cannot break this traffic into isolated streams.
      • CSV needs NTLM and SMB- don’t disable either.
      • ISCI Teams now work with MPIO and Jumbo Frames. Jumbo Needed for ISCSI
      • Using multiple Nic Brands Is now preferred.
      1. This series of Articles covers topics I did not go into In Depth. Topics Include:
        1. Mapping the OSI model
        2. VLANS
        3. IP routing
        4. Link Aggregation and Teaming
        5. DNS
        6. Ports, Sockets and Applications
        7. Bindings
        8. Load Balancing, Algorithms

 

 

Cluster nodes are multi-homed systems.  Network priority affects DNS Client for outbound network connectivity.  Network adapters used for client communication should be at the top in the binding order.  Non-routed networks can be placed at lower priority.  In Windows Server 2012/2012R2, the Cluster Network Driver (NETFT.SYS) adapter is automatically placed at the bottom in the binding order list

 

Network Evolution and common sense Network needs

This section is really addressing how we build clusters today. For example, See recently how I wrote a paper on using the old Isolation rules for a simple 2016 cluster, based off the old method of deployment. This method is elegant and works well, will little maintenance needed.

For 2012, and forward, we have the new design which is detailed in the Tech net article, “Network Recommendations for a Hyper-V Cluster in Windows Server 2012”. In this paper, it  Includes the modern setup, using the a software team, and scripted Network Isolation

This paper interleaves these two philosophies, at least that was the intent or message, you are always using one or the other as a guiding principle. Insofar as you have the technical reasoning to do so. what I mean is, if you have 10GB nics, you may fully move to the 2012 method. If you have like 3 1GB nics, you are leaning on the 2008 article to explain to the customer why live migration would not work properly.

Get logging information for Hyper-V and clustering from this article

The quick history of the CSV cluster as follows:

2008

Heartbeats/Intra Cluster Communications -in some documentation  (1GB)

CSV I/O Redirection  (1GB)

VM Network (1GB)

Cluster Network (1GB)

Management Network (1GB)

ISCSI Network – (1GB)

 

2012 and 2016

HeartbeatsNetwork Health monitoring in some documentation (QOS IMPORTANT)  (10GB)

Intra Cluster Communications (QOS IMPORTANT)  (10GB)

CSV I/O Redirection (Bandwidth Important)  (10GB)

ISCSI Network – Not registered in DNS (10GB)

 

This is where you can clearly see how new clusters in 2016, just don’t have the same specifications. The recommendation here is to adjust the Cluster Networks, by the number of network adapters, and what the throughput is. If the NIC setup looks like the 2008 cluster, then apply 2008 network setup guidelines. If the cluster has 2 or more 10GB nics, then treat it as the newer 2016 logic. This has worked well for me for some time now. This will ensure that you get the best Isolation and throughput for your customer.

So as you can see, the Number of NICS is going down, but the NIC SPEED is going up. To make matters more difficult, Microsoft Now states that to be optimized, a CSV cluster will have a combination of Isolation and Bandwidth. They are no longer able to lean on the hard 5 to 7 NIC requirement that once was the norm. For proof of this, you will need to watch this video entitled. ” Fail over Cluster Networking essentials. ”

So really, Support may not be giving you a great explanation as to why your CSV cluster is slow. It is really closely related to the Network Design. Does your Network look more like a 2008 cluster, or a 2012 or 16 cluster? This will give you justification as to why a cluster would be slow or fast.

Server 2012 Requirements are here, along with a basic script for Embedded Teams

In addition to the script above, you also have control over the heartbeat, and other things like priority of the various Cluster NICS and timeouts.

 

Settings you may look at to change if needed

The rest of this article, just shows you some config settings, if you find you have to make a manual change. With a 2016 cluster, they are saying its all automatic, and should not be changed.

While you can make changes to the following. The recommendation is to leave the settings alone. The automatic settings should adjust to the proper, situational network changes:

 

Configure Cluster Heart Beating

 

(Get-Cluster). SameSubnetDelay = 2

 

The Above command Is an example of how you set the following variables. They are posted below with their Default values

 

  • SameSubnetDelay (1 Second)
  • SameSubnetThreshold (5 heartbeats)
  • CrossSubnetDelay  (1 Second)
  • CrossSubnetThreshold (5 heartbeats)

The above setting Is for regular clustering. For Hyper-V clustering, consider the following defaults

  • SameSubnetThreshold (10 heartbeats)
  • CrossSubnetThreshold (20 heartbeats)

If you go more than 10 to 20 on these two settings, the documentations says the overhead starts to interfere, more than the benefit. FYI.

This Step Below is only for allowing the creation of the cluster on a Slow network. Set the value of SetHeartbeatThresholdOnClusterCreate  to 10, for a value of 10 seconds.

HKLM\SYSTEM\CurrentControlSet\Services\ClusSvc\Parameters
add DWORD value SetHeartbeatThresholdOnClusterCreate

 

Configure Full Mesh HeartBeat

(Get-Cluster). PlumbAllCrossSubnetRoutes = 1

 

Other Important changes to change Cluster Setup Parameters

Please Be advised, All the following syntax, has been duplicated from this publicly available Microsoft Article:

Change Cluster Network Roles ( 0=no cluster, 1=Only cluster communication, 3=Client and Cluster Communication)

  • (Get-ClusterNetwork “Cluster Network 1”). Role =3
  • Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role
  • ( Get-ClusterNetwork “Cluster Network 1” ).Metric = 900
  • ( Get-ClusterNetwork “Cluster Network 1” ).AutoMetric = $true

Set Quality of Service Policies (values 0-6) ( Must be enabled on all the nodes in the cluster and the physical network switch)

  • New-NetQosPolicy “Cluster”-Cluster –Priority 6
  • New-NetQosPolicy “SMB” –SMB –Priority 5
  • New-NetQosPolicy “Live Migration” –LiveMigration –Priority 3

 

Set Bandwidth policy (relative minimum bandwidth policy) (It is recommended to configure Relative Minimum Bandwidth SMB policy on CSV deployments)

  • New-NetQosPolicy “Cluster” –Cluster –MinBandwidthWeightAction 30
  • New-NetQosPolicy “Live Migration” –LiveMigration –MinBandwidthWeightAction 20
  • New-NetQosPolicy “SMB” –SMB –MinBandwidthWeightAction 50

 

If you need to add a Hyper-V replica

  • Add-VMNetworkAdapter –ManagementOS –Name “Replica” –SwitchName “TeamSwitch”
    Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName “Replica” –Access –VlanId 17
    Set-VMNetworkAdapter -ManagementOS -Name “Replica” -VmqWeight 80 -MinimumBandwidthWeight 10
    # If the host is clustered – configure the cluster name and role
    * (Get-ClusterNetwork | Where-Object {$_.Address -eq “10.0.17.0”}).Name = “Replica”
    *(Get-ClusterNetwork -Name “Replica”).Role = 3

From <https://technet.microsoft.com/en-us/library/dn550728(v=ws.11).aspx>

Configure Live Migration Network

  • # Configure the live migration network
    Get-ClusterResourceType -Name “Virtual Machine” | Set-ClusterParameter -Name MigrationExcludeNetworks -Value ([String]::Join(“;”,(Get-ClusterNetwork | Where-Object {$_.Name -ne “Migration_Network”}).ID))

From <https://technet.microsoft.com/en-us/library/dn550728(v=ws.11).aspx>

 

Other Commands

  • Enable VM team Set-VMNetworkAdapter -VMName <VMname> -AllowTeaming On
  • Restrict SMB – New-SmbMultichannelConstraint -ServerName “FileServer1” -InterfaceAlias “SMB1”, “SMB2”, “SMB3”, “SMB4”

 

 

The Fallacy of Performance or; Are you bringing your Support Agent Apples or Oranges? VM Virtualization Performance with DiskSpd.

The Fallacy of performance.

I don’t think you don’t already know this. My experience does tell me that we all group things together naturally and sometimes the performance issues we find, are really assertions made, with one piece of evidence. This performance claim is generally hard for a support agent to frame. Not that your case wont be worked, on. It will. It just it may take support teams hours or weeks to get to the truth of your statements.

I have to write this because it is so prevalent. When someone calls me and they want to open a support case, I generally try to standardize the case to some truthful statements, which I can prove, disprove, or alter.

However one such case type, that does not fit into such neat lines is the Virtualization of Performance case. Rather than describe in computer terms, let’s use American Auto Makers Ford and Chevy.

Baselines Matter

I Studied Ford and Chevy Specifications for months. I know the performance characteristics of each very well. Let’s even say I own a Chevy, and I am now looking for a second car of equal specifications. Let’s even say I own a Ford and am looking for a second ford. Then I purchase that second car, and on the way home, I find that the one car does not seem to meet the specifications of the first!

Of course I must call the car company and make the complaint that one car is not as fast as the other, or not as quick to brake, or some other specification. How about this; the air is not as cold in one as it is in the other!! This is what happens in Performance calls.

Just so you know we technically should not even entertain these types of questions. But in support we do, to some degree, because we want to help, and were not sure what your showing us yet. We don’t see the pieces for several calls.  You can’t force it, it just takes time.  This is because you are asking us to form a relationship to ideas that are not related. Two cars are not related, two computers are not related.

 

How about taking a cross country trip? One car crossed the country in three days. The other took 4. At this point you may be seeing my point. Trying to get down to difference in different items, of any type can be like comparing apples to oranges. What’s even worse is when one of the items is off limits. So My Ford is definitely slower, but my wife takes the other ford to work so we really can’t use that one for testing! Now what do we do?

Basic Performance assumptions

So there are some simple rules that you should apply to any performance problem:

 

  1. The performance should be documented and repeatable.
  2. More than one test should be run, and simple is usually more realistic.
  3. Tests should be standardized, down to a science, so that if applied to another matching scenario, you would expect similar results.
  4. Keep the time down to a short test. The longer the test, the more variables can be introduced.
  5. Do not focus on two separate car models not functioning the same, find a way to introduce a baseline into what a reasonable car will perform like. Then prove or disprove your baseline.

 

Now obviously, the complexity of computers can result in more rules, but if you follow these basics, you can at least find some sanity in your test results. In fact Support has an absolute need, that this happens. It is very possible, nothing is really, wrong, if we don’t get down to brass tacks.

So real world

Call into support and report when you run this command on one machine, things are fine. When you run on the other environment, things failed. This is the Disk Speed command. This is the replacement for SQLIO. I really like this tool.

 

  • diskspd –b8K –d30 –o4 –t8 –h –r –w25 –L –Z1G –c20G  D:\iotest.dat > DiskSpeedResults.txt

 

However, what is hiding in this statement, violates all the 5 rules above. This is an assertion, based on one command. Furthermore, you ran this command, in the test location,  over and over, while other VMS are also running, randomly, creating a random pattern of storage fragmentation, while the Production environment was only one once, in a very controlled situation. These commands were not run in a scientific fashion.

It literally took me a days to think of a way to baseline this situation and to  test this correctly. This is where the 5 rules came from. I think they are solid rules for support to go by. So here is how you test to make your case to Support:

User Guide and Product here

Introduce a Baseline. Anything is better then nothing

The Above diskspd command is complex and long. Come up with some simple tests and run more of them, over time. Second, test your commands, on a laptop, or desktop, with a specific Ram, Storage, and Processor Profile. Once you record all the results on the client machine, duplicate the test in the Virtual Machine. Make sure it’s the only Virtual Machine Running. Make sure nothing is running on the Host but this one VM, with specific resources.

Now below I am not giving you results. I am just giving you the commands, Along with some Instructions on how to use DSKSPD. I am also leaving you with Articles that VMware and Microsoft Hyper-V use, when asking for baseline testing. Notice, how many little requirements they have Seem familiar? There is a reason for this! We are all trying to be scientific.

Tests to establish a Baseline.

  • .\diskspd -c100M -d20 c:\test1 d:\test2
  • .\diskspd -c2G -b4K -F8 -r -o32 -W30 -d30 -Sh d:\testfile.dat
  • .\diskspd -t1 -o1 -s8k -b8k -Sh -w100 -g80 c:\test1.dat d:\test2.dat
  • .\diskspd.exe -c5G -d60 -r -w90 -t8 -o8 -b8K -h -L
  • .\diskspd.exe -c10G -d10 -r -w0 -t8 -o8 -b8K -h -L d:\testfile.dat
  • .\scriptname.ps1
  • Same as above- second location
  • .\Diskspd -b8k -d30 -Sh  -o8 -t8  -w20 -c2G d:\iotest.dat

 

This list will generate about 15 unique results. Any of these will run on a laptop or a server. Just make sure you read the text character decoder sheet available with the product.

So the instructions are very simple. The specs on the Hyper-V or VMware VM, must be the same as the laptop. My laptop has 16GB of ram, and 8 Processors.

The VM must be the only one running, and the OS should be a fresh Install. Now if the results of testing are in the ballpark of your comparison Client, then you are not having a performance issue.

The moral of the story is test from different perspectives, and use the Scientific method, as much as you are able to.

I hope this is helpful in your troubleshooting.

 

A few other Details

Here is a way to manually pre-create the files if disired

  • fsutil file createnew d:\iotest.dat 20000000
  • fsutil file createnew d:\iotest.dat 2000000000
  • fsutil file createnew d:\iotest.dat 20000000000

Here is all of the best articles on storage, and IO online right now. I was surprised that so many of  Storage Performance Needs are all in one place.

This could be an important point. If you came to this site because your numbers are not matching reality, your Monitoring tools, may not be collecting the right perfmon numbers, then you may need the Hyper-V performance script to use to see your actual VM numbers. try using this tool Run this tool on the host, while using diskspd on your VM.

DO not run more then one instance of Diskspd at once!! This will invalidate your tests!

 

Finally, as promised, here is how  VMware or Microsoft  handle these issues:

 

Louis

Deploy Remote Desktop Services in a Workgroup or Domain Easily!

image

For all you Terminal Server or Remote Desktop Services or RDP Geeks out there, Let me spend a minute to clarify a call driver that continues to be popular.

The Scenario is deploying Remote Desktop Services in a work group. Call this a corner case, or call it what you will. The reason this is a popular support call is due to the fact that there are two articles needed to complete the setup.

Oh sure Microsoft does tell you to add a policy after your setup, but they specify you use GPEDIT. not much help there… Until Today!

First you need to Deploy the roles correctly. The Specific KB I chose for this article, is the one you would use for the simplest setup. One that keeps you clear of very common mis-steps of walking through the setup in the Server Manager. If you did your deployment correctly, you didn’t even need Server Manager.

So begin by reading this:

Guidelines for installing the Remote Desktop Session Host role service on a computer running Windows Server 2012 without the Remote Desktop Connection Broker role service

“”RD Session Host without the Connection Broker

To be clear about this deployment:

  • No Domain Controller
  • No Hyper-V
  • Install Remote Desktop Services.
  • You won’t get to select more until the end. You may install the following
      1. Remote Desktop Licensing and
      2. Remote Desktop Session Host
      3. RDWEB
      4. RD Gateway
  • For a workgroup you do not have to add the Terminal Server to the Terminal Server License server group. Why? With No DC, you don’t have to report to anyone.
  • Activate the Terminal Server using one of the three methods.. This is most common
  • Activate your RD Cal Licenses with KB2833839 or cc725890

So Far, so good. Now this is where we start to diverge from some existing Documentation

  • If you are in a workgroup, go to “edit local users and groups”
  • Find the group folder and create a group for your RDP users and add your users to this group.
  • Alternatively, you may add your users to the RDP users group already there
  • Remember the group you are using. It becomes important
  • Now you are going to edit the Local Policy by doing the following:
      1. Start and Run GPEDIT.MSC
      2. Navigate to the following:

Local Computer Policy ->Computer Configuration-> Administrative Templates-> Windows Components-> Remote Desktop Services-> Remote Desktop Session Host-> Licensing

Figure 1. GPEDIT.MSCUntitled

You are going to see now, the two LSO (Local Security Object) you will be enabling.

Use the Specified Remote Desktop License Servers –Value- IP address of RD License Server.

Set the Remote Desktop Licensing Mode- Value – 2 or 4. 2 is for Device CAL and 4 is user Cal.

Figure 2. Local Policy. Untitled

Now there is another Policy to set. For this, you want to just go back to the top. Start out at Local Security policy like before (GPEDIT.MSC).See figure 3.

 

Figure 3 GPEDIT.MSCimage

 

Expand Computer –>Configuration,–> expand Windows Settings, –> expand Security Settings,–> expand Local Policies and then click User Rights Assignment.

image

 

image

Enable this policy and add the group you used earlier. It is highlighted in this article above. Add this group to this policy. In addition, add the Remote Desktop Users group to this policy if desired. Don’t and your administrator name here. The Admin already has access. If you ad your admin name, it will lock you out. So best to to stick to adding the Remote Desktop users group.

image

 

Notice it says administrators (Plural), that is fine, but the single administrator should not be in this list. there is a well known break here if you do that.  When you are finished, you just need to add the users you want to give access to, to the group we just added to the Policy (likely the remote desktop users group)

Step 8 comes right out of the KB2833839

  • Open an elevated Windows PowerShell prompt
  • Type the following command on the PS prompt and press Enter:
    $obj = gwmi -namespace “Root/CIMV2/TerminalServices” Win32_TerminalServiceSetting
  • Run the following command to set the licensing mode:
    Note: Value = 2 for Per device, Value = 4 for Per User
    $obj.ChangeMode(value)
  • Run the following command to replace the machine name with License Server:
    $obj.SetSpecifiedLicenseServerList(“LicServer”)
  • Run the following command to verify the settings that are configured using above mentioned steps:
    $obj.GetSpecifiedLicenseServerList()
    You should see the server name in the output.

You have now covered all your bases, and  your RDP should be happy!! It will be happy because you paid attention to all the rights things!

Now I did find an interesting article to which I cant really comment on. However, it is an interesting article. IT deals with some issues, you could run across.

Well That is it. I hope this has been helpful for workgroups or non-work groups. this basically  can be set up on either.

Getting Accurate Latency from Dynamically expanding Hyper-V virtual Machine Disks

 

This Article is about the tool called Hyper-V Performance Monitor Tool (PowerShell)

you can download it from the tech net article down the page, or use the link above.

So Hyper-V gets thrown around loosely these days, when you talk about Virtualization or Performance Tuning, or Planning or any other aspect of the product life-cycle of a new Host Deployment.

Over the last few years, we have made rapid changes from Physical Host Machines for production work loads, to these Virtual monstrosities, that now host our whole company.

Along with this change, you may recall that Early Hyper-V documentation has gently let us know that monitoring inside the Virtual Machine was not going to give us the parity results to the Physical counters, depending on configuration.  This is for a few reasons, which are beyond this articles scope. However, I would like to shine light on this, so more people can think differently about their Virtual performance.

The most common measure of how well a server is performing is Latency in Milliseconds. Every one is most concerned with how much latency is in the storage system. Perhaps with good reason. The San storage vendors can perform so fast now days, that you can throw the empire state building at a server, and the Latency is less then 10 Milliseconds (MS). Or is it throughput?

To be clear, we are interested more in Latency then throughput. Latency should be minimized, and throughput will generally increase.

Can I make a case that Counters are not reliable?

Well let me tell your actual Latency is not as easily attained as you would think. If I told you that your tests are lying to you, would you believe me? Lets say your in shock. Without knowing your design, my generalized answer has a very high chance of being correct.  The issue is that you have a SAN and you are tying to get latency by measuring responses to a file, that go though 4 different filters, and then has to wait until it gets queued to a disk subsystem, that is always expanding. I promise you; your numbers are incorrect.

If you cannot quantify how latency and throughput are different, but related, then I would say you should not stop reading.

Storage Latency of VM guests

There are many problems with calculating storage latency, but disk is the model we are going to use to illustrate how tricky it can be to find out how your VM is performing.

The most common approach to gaining latency information is to use a command line tool. Normally the tool will work fine. The model breaks down when the Disk itself is changing, along with the Ram and processor availability. The bottom line is a Virtual machine may lie to you about resource numbers at any given time.  Add to the mix, that the clock cycle is a weakness in any virtualization platform. That means that the calculation of time itself can result In poor results based on good math with bad numbers.

There are a crowd of you who say that is bull. Well all I can say is; don’t read this and good luck solving your latency issues.

Let me try to list some areas where the numbers may go awry. I am just making a one line explanation with a link, so you can read more. I don’t want this to be about the problems. Below, I talk more about the solution. Read more if you have a specific issue:

 

I could keep going. Do you get the feeling there are a ton of variables that change how storage latencies should be calculated?

From my experience, I have found that every set of servers are their own data set of network behavior. There are some basic assumptions I found to pass along to Admins who want to find out latency of Virtual machines.

Guidelines for VM latency Study

Who to Blame

So again, the basic message is that the calculation of latencies is totally based on the sum of the deployment factors. In one data center you may find under reporting, and the other Over Reporting. Support agents do not have the Onus to prove why one is slower then the other. We will have to look at your Design and deploy and try to make a story of things we can identify. it is not likely we will find that moment where the Deployment deviated from your Baseline storage latency measurements. We offer Best effort, but encourage you to strip down your deployment to make a core Baseline latency for a Dynamically expanding VM. All Vms will compare to that one. We go from there.

Using the Stop Gap solutions for Monitoring Virtual Machines

SO it was just a few years ago, this issue with VM monitoring was not easily remedied. You could certainly use the Perfmon counters to get VM stats. But Customers just want to run Disk Speed or SQLIO, and get an output to look at. This did not exist for quite some time. Thankfully there is a script out there, that will now carve out some parity to those tools. the link is at the TechNet Gallery:

Gallery.TechNet.Com

Hyper-V Performance Monitor Tool (PowerShell)

Below is the walk through of the basic performance collection.

you Just run the Script from an Admin PowerShell. There are a few ways to run it:

.\Monitor-HyperVGuestPerformance.ps1

### export data to csv via GUI, defaults to current dir
.\Monitor-HyperVGuestPerformance.ps1 -ExportToCsv

### retrieve data as PSobjects, great for parsing and logging, -name parameter is optional, defaults to automatic discovery
.\Monitor-HyperVGuestPerformance.ps1 -PSobjects

### specify host and interval/samples manually
.\Monitor-HyperVGuestPerformance.ps1  -Name host1,host2 -PSobjects -Interval 2 -MaxSamples 5
### accepts pipeline input
‘Host1′,’Host2’
| .\Monitor-HyperVGuestPerformance.ps1 -PSobjects

### Log to SQL server with Write-ObjectToSQL , this example uses SQL auth
.\Monitor-HyperVGuestPerformance.ps1 –PSobjects  |  Write-ObjectToSQL –TableName table –Database db -Server server –credential (get-credential)

 

image

If the domain connection fails, it tries for a Local connection:

 

image

In my case, I ran the tool on the Host, and this GUI below popped up. All I did was hit monitor, and I got an export vm_perfmon_stats file. This file can be used to find your latency.

Untitled

While this method may not be pretty, It does follow the rules for Hyper-V guest. The main purpose for this tool would be to use instead of SQLIO or DISK SPEED. tools like these should be used for hardware testing. A Hyper-V Server, running on ISCSI shared storage, with two VHDX files attached, is likely going to come back with Erroneous Latencies. This may not be perfect, but I do believe you will see a consistent result that is not a totally unbelievable number.

See I changed the Sample and interval:

Untitled

And I get a time-frame to wait for the test results:

image

Find the Link at the Microsoft TechNet Center. Thank you for taking the time to Read about Storage Performance for Hyper-V virtual Machines

I hope this helps in your Baseline Studies.

The result is a nice little Excel Display of the data, that I cleaned up a little with colors, to the Excel Fields.

 

image

Louis

How to activate a license server and install CALs

Glennopedia

This is the 3rd article in a series of articles on RDS (Remote Desktop Services)  In this installment I will cover how to activate a license server and installing RDS CALs (Client Access Licenses)

To activate the license server:

  1. Open the RD Licensing Manager.  (Right-click Stat button, Run, licmgr.exe)
  2. Expand All servers.
  3. Right-click the server name and choose Activate Server.
    Activate Server1
  4. Click Next on the Welcome screen.
  5. Click Next on the Connection Method screen.
  6. Enter the company information.  This should match the information given when the RDS CALs were purchased.  Click Next when complete.
  7. This screen is optional.  You can enter the information if you choose.  Click Next when complete.
  8. The Completing the Activate Server Wizard screen should come up and indicate the server was activated successfully.  If the CALs are available to install, Click Next.  Otherwise uncheck the “Start Install Licenses Wizard now” option and click Finish.

To Install RDS…

View original post 139 more words

For That Once in a Blue Moon when I Need That Program Called RoboCopy!

Hello everyone,

 

Untitled

Happy Friday! Let me start by saying I know many of you may not feel the need to keep this article. However, In that one Blue Moon, I guarantee you will need the information, because Robocopy has some odd scenarios.

I only call light to the thing we do with Robocopy: copy files. There are two scenarios for this. The target existed previously, or the target did not. Expressed another way, you are copying files or you are syncing files. It is a fine line indeed. So I tried to supply the basic commands below. If you are creating the target, then run the Create Target commands. If you are Syncing, then run the Sync Target Commands.

Create Target Commands

I know you likely already know how to use robocopy. But this little application can go wrong, if you make a mistake in your syntax. I decided to just share my basic commands with you. I generally just want a whole drive or a whole folder moved, with thread control.

The scope of my command is narrow, but it gets the job done in the fastest time, without a mistake.

the general form is

robocopy (source)(destination)(controls)

Here are solid examples of copying when the target does not yet exist.

In the above commands, the source is copied to the destination. You may add /SEC to any of these commands. This will copy and keep the security permissions. On a first time copy, the command will perform as expected.

Sync Target Commands
Now I am not getting into the insanity when the target exists. we all have too many things to do. If the Files have been copied before, to the destination, then you have to worry about files that have not changed from source to destination. My vote would be to delete all the target folders and recopy. If that does not work for you, then read on.

If you have an existing source and old target, meaning you are really syncing data, and not copying, there is another set of commands I will lay on you:

  • ROBOCOPY /Mir <Source> <Target>
  • ROBOCOPY /E /Copy:S /IS /IT <Source> <Target>

You have to run both commands! This is a work around, but it works. If you want to spend the hours of spinning up on robocopy, go ahead. I plan to just keep this article, with the two basic sets of commands, to get the work done.
I sourced my information from the following blog, which you can read and understand if you need detail:

Robocopy Mirror Switch Mirroring File Permissions

I hope this is helpful. I know we don’t do this too often, but I wanted to make sure I pointed to the Correct Article for this issue. This is likely going to remain a work around, based on the behavior of the command.

 

Have a great Early Spring weekend!

 

Louis Reeves

Switching Hyper-v CSV Clustering from Software teaming to Hardware isolation.

    I wrote this article for several reasons. There are a lot of reasons why you may want to go from a Software team to Hardware, or Vice Versa. I make no comments as to why you would do either.

    This is to say, There are some circumstances where you are better served with a Software team, and some where you will see better results with hardware and the NDIS driver.

    I will leave the why up to you. However, If you have the need, here is a rough guide to get you though the conversion of a Software Teamed 2012 or 2016 Cluster conversion of your network, To hardware NICS and Isolation.

    This article is high on text an low on screen shots. All I can do is apologize in advance.

    Some Additional titles:

    Converting the Windows Cluster from Software Teaming to Best Practices, Using Hardware Isolation.

    Another title to this article could be switching from Software teaming to Hardware isolation for CSV clusters.

    My software teamed cluster performs lousy.

    Two Standards in two Documents

    Why would you perform this change? You would do this if your using 1GB NICS for your cluster, and software teaming is not working well for you.

    I have found that the network changes around 2012 may be better suited to 10GB NICS then 1GB NICS. I find the 1GB NICS aggregation may not always be the best way to setup a cluster, depending on the needs of the customer, and the type of software running.

    I find that the original Network Best practices suit some customers from a maintenance perspective, as well as ease of understanding, when the IT persons are not full time network admins.

    This article demands that your Storage Networks are on a separate switch from the Clustering Networks.

    Furthermore, The Cluster Networks should be ideally isolated with Vlans, by Subnet, to prevent any collisions of packets from a unique subnet. If the Network is flat, That may not stop you from moving to this setup. Just know that some isolation is recommended. But, using a separate Subnet per Virtual network is still a must.

    The Design of your network should be based on the Initial Configuration Guide for Hyper-V networks for a CSV cluster (2010).

    If you do a Bing search for “network guide for Hyper-V clustering” , you will find this article,

    Hyper-V: Live Migration Network Configuration Guide (2010)

    And Bing “cluster guide for Hyper-V CSV cluster networks” and you should find:

    Network Recommendations for a Hyper-V Cluster in Windows Server 2012 (2013)

    The 2010 Documents lay out the Premise of Windows Clustering, with the goal of having CSV Live Migrations optimized. The 2013 Documents, Shows the embedded method of isolating the traffic within the Teamed interfaces.

    The Opinion Section

    Having been an Engineer lots of Hyper-V Cluster cases, the main reason for CSV cluster issues, was some form of mis-configuring the network, as per the best practices.

    In 2012, we have now added the additional choices of using a software Microsoft Team and or LBFO options using scripting to send the traffic down specific virtual network adapters.

    The change in server 2012 and 2016 has changed the perception of what can be done with the new technology. I now see the majority of Hyper-V networks using Microsoft teaming. Which is fine. Its fine, as long as you are using the virtual LANS; creating the isolation, along with the team. So the whole message of server 2012 and 2016 is Teaming and Isolation. What I am seeing all too often is just the teaming portion.

    For this reason, I am bringing back the original specifications, to make the statement, that Isolation is the Key element to a CSV cluster. Teaming can be fine, But Proper Isolation comes first.

    This means that if you are not a network person, and don’t have access to a network department to set things up for you, then you may want to stick with the original guidelines, which I am about to present to you. In this article, we are going to convert a Cluster from a Microsoft Teamed network, to a Vanilla Hardware Isolated, Cluster Network.

    The assumption here is that all of your Network adapters have been thrown into a team or two. In addition, No VLANS were created to isolate your traffic. If there were VLANS created, you simply use power-shell to delete them. See the following script, changing the commands new-LbfoTeam, to remove-LBFOteam and etc…

    Configuring networks 2/5

    Words of Caution

    You will start On a node, that is not being used by the cluster. We will make the changes to one node, at a time. When the first node is done, move to the next. 

    Please understand, This is a major change. You will likely have to take an outage to get this done across the whole cluster. I would make the changes on the first node (Node 1). Then shut all cluster nodes down, bringing up the node you converted. This will now be the primary cluster node, and as you update each additional node, they will come online, into the cluster represented by your Node 1 

     How to change you Cluster, One Node at a time

    So here are the steps:

    Pre Configuration

    1. Run Ipconfig /all > c:\ipconfig.txt on all cluster nodes.
    2. Pull the MACS on Software Teamed NICS in the Server Manager->NIC Teaming-> Teams 
    3. Save these files, even placing them in a spread, so you can match up the NICS you will be using. You need to make sure that you understand which NIC on each server will play each of the roles VMNIC, Live Migrate, CSV, Management, Storage, and Replication
    4. The columns of your spread are NIC name, NIC Description, MAC Address, OLD IP Address, New IP address, Cluster Workload and VLAN If Necessary
    5. Remember not to record NICS that have parenthesis around them or are multi-plex adapters. These will be going away. We will only rely on physical adapters.
    6. The Rows of your Spread are as follows:
      1. NODE X
        1. VMNIC (ex. no Subnet)
        2. LIVE MIGRATE (ex. 10.10.11.x)
        3. CSV  (ex. 10.10.12.x)
        4. HOST  (ex. 10.10.13.x)
        5. Replication  (ex. 10.10.14.x)
  • So your Subnets need to be assigned by #6 above. All VMNICS wont have an IP because they will all be dedicated to the Hyper-V switch. The easiest way to express this is to explain that Every Live Migrate will be on the 10.10.11 network. Every CSV will be on the 10.10.12 network. Every host will be on the 10.10.13 network. Every Replication will be on the 10.10.14 network.  Every host will have a dedicated NIC for each of these networks.
  • Replication above would be Ideal. If you don’t have enough NICS to have this network, then its ok. This NIC may also be called a Storage NIC. IF so, this NIC will Have MS Client sharing disabled, and uncheck register in DNS for this NIC.
  • the Cluster workload on your spread can be populated from the section below called workloads

Hopefully, these tidbits have helped you prepare for the actual work, because here we go. This will need to happen on every cluster Node.

Configuration

    1. Go into every VM and change the NIC to not configured
    2. Remove all Network adapters from the Software teams.
    3. Delete the Virtual Hyper-v Network – if external
    4. Verify that all virtual NICS are gone. Restart server if they are not.
    5. Label 5 (or more) NICS as follows: (minimum 4)
      1. VMNIC
      2. LIVE MIGRATE
      3. CSV
      4. Management
      5. Replication
      6. Storage1, 2 etc.
    6. Go to Programs and Features and right click the Network adapter management application.
      1. Select the “Change” option
      2. Add NDIS if Broadcom, or whatever the 4 letter acronym is for advanced service of the NIC brand installer.
    7. Place an IP, Subnet, DNS and Gateway into the Management NIC.
    8. Place only an IP and subnet, for the Storage NICS. Disable for Microsoft Networks, and Uncheck Register in DNS
    9. Set Ip and subnet for Live Migrate, CSV, and replication
    10. Re-create the Hyper-v Network using the VMNIC. It will be external and dedicated to the VMS. (check the box)
    11. For 1GB nics – go to every NIC in device manager and disable for VMQing. If 10GB nics, leave enabled

    That’s it. Once you have done the second machine, make sure you can ping across to the other NIC on host1.

    As long as you pass the ping test, you may move to changing the other Nodes in the cluster.

    Once you have completed the changes, and all machines ping properly, you may restart the nodes in a round robin style if you have any kinks

    The only thing left to do, Is to set your priorities for the networks. The priorities are in the center pane of cluster manager. Choose network in the left selection plane.

    Workloads

    The settings should be changed to the following:

    1. VMNIC – NONE (dedicated to VMS)
    2. LIVE MIGRATE – Cluster Communication only
    3. CSV – Cluster Communication only
    4. Management- client only
    5. Replication- Cluster and client
    6. Storage1, 2 etc. – N/A no cluster communication

    And that’s it. Once you have that established, your cluster will now be working on physical NICS only.

    Now with the NICS you have left, you may go back and add selective hardware teaming where it makes sense. No Teaming on Storage NICS. Honestly, it may be better to wait to see if you have any area of weakness. For Example, If the VMs seem slow during intense workdays, you may remove the VMNIC, from Hyper-V, Team it, and then put the multiplexor back in as the VMNIC used for Hyper-V VMs.

    I hope this has been helpful. Please remember that this works automatically, because the Cluster is able to identify NICS on different servers, and automatically matches them to the same subnet. The NICs from the various servers will show up together in the network tab, because they are on the same subnet. As such, the cluster knows that they will be performing a specific job in the cluster, based on the settings laid out in this article.

    This is not better or worse then software teaming, but it may be the right choice for some IT teams, who are more familiar with hardware NIC assignments as part of their Job description.

    Also please understand the days of teaming with 1GB NICS is coming to a close. Very quickly 10GB NICS are becoming the standard. The difference in these setups becomes MOOT when your no longer using 1GB adapters for clustering.

    Thank you,

    Have a nice Day.