Why Network Design is so Important for Hyper-V CSV Clusters. And Some Common Items to Check

I realize I have written several Hyper-V articles lately. They all come for the unique perspective of technical  support. I now see, I have been trying to put together some material to help understand how these new 2016 Clusters are from the 2012 Clusters from the 2008 Clusters.

So I want to express three clear goals in this paper. I want to define a list of items you may want to read, to make your Cluster as supportable as possible. Second, I simply want to speak to the Importance of your NIC hardware purchases, as it relates to the past and current Stance of Microsoft, with respect to Network requirements, of  Hyper-V cluster setups. Finally, I just summarize some of the command line setting you may look at if you don’t have the optimal NIC setup for a Cluster.

 

Disclaimers

 

      • Disclaimer (agent), While there are items in this discussion, that may not be good advice, for your particular Infrastructure, or your particular situation, I am writing this from a familiar perspective, that Admins, and Designers are approaching me from. Namely this is an “I want it all, and I want it all to work now!” type of design steps.
      •  Translation: (customer needs)Give me the settings that make the fullest use of my server, give me the most VMS, with the most possible resources, and I want to Live Migrate them all day long. Furthermore, I want to be able to host conference in an RDP session, on any one of my Cluster Nodes, and not have any problems.

 

Facts and Common issues in today’s and yesterday’s Hyper-V CSV Clusters

 

With this disclaimer in mind let us proceed. First things first. I am providing this list based on my 10 years with Hyper-V and Clustering, along with the reading and video information I have come across.:

 

      1. Never Use the (Hyper-V) shared Network Adapter as a NIC in the Host Server.
      2. Never Software Team with the NDIS driver Installed from your NIC vendor.
      3. Software teaming is fine for most workloads, unless you’re having latency problems. The answer is Hardware Teaming, or Vice Versa
      4. Don’t put SQL servers on your Hyper-V host.
      5. VM QUEUING can be a problem. Try your workloads with and without VM Queuing and see which works best for your situation.
      6. TCP Offload is not supported for Server 2012 Cluster Teams. Check the other settings here
      7. The Preferred software Team setting is Hyper-V PORT virtualized, switch Independent teaming is best. This is where we are at today. Remember, you had access to these statements in current documentation.
      8. If you Use the Multiplexer Driver as the Virtual Machine NIC, do not turn around and share that NIC with the Hyper-V Host. This is not pretty.
      9. Use Jumbo Frames, QOS etc., where you’re supposed to, according to the current guidelines
      10. Piggy Back off of #8 is that today’s Clusters with Hyper-V, are a balance of Isolation and Bandwidth networks. There is no hard fast rule on how many Network adapters you need.
      11. You cannot just say a node is too slow or fast. When you first install the server, you need to perform clearly laid out, baseline testing, where similar results can be obtained for your server, in pristine condition, with no other workloads. The Same is true for Virtual Machines.
      12. You cannot run a Hyper-V CSV cluster, with all your NICS in the TEAM. You need at least 3 networks in any version of Hyper-V. This is
        1. Cluster Only (Cluster Communication)
        2. Cluster and Client (Management etc.…)
        3. No Cluster Communication. (ISCSI)
      13. Run Cluster Validation- If your updates don’t match across your cluster, you need to get all your nodes to match, before the cluster will work properly.
      14. Clustering only recognizes one NIC per sub-net, when you add multiple NICS to cluster
      15. Back up applications and Antivirus, may have compatibility issues, disable both, and see if the issues disappear.
      16. Network Considerations
      • The Binding Order and DNS must match your current MS documentation. DO not miss this.
      • Cluster Setup now adds rules to the firewall automatically. If you are using Symantec Endpoint. These Firewall rules can serve as your port list to add to Symantec firewall.
      • You can Now Sysprep with Cluster Role installed now for Server 2012.
      • Your NETFT is enabled at the physical NIC, Where you find your IPV4 properties.  Do not Disable it.
      •  So if you are setting up converged clusters, you now have to rely on cluster validation to tell you, if you have enough networks to effectively set up your cluster. Resolve any of the network issues here in validation
      • CSV traffic includes Metadata Updates and Live migration data, as well as failure recovery (IE no storage connectivity) you cannot break this traffic into isolated streams.
      • CSV needs NTLM and SMB- don’t disable either.
      • ISCI Teams now work with MPIO and Jumbo Frames. Jumbo Needed for ISCSI
      • Using multiple Nic Brands Is now preferred.
      1. This series of Articles covers topics I did not go into In Depth. Topics Include:
        1. Mapping the OSI model
        2. VLANS
        3. IP routing
        4. Link Aggregation and Teaming
        5. DNS
        6. Ports, Sockets and Applications
        7. Bindings
        8. Load Balancing, Algorithms

 

 

Cluster nodes are multi-homed systems.  Network priority affects DNS Client for outbound network connectivity.  Network adapters used for client communication should be at the top in the binding order.  Non-routed networks can be placed at lower priority.  In Windows Server 2012/2012R2, the Cluster Network Driver (NETFT.SYS) adapter is automatically placed at the bottom in the binding order list

 

Network Evolution and common sense Network needs

This section is really addressing how we build clusters today. For example, See recently how I wrote a paper on using the old Isolation rules for a simple 2016 cluster, based off the old method of deployment. This method is elegant and works well, will little maintenance needed.

For 2012, and forward, we have the new design which is detailed in the Tech net article, “Network Recommendations for a Hyper-V Cluster in Windows Server 2012”. In this paper, it  Includes the modern setup, using the a software team, and scripted Network Isolation

This paper interleaves these two philosophies, at least that was the intent or message, you are always using one or the other as a guiding principle. Insofar as you have the technical reasoning to do so. what I mean is, if you have 10GB nics, you may fully move to the 2012 method. If you have like 3 1GB nics, you are leaning on the 2008 article to explain to the customer why live migration would not work properly.

Get logging information for Hyper-V and clustering from this article

The quick history of the CSV cluster as follows:

2008

Heartbeats/Intra Cluster Communications -in some documentation  (1GB)

CSV I/O Redirection  (1GB)

VM Network (1GB)

Cluster Network (1GB)

Management Network (1GB)

ISCSI Network – (1GB)

 

2012 and 2016

HeartbeatsNetwork Health monitoring in some documentation (QOS IMPORTANT)  (10GB)

Intra Cluster Communications (QOS IMPORTANT)  (10GB)

CSV I/O Redirection (Bandwidth Important)  (10GB)

ISCSI Network – Not registered in DNS (10GB)

 

This is where you can clearly see how new clusters in 2016, just don’t have the same specifications. The recommendation here is to adjust the Cluster Networks, by the number of network adapters, and what the throughput is. If the NIC setup looks like the 2008 cluster, then apply 2008 network setup guidelines. If the cluster has 2 or more 10GB nics, then treat it as the newer 2016 logic. This has worked well for me for some time now. This will ensure that you get the best Isolation and throughput for your customer.

So as you can see, the Number of NICS is going down, but the NIC SPEED is going up. To make matters more difficult, Microsoft Now states that to be optimized, a CSV cluster will have a combination of Isolation and Bandwidth. They are no longer able to lean on the hard 5 to 7 NIC requirement that once was the norm. For proof of this, you will need to watch this video entitled. ” Fail over Cluster Networking essentials. ”

So really, Support may not be giving you a great explanation as to why your CSV cluster is slow. It is really closely related to the Network Design. Does your Network look more like a 2008 cluster, or a 2012 or 16 cluster? This will give you justification as to why a cluster would be slow or fast.

Server 2012 Requirements are here, along with a basic script for Embedded Teams

In addition to the script above, you also have control over the heartbeat, and other things like priority of the various Cluster NICS and timeouts.

 

Settings you may look at to change if needed

The rest of this article, just shows you some config settings, if you find you have to make a manual change. With a 2016 cluster, they are saying its all automatic, and should not be changed.

While you can make changes to the following. The recommendation is to leave the settings alone. The automatic settings should adjust to the proper, situational network changes:

 

Configure Cluster Heart Beating

 

(Get-Cluster). SameSubnetDelay = 2

 

The Above command Is an example of how you set the following variables. They are posted below with their Default values

 

  • SameSubnetDelay (1 Second)
  • SameSubnetThreshold (5 heartbeats)
  • CrossSubnetDelay  (1 Second)
  • CrossSubnetThreshold (5 heartbeats)

The above setting Is for regular clustering. For Hyper-V clustering, consider the following defaults

  • SameSubnetThreshold (10 heartbeats)
  • CrossSubnetThreshold (20 heartbeats)

If you go more than 10 to 20 on these two settings, the documentations says the overhead starts to interfere, more than the benefit. FYI.

This Step Below is only for allowing the creation of the cluster on a Slow network. Set the value of SetHeartbeatThresholdOnClusterCreate  to 10, for a value of 10 seconds.

HKLM\SYSTEM\CurrentControlSet\Services\ClusSvc\Parameters
add DWORD value SetHeartbeatThresholdOnClusterCreate

 

Configure Full Mesh HeartBeat

(Get-Cluster). PlumbAllCrossSubnetRoutes = 1

 

Other Important changes to change Cluster Setup Parameters

Please Be advised, All the following syntax, has been duplicated from this publicly available Microsoft Article:

Change Cluster Network Roles ( 0=no cluster, 1=Only cluster communication, 3=Client and Cluster Communication)

  • (Get-ClusterNetwork “Cluster Network 1”). Role =3
  • Get-ClusterNetwork | ft Name, Metric, AutoMetric, Role
  • ( Get-ClusterNetwork “Cluster Network 1” ).Metric = 900
  • ( Get-ClusterNetwork “Cluster Network 1” ).AutoMetric = $true

Set Quality of Service Policies (values 0-6) ( Must be enabled on all the nodes in the cluster and the physical network switch)

  • New-NetQosPolicy “Cluster”-Cluster –Priority 6
  • New-NetQosPolicy “SMB” –SMB –Priority 5
  • New-NetQosPolicy “Live Migration” –LiveMigration –Priority 3

 

Set Bandwidth policy (relative minimum bandwidth policy) (It is recommended to configure Relative Minimum Bandwidth SMB policy on CSV deployments)

  • New-NetQosPolicy “Cluster” –Cluster –MinBandwidthWeightAction 30
  • New-NetQosPolicy “Live Migration” –LiveMigration –MinBandwidthWeightAction 20
  • New-NetQosPolicy “SMB” –SMB –MinBandwidthWeightAction 50

 

If you need to add a Hyper-V replica

  • Add-VMNetworkAdapter –ManagementOS –Name “Replica” –SwitchName “TeamSwitch”
    Set-VMNetworkAdapterVlan -ManagementOS -VMNetworkAdapterName “Replica” –Access –VlanId 17
    Set-VMNetworkAdapter -ManagementOS -Name “Replica” -VmqWeight 80 -MinimumBandwidthWeight 10
    # If the host is clustered – configure the cluster name and role
    * (Get-ClusterNetwork | Where-Object {$_.Address -eq “10.0.17.0”}).Name = “Replica”
    *(Get-ClusterNetwork -Name “Replica”).Role = 3

From <https://technet.microsoft.com/en-us/library/dn550728(v=ws.11).aspx>

Configure Live Migration Network

  • # Configure the live migration network
    Get-ClusterResourceType -Name “Virtual Machine” | Set-ClusterParameter -Name MigrationExcludeNetworks -Value ([String]::Join(“;”,(Get-ClusterNetwork | Where-Object {$_.Name -ne “Migration_Network”}).ID))

From <https://technet.microsoft.com/en-us/library/dn550728(v=ws.11).aspx>

 

Other Commands

  • Enable VM team Set-VMNetworkAdapter -VMName <VMname> -AllowTeaming On
  • Restrict SMB – New-SmbMultichannelConstraint -ServerName “FileServer1” -InterfaceAlias “SMB1”, “SMB2”, “SMB3”, “SMB4”

 

 

Getting Accurate Latency from Dynamically expanding Hyper-V virtual Machine Disks

 

This Article is about the tool called Hyper-V Performance Monitor Tool (PowerShell)

you can download it from the tech net article down the page, or use the link above.

So Hyper-V gets thrown around loosely these days, when you talk about Virtualization or Performance Tuning, or Planning or any other aspect of the product life-cycle of a new Host Deployment.

Over the last few years, we have made rapid changes from Physical Host Machines for production work loads, to these Virtual monstrosities, that now host our whole company.

Along with this change, you may recall that Early Hyper-V documentation has gently let us know that monitoring inside the Virtual Machine was not going to give us the parity results to the Physical counters, depending on configuration.  This is for a few reasons, which are beyond this articles scope. However, I would like to shine light on this, so more people can think differently about their Virtual performance.

The most common measure of how well a server is performing is Latency in Milliseconds. Every one is most concerned with how much latency is in the storage system. Perhaps with good reason. The San storage vendors can perform so fast now days, that you can throw the empire state building at a server, and the Latency is less then 10 Milliseconds (MS). Or is it throughput?

To be clear, we are interested more in Latency then throughput. Latency should be minimized, and throughput will generally increase.

Can I make a case that Counters are not reliable?

Well let me tell your actual Latency is not as easily attained as you would think. If I told you that your tests are lying to you, would you believe me? Lets say your in shock. Without knowing your design, my generalized answer has a very high chance of being correct.  The issue is that you have a SAN and you are tying to get latency by measuring responses to a file, that go though 4 different filters, and then has to wait until it gets queued to a disk subsystem, that is always expanding. I promise you; your numbers are incorrect.

If you cannot quantify how latency and throughput are different, but related, then I would say you should not stop reading.

Storage Latency of VM guests

There are many problems with calculating storage latency, but disk is the model we are going to use to illustrate how tricky it can be to find out how your VM is performing.

The most common approach to gaining latency information is to use a command line tool. Normally the tool will work fine. The model breaks down when the Disk itself is changing, along with the Ram and processor availability. The bottom line is a Virtual machine may lie to you about resource numbers at any given time.  Add to the mix, that the clock cycle is a weakness in any virtualization platform. That means that the calculation of time itself can result In poor results based on good math with bad numbers.

There are a crowd of you who say that is bull. Well all I can say is; don’t read this and good luck solving your latency issues.

Let me try to list some areas where the numbers may go awry. I am just making a one line explanation with a link, so you can read more. I don’t want this to be about the problems. Below, I talk more about the solution. Read more if you have a specific issue:

 

I could keep going. Do you get the feeling there are a ton of variables that change how storage latencies should be calculated?

From my experience, I have found that every set of servers are their own data set of network behavior. There are some basic assumptions I found to pass along to Admins who want to find out latency of Virtual machines.

Guidelines for VM latency Study

Who to Blame

So again, the basic message is that the calculation of latencies is totally based on the sum of the deployment factors. In one data center you may find under reporting, and the other Over Reporting. Support agents do not have the Onus to prove why one is slower then the other. We will have to look at your Design and deploy and try to make a story of things we can identify. it is not likely we will find that moment where the Deployment deviated from your Baseline storage latency measurements. We offer Best effort, but encourage you to strip down your deployment to make a core Baseline latency for a Dynamically expanding VM. All Vms will compare to that one. We go from there.

Using the Stop Gap solutions for Monitoring Virtual Machines

SO it was just a few years ago, this issue with VM monitoring was not easily remedied. You could certainly use the Perfmon counters to get VM stats. But Customers just want to run Disk Speed or SQLIO, and get an output to look at. This did not exist for quite some time. Thankfully there is a script out there, that will now carve out some parity to those tools. the link is at the TechNet Gallery:

Gallery.TechNet.Com

Hyper-V Performance Monitor Tool (PowerShell)

Below is the walk through of the basic performance collection.

you Just run the Script from an Admin PowerShell. There are a few ways to run it:

.\Monitor-HyperVGuestPerformance.ps1

### export data to csv via GUI, defaults to current dir
.\Monitor-HyperVGuestPerformance.ps1 -ExportToCsv

### retrieve data as PSobjects, great for parsing and logging, -name parameter is optional, defaults to automatic discovery
.\Monitor-HyperVGuestPerformance.ps1 -PSobjects

### specify host and interval/samples manually
.\Monitor-HyperVGuestPerformance.ps1  -Name host1,host2 -PSobjects -Interval 2 -MaxSamples 5
### accepts pipeline input
‘Host1′,’Host2’
| .\Monitor-HyperVGuestPerformance.ps1 -PSobjects

### Log to SQL server with Write-ObjectToSQL , this example uses SQL auth
.\Monitor-HyperVGuestPerformance.ps1 –PSobjects  |  Write-ObjectToSQL –TableName table –Database db -Server server –credential (get-credential)

 

image

If the domain connection fails, it tries for a Local connection:

 

image

In my case, I ran the tool on the Host, and this GUI below popped up. All I did was hit monitor, and I got an export vm_perfmon_stats file. This file can be used to find your latency.

Untitled

While this method may not be pretty, It does follow the rules for Hyper-V guest. The main purpose for this tool would be to use instead of SQLIO or DISK SPEED. tools like these should be used for hardware testing. A Hyper-V Server, running on ISCSI shared storage, with two VHDX files attached, is likely going to come back with Erroneous Latencies. This may not be perfect, but I do believe you will see a consistent result that is not a totally unbelievable number.

See I changed the Sample and interval:

Untitled

And I get a time-frame to wait for the test results:

image

Find the Link at the Microsoft TechNet Center. Thank you for taking the time to Read about Storage Performance for Hyper-V virtual Machines

I hope this helps in your Baseline Studies.

The result is a nice little Excel Display of the data, that I cleaned up a little with colors, to the Excel Fields.

 

image

Louis

Switching Hyper-v CSV Clustering from Software teaming to Hardware isolation.

    I wrote this article for several reasons. There are a lot of reasons why you may want to go from a Software team to Hardware, or Vice Versa. I make no comments as to why you would do either.

    This is to say, There are some circumstances where you are better served with a Software team, and some where you will see better results with hardware and the NDIS driver.

    I will leave the why up to you. However, If you have the need, here is a rough guide to get you though the conversion of a Software Teamed 2012 or 2016 Cluster conversion of your network, To hardware NICS and Isolation.

    This article is high on text an low on screen shots. All I can do is apologize in advance.

    Some Additional titles:

    Converting the Windows Cluster from Software Teaming to Best Practices, Using Hardware Isolation.

    Another title to this article could be switching from Software teaming to Hardware isolation for CSV clusters.

    My software teamed cluster performs lousy.

    Two Standards in two Documents

    Why would you perform this change? You would do this if your using 1GB NICS for your cluster, and software teaming is not working well for you.

    I have found that the network changes around 2012 may be better suited to 10GB NICS then 1GB NICS. I find the 1GB NICS aggregation may not always be the best way to setup a cluster, depending on the needs of the customer, and the type of software running.

    I find that the original Network Best practices suit some customers from a maintenance perspective, as well as ease of understanding, when the IT persons are not full time network admins.

    This article demands that your Storage Networks are on a separate switch from the Clustering Networks.

    Furthermore, The Cluster Networks should be ideally isolated with Vlans, by Subnet, to prevent any collisions of packets from a unique subnet. If the Network is flat, That may not stop you from moving to this setup. Just know that some isolation is recommended. But, using a separate Subnet per Virtual network is still a must.

    The Design of your network should be based on the Initial Configuration Guide for Hyper-V networks for a CSV cluster (2010).

    If you do a Bing search for “network guide for Hyper-V clustering” , you will find this article,

    Hyper-V: Live Migration Network Configuration Guide (2010)

    And Bing “cluster guide for Hyper-V CSV cluster networks” and you should find:

    Network Recommendations for a Hyper-V Cluster in Windows Server 2012 (2013)

    The 2010 Documents lay out the Premise of Windows Clustering, with the goal of having CSV Live Migrations optimized. The 2013 Documents, Shows the embedded method of isolating the traffic within the Teamed interfaces.

    The Opinion Section

    Having been an Engineer lots of Hyper-V Cluster cases, the main reason for CSV cluster issues, was some form of mis-configuring the network, as per the best practices.

    In 2012, we have now added the additional choices of using a software Microsoft Team and or LBFO options using scripting to send the traffic down specific virtual network adapters.

    The change in server 2012 and 2016 has changed the perception of what can be done with the new technology. I now see the majority of Hyper-V networks using Microsoft teaming. Which is fine. Its fine, as long as you are using the virtual LANS; creating the isolation, along with the team. So the whole message of server 2012 and 2016 is Teaming and Isolation. What I am seeing all too often is just the teaming portion.

    For this reason, I am bringing back the original specifications, to make the statement, that Isolation is the Key element to a CSV cluster. Teaming can be fine, But Proper Isolation comes first.

    This means that if you are not a network person, and don’t have access to a network department to set things up for you, then you may want to stick with the original guidelines, which I am about to present to you. In this article, we are going to convert a Cluster from a Microsoft Teamed network, to a Vanilla Hardware Isolated, Cluster Network.

    The assumption here is that all of your Network adapters have been thrown into a team or two. In addition, No VLANS were created to isolate your traffic. If there were VLANS created, you simply use power-shell to delete them. See the following script, changing the commands new-LbfoTeam, to remove-LBFOteam and etc…

    Configuring networks 2/5

    Words of Caution

    You will start On a node, that is not being used by the cluster. We will make the changes to one node, at a time. When the first node is done, move to the next. 

    Please understand, This is a major change. You will likely have to take an outage to get this done across the whole cluster. I would make the changes on the first node (Node 1). Then shut all cluster nodes down, bringing up the node you converted. This will now be the primary cluster node, and as you update each additional node, they will come online, into the cluster represented by your Node 1 

     How to change you Cluster, One Node at a time

    So here are the steps:

    Pre Configuration

    1. Run Ipconfig /all > c:\ipconfig.txt on all cluster nodes.
    2. Pull the MACS on Software Teamed NICS in the Server Manager->NIC Teaming-> Teams 
    3. Save these files, even placing them in a spread, so you can match up the NICS you will be using. You need to make sure that you understand which NIC on each server will play each of the roles VMNIC, Live Migrate, CSV, Management, Storage, and Replication
    4. The columns of your spread are NIC name, NIC Description, MAC Address, OLD IP Address, New IP address, Cluster Workload and VLAN If Necessary
    5. Remember not to record NICS that have parenthesis around them or are multi-plex adapters. These will be going away. We will only rely on physical adapters.
    6. The Rows of your Spread are as follows:
      1. NODE X
        1. VMNIC (ex. no Subnet)
        2. LIVE MIGRATE (ex. 10.10.11.x)
        3. CSV  (ex. 10.10.12.x)
        4. HOST  (ex. 10.10.13.x)
        5. Replication  (ex. 10.10.14.x)
  • So your Subnets need to be assigned by #6 above. All VMNICS wont have an IP because they will all be dedicated to the Hyper-V switch. The easiest way to express this is to explain that Every Live Migrate will be on the 10.10.11 network. Every CSV will be on the 10.10.12 network. Every host will be on the 10.10.13 network. Every Replication will be on the 10.10.14 network.  Every host will have a dedicated NIC for each of these networks.
  • Replication above would be Ideal. If you don’t have enough NICS to have this network, then its ok. This NIC may also be called a Storage NIC. IF so, this NIC will Have MS Client sharing disabled, and uncheck register in DNS for this NIC.
  • the Cluster workload on your spread can be populated from the section below called workloads

Hopefully, these tidbits have helped you prepare for the actual work, because here we go. This will need to happen on every cluster Node.

Configuration

    1. Go into every VM and change the NIC to not configured
    2. Remove all Network adapters from the Software teams.
    3. Delete the Virtual Hyper-v Network – if external
    4. Verify that all virtual NICS are gone. Restart server if they are not.
    5. Label 5 (or more) NICS as follows: (minimum 4)
      1. VMNIC
      2. LIVE MIGRATE
      3. CSV
      4. Management
      5. Replication
      6. Storage1, 2 etc.
    6. Go to Programs and Features and right click the Network adapter management application.
      1. Select the “Change” option
      2. Add NDIS if Broadcom, or whatever the 4 letter acronym is for advanced service of the NIC brand installer.
    7. Place an IP, Subnet, DNS and Gateway into the Management NIC.
    8. Place only an IP and subnet, for the Storage NICS. Disable for Microsoft Networks, and Uncheck Register in DNS
    9. Set Ip and subnet for Live Migrate, CSV, and replication
    10. Re-create the Hyper-v Network using the VMNIC. It will be external and dedicated to the VMS. (check the box)
    11. For 1GB nics – go to every NIC in device manager and disable for VMQing. If 10GB nics, leave enabled

    That’s it. Once you have done the second machine, make sure you can ping across to the other NIC on host1.

    As long as you pass the ping test, you may move to changing the other Nodes in the cluster.

    Once you have completed the changes, and all machines ping properly, you may restart the nodes in a round robin style if you have any kinks

    The only thing left to do, Is to set your priorities for the networks. The priorities are in the center pane of cluster manager. Choose network in the left selection plane.

    Workloads

    The settings should be changed to the following:

    1. VMNIC – NONE (dedicated to VMS)
    2. LIVE MIGRATE – Cluster Communication only
    3. CSV – Cluster Communication only
    4. Management- client only
    5. Replication- Cluster and client
    6. Storage1, 2 etc. – N/A no cluster communication

    And that’s it. Once you have that established, your cluster will now be working on physical NICS only.

    Now with the NICS you have left, you may go back and add selective hardware teaming where it makes sense. No Teaming on Storage NICS. Honestly, it may be better to wait to see if you have any area of weakness. For Example, If the VMs seem slow during intense workdays, you may remove the VMNIC, from Hyper-V, Team it, and then put the multiplexor back in as the VMNIC used for Hyper-V VMs.

    I hope this has been helpful. Please remember that this works automatically, because the Cluster is able to identify NICS on different servers, and automatically matches them to the same subnet. The NICs from the various servers will show up together in the network tab, because they are on the same subnet. As such, the cluster knows that they will be performing a specific job in the cluster, based on the settings laid out in this article.

    This is not better or worse then software teaming, but it may be the right choice for some IT teams, who are more familiar with hardware NIC assignments as part of their Job description.

    Also please understand the days of teaming with 1GB NICS is coming to a close. Very quickly 10GB NICS are becoming the standard. The difference in these setups becomes MOOT when your no longer using 1GB adapters for clustering.

    Thank you,

    Have a nice Day.

Update Hyper-V CSV Cluster and Be Successful Doing it

 

Hello Peoples,

Today I write to one of my customers, and I thought this would make a good public confirmation for some Admin out there, who his having a bad Saturday, because his updates are not going so well.

 

Below you will find a basic process for doing NIC updates to a Hyper-V cluster. This is not an MS best Practices article, But in some capacity, It should be. Follow this process to keep your cluster nodes a clean as they can be, with your Networks as fast and updated as they can be.

 

Neuron_Global_Blog_Innovative_Business_Networking_Strategy

Before you begin, Make sure you run Ipconfig /all > mytxt.txt. THis is so you can put the network back together, if needed.

To update the Network Drivers on your Hyper-V Cluster.

  • 1. Take all the Virtual Machines off the Server to be updated.
  • 2. Remove it from the Hyper-V cluster.
  • 3. Remove all Software or Hardware Teams
  • 4. If any VMs are staying on the Cluster node, make sure their NIC is set to not configured, so they are not attached to any virtual network.
  • 5. At this point, you should only have physical NICS in your Device Manager and NCPA.CPL.
  • 6. In device manager, show hidden devices to make sure you don’t have a ghost NIC. If you do, deleted it.
  • 7. Remove the existing Network Driver Package.
  • 8. If Possible, delete the driver from the server.
  • 9. The NICS may auto discover, and they may be assigned an MS DRIVER. That Is ok.
  • 10. The main thing, is they should not be assigned a Broadcom or Intel driver, as those should be deleted.
  • 11. Restart the Server.
  • 12. Apply the new Firmware first, and then Apply the Driver package.
  • 13. Restart the server and begin adding your teams and Virtual networks.
  • 14. Re-add your Server to the cluster.
  • 15. Proceed as normal

If you have any additional trouble

Now notice I did not say you have to delete the virtual networks from Hyper-V. If you do the updates like I have laid out, then you should not have to . But, If you go through the process, and you did not succeed, then you must go Through the same process, but you will have to delete all virtual networks from the console.

Yes this will mean you will need to re-configure the virtual networks, and re-assign the VMs to the new networks you created. You should always be prepared to do this, if you are going to update your server.

Even as Cluster Aware Updating becomes Popular, you should always be prepared to Re-Deploy you Virtual Network.

If you are asking why these things are, My response is You don’t want to know some of the problems that can happen! So let me add to this, in closing, that the upside to doing it this way are many. One good things, is that you may decide to go ahead to reset the Winsock and TCP stack at this time. This will guarantee that your network is clean and that there are no problems with the stack. MS networks are very flexible and dynamic. However, the load and changes we put on them

Require us to make sure we keep things as clean operating as possible.

I hope this message has been informative.

2012 Domain Controller is not a Hyper-V or Cluster Node

Good Morning,

This is not so much of a blog post as a quick note to encourage all to keep a Physical DC.  This was a much needed article to clarify what we know.  Certainly dont  use your DC as a fail over cluster node. Microsoft will no be supporting this in Server 2012.  You probably don’t want your parent partition as the DC in your hyper-v box either. You can virtualize the DC in 2012 but please have a physical DC as well. Happy New year!

http://support.microsoft.com/kb/2795523/en-us?sd=rss&spid=16526

 

L

Using Powershell and Excel to compare Microsoft Hotfixs on Servers and Virtual Machines.

Hello All,

 

Two Types of Comparisons Devised

I have been working several Clustering and Hyper-V cases Recently.  I have come to a point where I notice the Virtual Machines rarely have the same level of  Microsoft Updates that the Cluster Node does. I finally had a few minutes to look at this issue, and I even made an Excel Spreadsheet to compare updates to other machines.

Fortunately, When I got serious about making a Spread Sheet Script, I noticed there were some pretty good work already available. In fact, I found one that will likely serve to compare updates on all the Nodes, all the VMs and more. I am sure I can do cross checks using a Host as the standard, and a bunch of VM’S as the comparisons.

So we are basically showing off two different ways of doing the same thing.  The Gallery script vs. MyExcelHotfixCompare. These are different things for different reasons. But its good to have them in the same place, Because they both have their own purpose.

 

 

First thing you want to do have is a copy of the spread sheet, MYexcelHotfixCompare , and a copy of the The Gallery script

 

Why Two Comparisons

So I made one up, and I found the other by accident! That is true. However, I discovered these both served different purposes.

The First method came from Stane Monsik. What this count does, is aggregate the KB’s for every server in the survey. So your Final report answers the question “when you updated Server Group X with KB Group Summation(A-Z), did you do that same thing to all the other servers? This is what I started out to have answered! If the answer to this is no, then I always tell my customer to finish his updates right away.

However, another question I was often asking, was not being answered by this script. The Questions was, When we know that there are X updates packages for Technology Y, how many of those updates did you apply to your server? Then Question #2,  For X and Y, did you apply these updates to just the server which had problems? The answer to this question should be that you updated all the servers which contained technology Y, with every Update Package X. I found this question was often met with Guesstimates more than actual proof; this was an easy question to get a good guess on. It’s also an easy question to just accept the guess and move on.  In today’s complex world, that is no longer acceptable. So I set out to easily answer this kind of question.

 

Method1 Script

This script will take “a list of Machines” and show you what updates the entire list has and does not have. Every KB gets added to the list, no matter what server it was found on.

The Script, located at Gallery.Tech.net.Microsoft.Com , Goes to every server and pulls the list of hot fixes. Then it goes through the hot fix list and checks every computer, to see if the hot fix has been applied to the computer.

To make GetHotfixCompare_Gallery work, all you have to do is open the .PS1 file with notepad and add your list of computer names. You can add as many computers as you like, but realize it takes longer and longer and longer to complete. The text you are editing looks like this in the file:  See Figure 1. :

# —————————————————————————————————-

$computers = “ComputerName1”, “ComputerName2”, “ComputerName3”, “ComputerName4”, “ComputerName5”

# —————————————————————————————————-

The Result is a very nice HTML report:

 

Figure 1.

1comparetable

 

Once you populate the script with edits on the names of computers, you run the PS1 file by right Clicking the file and choosing Run With Power Shell. The Script runs (longer the more computers you select), then generates an HTML file of your KB’s. There is an orderly chart of stars telling you which KB’s have stars for the machine, and which do not have stars.

 

All in all, A  very good method. But in order for me to answer the technology related questions, I needed to use something different.

 

Method 2 Spread Sheet comparison (MY_excel_GetHotfixCompare)

On the Excel Help article, called “Compare two lists and highlight matches and differences” I use Example 4 of that section. Example 4 runs a comparison from column one (the update list of a server (get-hotfix)  against column two (the list of updates for the specific technology) .  The comparison Colorizes the two list columns to highlight where the updates match or are unique.

MY_ExcelGetHotfixCompare Is the name of the spread, and you just need to right click and save as to save the file as an XLSX file.

This method will always work, as long as you can find a list of key updates, Microsoft says is relevant. This list below is not unabridged. This is just areas where I work most often. Below are some Example Technology Patch lists, that will power column two, on your spread sheets.

Server 2012 and Exchange make you dig a little bit to get the actual KB list, but you get the Idea. The point is to get the list of “important hotfixes” and be able to use them to “audit” you current situation. So let’s get to the formulas and make a spread sheet!

 

Get Hot-fix Compare Spread Sheet

The Compare column Colorizes numbers in column A , that are also in column B. And Vice Versa. So there is only one caveat to this method. If the KB shows up twice in the same column, then the cell becomes colored, denoting that the patch has a match. But, This is an impossibility of the get-hot fix command. As long as your pasting in get-Hotfix results from only one machine per column, you will get a valid result. So the limitation of this, or any tool reviewing hot-fixes; don’t try to do two machines in one spread.

Steps to create:

  1. Open Excel and put your Values for Get-Hotfix (just the KB) into column 1.
  2. Copy the Hot-fix numbers for the relevant Microsoft KB numbers from your articles like above, into Column 2.
  3. With Excel at the Home Tab, Choose “Conditional Formatting” and Manage Rules . You will see a pop up window, New Formatting rule: (figure Below)

 

2spread

 

  1. Choose New Rule and “Format only unique or duplicate values”
  2. There is a little Drop down that says Duplicate or Unique- You will make that choice here

3spread

 

6.Then you see where it says no format set? (above) This is where you choose your color.

7. Hit the Format button. Then choose the Fill tab.

8. Now see the Fill tab below Choose your color and hit Ok.

3color

 

9 Then Choose OK Below

5

 

10. Now Comes the important part. You simply highlight all values in both columns and there is a wizard that will form the formula for you based on how many rows each column has in it. I am showing the screenshot below, after I clicked into the “applies to field” and held down the left click of the mouse while I selected both columns and went down enough rows to cover all values in both columns. I also found that if you pre-select both columns, before clicking conditional format, then these vales will be there automatically

6

11.  The only thing you can’t see in the picture above is the dashed line that is now hovering over both columns. Once I hit OK or apply, it all goes away and my formula is:

=$A$3:$B$13

Pre-Select your columns before you start with Conditional formatting

Now I found out some weirdness about the formula. If you have to go back and change the length of the column, you have to make the formula a little differently.  You click into the applies to field

and highlight Column one, then you type a comma, and then you highlight column 2. So the formula looks like this

=$A$3:$A$13,$B$3:$B$13

The more general solution is coming up below:

=$A:$A,$B:$B

I like the second formula better, because you can see the start and finish of each column. But the more general one covers all KB in both Columns. cant beat that. I am editing this sentence,

after the fact, and I found that if you select the columns, by Highlighting the columns entirely, you don’t have to deal with touching any formula aspect at  all.

Here the download again for My Excel HotfixCompare , You can also watch the video at the top and make your own spread.

Believe it or not, that is it! When you hit OK, your values immediately calculate out:

7

 

The next issue I tried to solve was the length of the column. It looks to me that this is the master formula, no matter how many KB s you have  (=$A:$A,$B:$B). This is for column A and B.

I do believe you could replace these values with columns a and c or b and E or whatever. This can help you move columns for your ease of use and you list can be as long or short as you want.

So use this in your Duplicate Values formula:

=$A:$A,$B:$B

Again, I am coming back after the fact, and just saying to Highlight the columns you want to use, before you start, and you don’t have to deal with the formulas at all. I have a second Excel

Method for comparing up to 6 servers, but I will let you just download the Excel File and see how it works!!! Watch the video, I performed that in about 1 minute in the video. also

after the fact, and I found that if you select the columns, by Highlighting the columns entirely, you don’t have to deal with touching any formula aspect at  all.

 

How do I get my Hot-fix list = Get-Hotfix

So this is all we have left to cover. Below these are the steps I use to get that list of KB numbers isolated from the rest of the information you find when you run the Get-Hotfix command.

 

1. Open PowerShell as Administrator
2. You may run Get-Hotfix > c:\myhotfix.txt
3. Open Notepad ++ and Open myhotfix.txt
4. Hold Control and Alt Key on your keyboard at the same time
5. While (4) use the mouse to select just the KB column
6. once the KBs are highlighted Right click and choose copy.
7. Paste you KBs into notepad for transport to your work station.

That’s It! I hope you have enjoyed this article. In closing I have the original link for the script in the initial discussion. I also have a method to update your Hyper-V Cluster automatically with a script. That can be pretty handy.

 

The Fallacy of Performance or; Are you bringing your Support Agent Apples or Oranges? VM Virtualization Performance with DiskSpd.

The Fallacy of performance.

I don’t think you don’t already know this. My experience does tell me that we all group things together naturally and sometimes the performance issues we find, are really assertions made, with one piece of evidence. This performance claim is generally hard for a support agent to frame. Not that your case wont be worked, on. It will. It just it may take support teams hours or weeks to get to the truth of your statements.

I have to write this because it is so prevalent. When someone calls me and they want to open a support case, I generally try to standardize the case to some truthful statements, which I can prove, disprove, or alter.

However one such case type, that does not fit into such neat lines is the Virtualization of Performance case. Rather than describe in computer terms, let’s use American Auto Makers Ford and Chevy.

Baselines Matter

I Studied Ford and Chevy Specifications for months. I know the performance characteristics of each very well. Let’s even say I own a Chevy, and I am now looking for a second car of equal specifications. Let’s even say I own a Ford and am looking for a second ford. Then I purchase that second car, and on the way home, I find that the one car does not seem to meet the specifications of the first!

Of course I must call the car company and make the complaint that one car is not as fast as the other, or not as quick to brake, or some other specification. How about this; the air is not as cold in one as it is in the other!! This is what happens in Performance calls.

Just so you know we technically should not even entertain these types of questions. But in support we do, to some degree, because we want to help, and were not sure what your showing us yet. We don’t see the pieces for several calls.  You can’t force it, it just takes time.  This is because you are asking us to form a relationship to ideas that are not related. Two cars are not related, two computers are not related.

 

How about taking a cross country trip? One car crossed the country in three days. The other took 4. At this point you may be seeing my point. Trying to get down to difference in different items, of any type can be like comparing apples to oranges. What’s even worse is when one of the items is off limits. So My Ford is definitely slower, but my wife takes the other ford to work so we really can’t use that one for testing! Now what do we do?

Basic Performance assumptions

So there are some simple rules that you should apply to any performance problem:

 

  1. The performance should be documented and repeatable.
  2. More than one test should be run, and simple is usually more realistic.
  3. Tests should be standardized, down to a science, so that if applied to another matching scenario, you would expect similar results.
  4. Keep the time down to a short test. The longer the test, the more variables can be introduced.
  5. Do not focus on two separate car models not functioning the same, find a way to introduce a baseline into what a reasonable car will perform like. Then prove or disprove your baseline.

 

Now obviously, the complexity of computers can result in more rules, but if you follow these basics, you can at least find some sanity in your test results. In fact Support has an absolute need, that this happens. It is very possible, nothing is really, wrong, if we don’t get down to brass tacks.

So real world

Call into support and report when you run this command on one machine, things are fine. When you run on the other environment, things failed. This is the Disk Speed command. This is the replacement for SQLIO. I really like this tool.

 

  • diskspd –b8K –d30 –o4 –t8 –h –r –w25 –L –Z1G –c20G  D:\iotest.dat > DiskSpeedResults.txt

 

However, what is hiding in this statement, violates all the 5 rules above. This is an assertion, based on one command. Furthermore, you ran this command, in the test location,  over and over, while other VMS are also running, randomly, creating a random pattern of storage fragmentation, while the Production environment was only one once, in a very controlled situation. These commands were not run in a scientific fashion.

It literally took me a days to think of a way to baseline this situation and to  test this correctly. This is where the 5 rules came from. I think they are solid rules for support to go by. So here is how you test to make your case to Support:

User Guide and Product here

Introduce a Baseline. Anything is better then nothing

The Above diskspd command is complex and long. Come up with some simple tests and run more of them, over time. Second, test your commands, on a laptop, or desktop, with a specific Ram, Storage, and Processor Profile. Once you record all the results on the client machine, duplicate the test in the Virtual Machine. Make sure it’s the only Virtual Machine Running. Make sure nothing is running on the Host but this one VM, with specific resources.

Now below I am not giving you results. I am just giving you the commands, Along with some Instructions on how to use DSKSPD. I am also leaving you with Articles that VMware and Microsoft Hyper-V use, when asking for baseline testing. Notice, how many little requirements they have Seem familiar? There is a reason for this! We are all trying to be scientific.

Tests to establish a Baseline.

  • .\diskspd -c100M -d20 c:\test1 d:\test2
  • .\diskspd -c2G -b4K -F8 -r -o32 -W30 -d30 -Sh d:\testfile.dat
  • .\diskspd -t1 -o1 -s8k -b8k -Sh -w100 -g80 c:\test1.dat d:\test2.dat
  • .\diskspd.exe -c5G -d60 -r -w90 -t8 -o8 -b8K -h -L
  • .\diskspd.exe -c10G -d10 -r -w0 -t8 -o8 -b8K -h -L d:\testfile.dat
  • .\scriptname.ps1
  • Same as above- second location
  • .\Diskspd -b8k -d30 -Sh  -o8 -t8  -w20 -c2G d:\iotest.dat

 

This list will generate about 15 unique results. Any of these will run on a laptop or a server. Just make sure you read the text character decoder sheet available with the product.

So the instructions are very simple. The specs on the Hyper-V or VMware VM, must be the same as the laptop. My laptop has 16GB of ram, and 8 Processors.

The VM must be the only one running, and the OS should be a fresh Install. Now if the results of testing are in the ballpark of your comparison Client, then you are not having a performance issue.

The moral of the story is test from different perspectives, and use the Scientific method, as much as you are able to.

I hope this is helpful in your troubleshooting.

 

A few other Details

Here is a way to manually pre-create the files if disired

  • fsutil file createnew d:\iotest.dat 20000000
  • fsutil file createnew d:\iotest.dat 2000000000
  • fsutil file createnew d:\iotest.dat 20000000000

Here is all of the best articles on storage, and IO online right now. I was surprised that so many of  Storage Performance Needs are all in one place.

This could be an important point. If you came to this site because your numbers are not matching reality, your Monitoring tools, may not be collecting the right perfmon numbers, then you may need the Hyper-V performance script to use to see your actual VM numbers. try using this tool Run this tool on the host, while using diskspd on your VM.

DO not run more then one instance of Diskspd at once!! This will invalidate your tests!

 

Finally, as promised, here is how  VMware or Microsoft  handle these issues:

 

Louis

Deploy Remote Desktop Services in a Workgroup or Domain Easily!

image

For all you Terminal Server or Remote Desktop Services or RDP Geeks out there, Let me spend a minute to clarify a call driver that continues to be popular.

The Scenario is deploying Remote Desktop Services in a work group. Call this a corner case, or call it what you will. The reason this is a popular support call is due to the fact that there are two articles needed to complete the setup.

Oh sure Microsoft does tell you to add a policy after your setup, but they specify you use GPEDIT. not much help there… Until Today!

First you need to Deploy the roles correctly. The Specific KB I chose for this article, is the one you would use for the simplest setup. One that keeps you clear of very common mis-steps of walking through the setup in the Server Manager. If you did your deployment correctly, you didn’t even need Server Manager.

So begin by reading this:

Guidelines for installing the Remote Desktop Session Host role service on a computer running Windows Server 2012 without the Remote Desktop Connection Broker role service

“”RD Session Host without the Connection Broker

To be clear about this deployment:

  • No Domain Controller
  • No Hyper-V
  • Install Remote Desktop Services.
  • You won’t get to select more until the end. You may install the following
      1. Remote Desktop Licensing and
      2. Remote Desktop Session Host
      3. RDWEB
      4. RD Gateway
  • For a work group you do not have to add the Terminal Server to the Terminal Server License server group. Why? With No DC, you don’t have to report to anyone.
  • Activate the Terminal Server using one of the three methods.. This is most common
  • Activate your RD Cal Licenses with KB2833839 or cc725890

So Far, so good. Now this is where we start to diverge from some existing Documentation

  • If you are in a work-group, go to “edit local users and groups”
  • Find the group folder and create a group for your RDP users and add your users to this group.
  • Alternatively, you may add your users to the RDP users group already there
  • Remember the group you are using. It becomes important
  • Now you are going to edit the Local Policy by doing the following:
      1. Start and Run GPEDIT.MSC
      2. Navigate to the following:

Local Computer Policy ->Computer Configuration-> Administrative Templates-> Windows Components-> Remote Desktop Services-> Remote Desktop Session Host-> Licensing

Figure 1. GPEDIT.MSCUntitled

You are going to see now, the two LSO (Local Security Object) you will be enabling.

Use the Specified Remote Desktop License Servers –Value- IP address of RD License Server.

Set the Remote Desktop Licensing Mode- Value – 2 or 4. 2 is for Device CAL and 4 is user Cal.

Figure 2. Local Policy. Untitled

Now there is another Policy to set. For this, you want to just go back to the top. Start out at Local Security policy like before (GPEDIT.MSC).See figure 3.

 

Figure 3 GPEDIT.MSCimage

 

Expand Computer –>Configuration,–> expand Windows Settings, –> expand Security Settings,–> expand Local Policies and then click User Rights Assignment.

image

 

image

Enable this policy and add the group you used earlier. It is highlighted in this article above. Add this group to this policy. In addition, add the Remote Desktop Users group to this policy if desired. Don’t and your administrator name here. The Admin already has access. If you ad your admin name, it will lock you out. So best to to stick to adding the Remote Desktop users group.

image

 

Notice it says administrators (Plural), that is fine, but the single administrator should not be in this list. there is a well known break here if you do that.  When you are finished, you just need to add the users you want to give access to, to the group we just added to the Policy (likely the remote desktop users group)

Step 8 comes right out of the KB2833839

  • Open an elevated Windows PowerShell prompt
  • Type the following command on the PS prompt and press Enter:
    $obj = gwmi -namespace “Root/CIMV2/TerminalServices” Win32_TerminalServiceSetting
  • Run the following command to set the licensing mode:
    Note: Value = 2 for Per device, Value = 4 for Per User
    $obj.ChangeMode(value)
  • Run the following command to replace the machine name with License Server:
    $obj.SetSpecifiedLicenseServerList(“LicServer”)
  • Run the following command to verify the settings that are configured using above mentioned steps:
    $obj.GetSpecifiedLicenseServerList()
    You should see the server name in the output.

You have now covered all your bases, and  your RDP should be happy!! It will be happy because you paid attention to all the rights things!

Now I did find an interesting article to which I cant really comment on. However, it is an interesting article. IT deals with some issues, you could run across.

Based on a comment to this article, I want to address this KB    “Best practices for setting up Remote Desktop Licensing (Terminal Server Licensing) across Active Directory Domains/Forests or Work group”

This is where it tells you not to use a work group server. So Please consider this against this KB_

Guidelines for installing the Remote Desktop Session Host role service on a computer running Windows Server 2012 without the Remote Desktop Connection Broker role service

One article says don’t use a work-group. The other says “This server can be part of a work-group or may be configured as a DC.” Which do you believe? They are both true. Microsoft does not want you to use it in a work-group, but you can because they provided a way for you to do it. The cost? Printer Redirection issues, and a few other small things, but you can do it if you need to. Bottom line is Install your Terminal Server in a Domain. But, If you have no other choice then use a work-group if you have to.

 

Well That is it. I hope this has been helpful for work groups or non-work groups. this basically  can be set up on either.

 

 

Using Jperf Graphical Interface for Iperf, for basic network testing and Diskspeed or Dskspeed for storage testing.

 

Late night yawl.

Its 11:52 PM on 3/28/2017. It has been pouring in Edmond for a few hours now. For some reason I am not sleepy yet. I thought I would spend a few minutes putting together how to use a tool I was interest in.

Jperf is a script that runs on windows, and takes the place of IPERF.This little GUI should do most of the things you need, from a two windows machines. If you want to do it on Linux, then you can just use the Iperf command line.

Half the battle for Jperf is just finding the Bits. They are located here:

JPERF Download

There is also a copy of DiskSPeed or DSKSPEED.

The concept of running Jperf is simple:

  • Java must be running on both machines
  • Unzip the Jperf Folder to the Desktop
  • Use PowerShell 5 to run the Batch file on both machines
  • The server just sends data to the test subject, so to start the server, you just select the Server Radio button and hit Run IPERF in the upper right hand corner.
  • This will look like this:

 

1

 

and to run the script, all you had to do was run a batch file called Jperf.bat:

 

2

Now after starting the Jperf on the Server machine, you simply run the same batch file, on the machine that is going to be the client. Choose the client radio button, the Server Port, and Server IP. You have other choices, but you can test various packet configurations after you have showed that the two applications are doing the basic monitoring succeeding.

 

SO now that you have both GUIs up, chose run IPERF on the server, then on the client. you should see some graphing immediately.

3

The tool is nice to just show how the network is working between two points, with various network conditions, that you can control.

If you look at the enclosed User Guide, you will find where it calls out a baseline number you can use for making comparisons  the average file transfer size and the time for that average transfer, in the last line of the test is what I am talking about (see figure below)

These numbers you want to take note of. you can use this as your baseline, to compare other situations or other iterations of the test.

4

That’s IT!

Good luck and I hope this helps to explain how you can use IPERF with a basic GUI and provide good troubleshooting  information The tool is called Jperf and It uses Iperf. Its all in the download folder.

Next Time

Watch out next time when I will review the replacement for SQLIO. The tool is called DSKSPD or Disk Speed. We will talk about how  “dsksped test not valid for hyper-v virtual machine storage” Hyper-v performance is not as easy as just using Disk SPeed

MS even states that Dynamically expanding disks should not be used for production. That is also where counters start to come back with irrational results.

Here is something to read on Hyper-v Storage optimization, until I have time to write the next article

What you Should Not Do With Your VMware Lync Deployment

 

TO begin,  let me share an article that articulates what may of you have gone through to get to this point. Many of you started at lync 2010 and are now on Skype for Business. Are you using the same hardware? If so, your problems have come from the past to haunt you.

*Disclaimer: I use Lync and SFB and Skype for business, all to refer to the general term of Lync or Skype. I believe we are all moving to just Skype, but it would be nice if MS just rolled it all to just Skype! one word please!

Untitled

This article is written for a customer I have in a nearby nation, to the United States. He has a well used Lync 2013 Deployment. He has also had a lot of trouble. From having calls fail on Fridays, to having the Web sites fail on Wednesday, every week, for several weeks in a row.

I had some time to look at this case from a distance, after being very close to the situation for several weeks in a row. As a series of chance activities, this customer ended up going to the VMware department, instead of Lync. This was a very good route! As it turns out, The VMware agent caught the Virtual Machine with 4GB of ram!.

What a coincidence! I had caught the same customer changing his resources myself! That was all it took. All the pieces were clear.

I need to express in no uncertain terms, how vigilant you must be if you decide to Virtualize Lync or Skype for Business! While there may be some issues with Lync and Hyper-V, I am being honest, when I say that most of the virtual issues have been with VMware. Now I know I may take flack for that statement. I am a Microsoft Support Agent, so I do have a bias. I don’t make that statement to support my own brand. I am a Lync Professional and I am telling you my honest experience over 7 years with Lync and Skype for Business.

 

Untitled

 

With that said, let me break it down for you, as much as I can. Let me first bulletize what the supported Lync Server would look like on ESX:

Skype Server on ESX

  • 1. The VHDX needs to be fully expanded, and should not be an remote disk- ISCSI disk.
  • The Ram must not be dynamic, and must be a minimum of 24GB of ram. 32 or more is recommended.
  • The processors must not have NUMA complications, and Hyper threading should be disabled. The ESX should only be presenting physical processors on a machine where SFB is installed.
  • Networking is in flux. I think this is where some issues are still being resolved. Let me say that I recommend you use a dedicated physical nic to the VMware VM, which ever Nic type that is. VMXnet3 nics seem to have an issue, but please keep reading the documentation. Example and Example
  • The Balloon Driver needs to be disabled completely.
  • A Lync/SFB machine should not be moved, or migrated from host to host, even for maintenance.
  • Vms should be left alone to run, and cannot stand having resources change at all. This is a real time application, and cannot be live migrated like many machine types can.
  • Do not use snapshots on Vmware, as this will just cause an issue, if you ever tried to roll back. Just full back up, and restore if necessary.
  • Do not install Antivirus or threat protection on Lync systems.
  • Rely on the security provided by an Edge and reverse proxy server.

I suppose I could keep going. So there I said it! I don’t know if your getting the feeling that Lync should be on a physical machine? I have seen many VM lync deployments, and I wont go so far as to say that. I will say, If you choose SFB and you deploy to ESX, you must remain committed to the best practices. You cant even slip once. I guarantee you will see problem.

*******************************************

EDIT Due to Feedback from another Skype Fan

I was called out on the fact that I cannot maintain the Skype cannot be virtualized. In todays market, we must be able to function with any kind of Hypervisor. I do not disagree with that position. Believe me, I did not want to come off that way. So let me clarify, to say Virtualization is not the problem, so much as Resource control. I am not back tracking much. I am saying that your are not likely  going to be  free to just move a Lync server around, if you don’t want interruptions. The SDP protocol, is not built to be live migrated! My commenter below rightfully said that all three of the technologies I mentioned worked. I don’t disagree. The issue with real time voice, is in those parts of the day when things are busy, and you find you need to move resources around, or you have automation to move resources around. This not going to work with SKYPE! Skype needs dedicated resources, and you should not change them!

I hope it is clear what my intentions are!! I love VMware and I love Hyper-V. However, all of my problem environments are with VMWare. This is on Lync, not VMware! I can say simply that Skype is not very happy with resources that are changing dynamically. So now.. back to your article!

 

*******************************************************************

 

Untitled

So before you go looking for where I am incorrect in these statements, Please start with the requirements for todays SFB server. Requirements for SFB server, are not small. and then, I don’t mind if you jump into the fire. Here let me supply some oxygen for you! Below find ways to enter the battle:

I think that is enough to get you started. So no matter what you think about Lync, it all boils down to two things, Compute resources, and the network. If either of those is a fail, then your lync deployment is a fail. Microsoft puts it more eloquently:

Lync Server is designed to consume the full capabilities of a physical server and will perform poorly if the required compute resources are not available when needed. To help the virtualization of Lync Server 2013 to be successful for your organization, follow the guidelines provided in this white paper

Untitled

My friends, you came to this sight, perhaps to find out what not to do with your Lync Server. Look at the statement above. My experience is if you have been having issues for month’s, then there is a 69% chance your answer is in the blue text above. My chances of being right are much lower if you deploy your Lync server like the bullet points at the top of this article.

Fuel? Check. Oxygen? Check. Do you need a light?

 

Good evening all and I hope your VMS are tuned and your Free memory is abundant!

Louis