The Microsoft Call Quality Methodology Scorecard is the tool that brings everything all together, all information gathered will be imported into this tool to analyze the Server and Call Quality Heath of the Lync Deployment.
Note: Make sure that you “Enable Editing” and “Enable Content”
Configuring the Score Card Settings
Before you start to import any of the results make sure that the following values are set correctly.
The Date Format is something we already know as we have queried the SQL Server for the correct Date Format. In my case the Date Format is M/D/Y
For the Decimal Separator you should have a look into the CSV files in order to get this value right. In my case the Decimal Separator is Comma
One other setting can be customized, The Streams Cutoff number specifies how many streams a given row needs to have to be taken into consideration for Top 10 Issues. This way, you can filter out rows with only a few streams included in the results since these will skew the Top 10
Importing & Sanitizing the results
Now we are ready to Import the CQM Query Results by clicking the Import CQM Query Results
This will then open the Folder Picker Dialog where you will need to select the folder where all the CSV files are located.
If the Import is successful you will see that all the CSV files are imported into the Excel Sheet into separate tabs based on the collected CSV files (21 additional tabs in total). The below example tab is the result of the imported Endpoint_0_Device.csv file.
Now that we have imported the Data we can start generating the results. One other important factor to consider is removing the Off-Work (or Weekend) days from the result as this will skew the results. This is due to a limited number of calls during that time and as a result the Trending Charts will not look very elegant (see the comparison later in the Trending Charts Section)
Now we can start the sanitizing process by removing the Off-Work days from the Result sets
The example below highlights April 13th as the value before the sanitizing process
The result of the sanitizing process is that April 13th, an Off-Work day has been removed from the Results
Now we are ready to generate all the interesting reports, as a first report we Generate the Streams Distribution Chat which explains what type of streams that have been found and what type of connection is used.
This generated a nice summary page with the total amount of calls, poor stream calls and distribution between type of calls including nice graphs showing the actual division between the different connection types and poor streams percentages.
Additionally you can even modify these charts in order to drill down into a custom view of your choice.
In this case we only want to focus on the Wireless Streams distribution.
In this case we only want to see the PoorStreams and don’t care about the total amount of streams per connection type.
Next we Generate the Trending Charts.
These Charts dig into the entire Date Range and draws out nice Charts per connection type (10 different connection streams in total) over time, showing you the trends so you can focus on the most obvious and visible ones.
For example there have clearly been at least two Network issues in the Trending Chart as there are two peaks both in Wired (more visible) and the Wireless trending charts. The PoorStreamRatio for Wireless is clearly dropping over time which is a good sign.
Note: This result has been generated with removal of Off-Work Days
The next graphs show the results for both CQM Result without removal of Off-Work Days and the Results with removal of Off-Work Days for both Wireless and External. You will see that the removal of the Off-Work Days clearly smoothens the Trending Chart.
Wireless Trending Chart (Off-Work Days included)
Wireless Trending Chart (Off-Work Days removed) is clearly showing a much nicer graph and is easier to interpret.
External Trending Chart (Off-Work Days included)
External Trending Chart (Off-Work Days excluded) which has an even bigger impact on the Trending Chart.
Now that we have generated the Distribution and Trending charts we can move on to Generate the Top issues.
This Top issues list helps you to identify where to start when you want to optimize the Call Quality.
The following Top Issues are Reported:
- Top 10 PoorStreamsRatio issues for LastMile_0_Wired where AllStreams > 50
- Top 10 PoorStreamsRatio issues for LastMile_1_Wireless where AllStreams > 50
- Top 10 PoorStreamsRatio issues for Plant_1_AVMCU_Mediation where AllStreams > 50
- Top 10 PoorStreamsRatio issues for Plant_2_Mediation_Gateway where AllStreams > 50
- Top 10 AvgSendListenMOS issues for Endpoint_0_Device where NumStreams > 50
- Top 10 CallerAudioMicGlitchRate issues for Endpoint_1_System
- Top 10 Subnet issues for Endpoint_2_Relay where Streams > 50
Now we can finally Generate the overall ScoreCard that will give us an overall view on the Health of the Lync Deployment.
This ScoreCard shows 3 areas of attention completely aligned with the CQM Methodology which looks at the Server Health, Client Health and Last Mile Heath. This is a very visible report that allows you to focus on the right things in order to achieve maximum Call Quality.
Unfortunately there has not been any work done yet to incorporate the result of the Server KHIs into the report, I would expect this to be incorporated in a future version. But you can do this manually yourself and generate a health color based on the results of the KHIs Report.
This is an example of a full report. This report shows clearly by highlighting the areas of attention and making it easier to prioritize your work and distinguish between Servers, Clients and Networks. The color coding is based on the Target values, which you can change to match your environment.
For example updating the AVMCU to Mediation Target from 2 to 3 has a visible result on the ScoreCard
Now the color has changed to Yellow instead of Red based on the new Target score you have specified.
Additionally the ScoreCard will literally give you pointers on how to enhance each section specific. In the Actions Column which is highlighted below gives you the next steps you should look at in order to help you focus on remediation.
A detailed example is highlighted here below.
For the Wireless Network the following suggestions are made:
- Identify hot wireless and wireless P2P subnets - look at the TopIssues tab
Identify subnet locations
- Verify poor call quality is reported by users on hot subnets
- Troubleshoot network issues on hot subnets (QoS, etc.)
- Repeat until GREEN
Now it is up to you to make sure that you do the right things to enhance the Call Quality.
While this is not an exact science, CQM does give you the ability to view the health of different components of your Lync deployment, it highlights trending of Call quality over the previous period and you have the ability to tackle the most important things first in order to remediate Call Quality issues.
Remember that this is NOT a one-time exercise as you will take actions to enhance the overall Call Quality and need to validate regularly where Quality has been enhanced and where you still need to focus on to complete the journey to (near) perfect Call Quality.
All in all this is a great effort from Microsoft but it still lacks a lot of automation if you consider all the work that needs to be done in order to get the Server Health in a nice diagram including the collection and creation of all the Scorecard metrics. It would be awesome if Microsoft could include these reports in the QoE reports directly or integration of these concepts in 3rd party monitoring tools.