X Call quality MOP

The primary objective of this document is to provide standardized workflow and training information so that applicable personnel can 1) easily and quickly perform associated tasks to achieve the desired result and/or 2) understand and locate any additional data points and/or accompanying MoPs.
The basic workflow of this process document is to systemically define the following:
How to troubleshoot a call quality report
After reading the understanding this MOP personnel will know how to successfully:
Troubleshoot a call quality report on or off-net and handle it appropriately.

East/West Configuration Information

Please be HYPER AWARE of this situation. A change was made a few months ago where if a user is provisioned and phones are authenticated or built via TFTP, if the customer is located WEST of the Mississippi  the config will point to west01.voip.evolveip.net (or west02) -  if they are EAST of the Mississippi the config will build to east01.voip.evolveip.net (or east02)

If the customer has a managed circuit with Evolve IP (T1, Fiber, etc) the config has to point to east regardless of where the customer is located geographically. If you build a phone in toolkit, you must go back and check it’s pointing to the appropriate server

 Call quality issues:
Evolve IP strives to offer the highest quality service to our customers. For customers that have managed access (T1, fiber, etc) with Evolve IP, then we are responsible for ensuring the highest possible quality end-to-end. For customers that provide their own 3^rd-party access and connect to us using the public Internet, we are unable to guarantee the same level of quality because we do not control the path between the customer's network and ours. Furthermore, while our phones will tag packets for QoS, most (if not all) Internet providers do not honor the QoS markings (some may strip the markings) and voice traffic is treated equally with all other traffic. Therefore, when an "off-net" customer contacts us about quality issues it is important to check that our equipment is configured and behaving properly, and then offer guidance to the customer on what information they may need to pursue the issue with their ISP.
We would be looking for root causes and reasons of unacceptable network performance. Report threshold violations can include: Low MOS scores (<3.5), Packet Loss (>1.0%), Jitter (+/-30ms). Network performance will be degraded and voice quality will most likely suffer.
Quick List:

1. Ask Customer:
   1. How many phones are affected by this issue?
      1. All
      2. Some
      3. one
   2. Obtain call example:
      1. Caller
      2. Callee
      3. Time/date
   3. Describe symptom:
      1. Choppy?
      2. Robotic?
      3. Echo or hollow?
      4. Dropped call?
   4. Which side of the call is hearing the quality problem?
      1. customer-side
      2. other side 
      3. both
   5. Does it happen all of the time or sometimes?
2. Collect:
   1. Wireshark of call example
      1. If any errors are seen, open a ticket with carrier
      2. Verify call came from expected IP, otherwise check phone VLAN / cabling
   2. Palladion voice quality details
   3. MOSgraphinator Details
   4. Interface statistics
      1. Edgewater
         1. Verify no errors
         2. Check duplex is full
      2. Onsite Cisco equipment
      3. Switches
      4. HS01 (onnet)
         1. If errors are seen, open ticket with carrier

Detail:

1. Ask the customer:
   1. How many phones are affected by this issue? ( All, some, one?)
      1. If a single user, then the issue may be cabling for that phone. Check LAN MOS. Also, swapping the phone with a known working phone is an easy way to troubleshoot if the issue is with the phone or with cabling.
   2. Obtain the details of a call example.
      1. Caller
      2. Calling
      3. Time/Date
   3. Describe the quality issue they are experiencing: is it broken up / choppy? Is it "robotic" sounding? Do they hear echos?
      1. This will help determine the type of trouble to look for:
         1. Choppy or gaps in speech: packet loss, which may be caused by jitter or over utilized bandwidth
         2. Robotic sounds or other sound quality: jitter, out of order packets
         3. Echo or hollow sounds: latency
         4. Dropped Call
   4. Which side of the call is hearing the quality problem (customer-side, other side, both?) This will tell you in which direction the problem is occurring.
      1. Customer-side hears the issue: issue is on the receive side of the WAN and/or the phone
      2. Other-side hears the issue: issue is on the transmit side of the phone or the WAN. (Note: This is often caused with DSL customers that have a small amount of "up" bandwidth.)
      3. Both sides hear the issue: need to look in both directions
   5. Does this happen 100% of the time, or only sometimes? If sometimes, does it occur when they download large files? Is it during the same time each day?
      1. Note: This may indicate bandwidth saturation and/or lack of QoS on the Internet.
2. Pull the call example with Palladion
   1. On the left hand side of the screen, select calls
   2. Filter with the call information obtained in 1.b. Make sure you collect all legs of the call. You should be able to see the call comeing into the platform from the source and reaching the destination from the platform. (ie if a call comes from the PSTN to a user, you will see the call come in to the VoIP platform, then leave the platform destined for the device the phone is behind)
   3. Open the call detail window for the call and review the voice quality
      1. Each direction of each leg of the call that detail is available for is reported here. You can determine exactly where the issue was by looking at the detail of each leg, finding low MOS, then determining what the cause was.
3. Another tool that we have is MOSgraphinator. It collects stats from the edgewaters and allows us to graph the calls out to look for issues.
   1. MOSgraphinator can only report on details it receives
      1. (A) stats that are reported LAN are only stats that are inbound to the edgewater from the LAN (so it would be what the far end of the call hears)
         1. If the customer reports that the other side of the call reported an issue, but the MOS is fine received from the LAN, most likely the LAN is not the source of the issue, WAN, core or PSTN are suspect
      2. (B) stats reported WAN are only stats that are received inbound to the edgewater from the WAN. (What the user behind the edgewater hears)
         1. If the customer reports that they hear an issue, but the WAN stats of the edgewater are clear, most likely the WAN link is clear. The issue is likely in the LAN however it is also possible that the issue occurred in the PSTN.
   2. Pull stats for call(s) in question via http://mosgraphinator.evolveip.net/
      1. Select IAD#
         1. MOSgraphinator uses the PTR record of the IP of the Edgewater. If you know the IP address but not what the PTR is, you can look them up by using the –a argument of the ping utility: ping –a (IP address) or by using the nslookup utility.
      2. Beneath Select IAD, choose if you want the MOS for the LAN, WAN or BOTH. You can change this filter after the report is run as well.
         1. LAN - MOS will only display for the Call Leg between the Edgewater and the phone. Low MOS on this side points to an internal LAN issue.
         2. WAN - MOS will only display for the call leg between the Edgewater and the Evolve IP voice platform. Low MOS on this side points to a carrier issue.
         3. BOTH – MOS for both legs of the call (LAN and WAN) will show together.
      3. Search by Number: Filter by most recent example, should you have 1 in particular. Leave the phone # blank if report of issue is multiple or on all calls. The best number to filter by is the remote side of the call, using the local user's number will download all calls.
      4. The default threshold values reflect our acceptable metric thresholds. You can adjust them, but keep in mind what is best common practice and deemed acceptable company-wide.
      5. Graphinate It!
      6. Below is an example of the output from the above search:
      7. Advise the customer what the MOS (and, if low, the reason (packet loss, jitter, etc) for the call shows if its offnet or customer managed LAN
      8. If further detail is required, you can download the call detail by clicking "Click Here for CSV of the Data".
         1. The score_LP(Loss Packets) column is the difference between the score_RR (actual received) score_SRE (expected received). To determine if the packet loss exceeds 1%, divide the score_LP column by the score_SRE column.
            1. To hear an example of a call with packet loss, click here.
         2. Score_OOP displays the number of packets that arrived out of sequence for the call
         3. Score_MPJ (Maximum Positive Jitter) and score_MNJ (Maximum Negative Jitter) represent the jitter on the call.
         4. Score_CLP (Consecutive Loss Packets) displays the maximum number of packets that happened consecutively. The more that occur at the same time, the more likely the drop will be heard. Note, 1 second of audio = 50 packets.
         5. score_SDD displays the Source DID/ endpoint
         6. score_DDD displays the Destination DID/endpoint
         7. score_MOS displays the final MOS of the call.
            1. G729 call has a max MOS of 3.9.
            2. G711 call has a max MOS of 4.4
            3. G722 call has a max MOS of 4.4
         8. Score_BTC advises how many times the call dropped below the MOS threshold that is set on the Services Configuration Page of the Edgewater.
         9. For a complete definition of each field, click the "Help" link on the mosgraphinator results page.
4. Ping the IP address of the Edgewater to determine if there is latency in reaching the customer's network. It is best to perform this ping from a core server (such as tftp01) and over an extended period of time (1-2 minutes continuously – in Linux, this is the default. In Windows, use the –t argument). While other factors, such as jitter and dropped packets, are more detrimental to call quality, excessive latency can also be problematic and cause both parties on the call to "speak over top of one another" and can also cause echo or they may say it sounds "hollow". The goal is to have response times <100ms, but VoIP can also be carried with "good" quality of up to 150ms.
5. Follow: G:\MOPS Published\03. Customer or Service Repair\MOP 3.1.1.1 Basic Call Quality Troubleshooting.docx
   1. (onnet) check the circuit for errors
   2. Check managed equipment interfaces for errors / proper negotiation
   3. Check for Traffic Shaping in the Edgewater
   4. Check VLAN configuration on telephone
6. (offnet)Determine what carrier the user has. This will be especially helpful if the caller isn't sure who their ISP is.
   1. Use Arin.net (whois in the upper right corner of the screen) to confirm:
   2. Do your due diligence, but be firm & urge customer to get their carrier involved. Collecting the ISP, speed test, details on when the issue occurs and what is reported by MOSgraphinator will help the customer discuss their issue with their ISP.

REFERENCES:

The below online resources are also excellent resources:

• http://www.voiptroubleshooter.com/diagnosis/index.html
• http://www.voip-info.org/wiki/view/Causes+of+Echo