The customer has reported choppy audio: audio cutting in and out, making it difficult for users to hear each other.  This could affect one user or both.

Information Gathering

Troubleshooting

      Identify Source of Packet Loss

      Packet Loss From Behind an Edgewater

     Additional Troubleshooting and Next Steps


Information to gather from the customer:

  1. Circumstances of the call–Think in terms of Who, What, When, Where and How:
    1. Who: The parties involved, usually an Originating Party who called out to a Terminating Party
      1. Originating Party (calling number):
      2. Terminating Party (called number):
      3. Direction of issue: Does the audio cut out for only one party (just the caller?  just the callee?) or does it cut out for both sides?
      4. Scope of the issue:  How many users are experiencing choppy calls?
        1. Single user
          1. If remote, investigate packet loss on network
          2. If in office, investigate physical layer
        2. Multiple users - Investigate packet loss on Edgewater
        3. All users
          1. Site-wide - Investigate packet loss on Edgewater
          2. Enterprise-wide - If in different locations, there could be an issue on the server or back end
    2. What: The context of the call.  Inbound?  Outbound? Internal? External? Is it only certain kinds of calls, or is it all calls?
      1. Direction (Outbound vs. Inbound): Is the choppiness only on outbound calls, inbound calls, or both?
      2. Endpoint: Method of placing and receiving call
        1. Deskphone:
          1. Manufacturer: Polycom, Yealink, Cisco or other?
          2. Primary Device or SCA?
          3. Hotel/Open seating
        2. Remote Office
        3. Softphone
          1. UC-One
          2. Skype
          3. Teams
        4. Cell phone - If a cell phone is involved, poor audio quality could be due to the cellular network.
      3. Application: Was any call tracking application in use?
        1. Unity
        2. ECS
        3. Call center
        4. Receptionist
      4. Any additional details pertinent to the call (Was the user transferred or put on hold?)
        • Any error messages on the phone or application?
        • Headset?
    3. When: Was this a one-time occurrence, intermittent or persistent with every call?  Try to get a specific call example.
      1. Date:
      2. Time (including time zone): 
      3. When did the issue start?
        1. Today at a specific time
        2. Previous day
        3. Does this happen at a certain time of day?
          • If at certain times, could be due to call volume (connection bandwidth) or network issues (network saturation or QoS issues)
          • Any unusual weather conditions such as rain or electrical storms?  Could indicate bad wiring or electrostatic discharge (ESD)
        4. Has never worked since implementation
      4. Does the choppiness start after a certain period of time, such as after ten minutes?  Does the issue occur only on longer calls?  If so, and the user's remote, this could indicate an issue with Stateful Inspection on the local network's firewall.  Provide firewall guide.
      5. Frequency
        1. Every call?
          1. Check for packet loss
          2. If none, investigate physical layer
        2. Intermittent?
        3. Can it be replicated?
          1. On user's end?
          2. On our end?
    4. Where: Location of each party. 
      1. Remote or in an office behind an Edgewater? (Ask the user, "Are you in the office or remote/at home?")
        • Check phone in OCOM to see what IP address it's registering from.  If the IP can't be found in Nagios, it's probably remote.
      2. Does Evolve own the numbers involved or are they off-platform numbers owned by another carrier? (Some troubleshooting may be necessary to determine this)
    5. How: How was the call initiated?
      1. Direct call: Originating Party dialed Terminating Party directly
      2. Queued call: The call was presented to the Terminating Party through a call center.
      3. Transfer: The Originating Party reached a Terminating Party, then was transferred to a secondary Terminating Party.
  2. Obtain a call example - Call Examples must be from within the past 24 hours so that if needed they can be investigated with underlying carriers.
    • Caller:
    • Callee:
    • Time/Date (including Time Zone):
    • Call Example Verbiage:

                   To facilitate troubleshooting the reported issue, a call example is required. 

                   Please provide us with a specific, recent call example (from within the last 24 hours) of the reported issue, so we can verify its routing and quality.

                   Please include the following information:

                  Caller number: 
                  Callee number: 
                  Date and time (including time zone): 
                  Issue description:

                  **************************Your ticket is now being placed on a HOLD status pending the requested information.

             

 Jump to top of page

Troubleshooting

  1. Find the call example in OCOM and document the call example and PCAP in the ticket
    1. OCOM
      1. Search by the offnet number.  This is the best way to make sure you get all the legs of a call.  Can search under "User Tracking" or "Calls."
      2. Calls from a softphone like UC-One will show as inbound.
      3. Is there any packet loss or jitter?  
        1. If so, identify the source
          1. Carrier: Open a ticket with the offending carrier for investigation.
          2. Local network:
            1. If in the office, is there packet loss on the WAN connection of the Gateway/Edgewater? 
              1. If we manage the circuit, open a ticket with the carrier
              2. If customer manages circuit, advise them to contact their ISP and close the ticket
              3. If only one user is having issues, confirm physical layer
                1. If using headset, try with handset and speakerphone  
                2. Replace ethernet cable
                3. Move phone to a known-good location
                4. Any issues with the local switch?
            2. If remote, provide firewall guide and advise the user to call their ISP.
        2. If not, investigate local/physical layer.
          1. If using headset, try with handset and speakerphone 
          2. Replace ethernet cable
          3. Move phone to a known-good location
          4. Any issues with the local switch?

Call Example Analysis - How to Determine if Packet Loss is Coming from Carrier or User Network

  1. Example of packet loss coming from the carrier:





 Jump to top of page


2. Example of packet loss coming from the client/user site:
















Once you've found the call example in OCOM, double click it to pull up the Call Info window.

In the first tab (Segments), the call looks fine. But to find packet loss, we'll need to move on over to the Media Summary tab.


















Here on the Media Summary tab, we can see a packet loss rate of 21.72% for this leg of the call, which is enough to cause choppiness.  For particularly bad calls, you might see upwards of 98-99% packet loss.

In this example, we can see the packet loss occurring between the two IP addresses 199.66.103.41 and 4.55.18.134.  Note that 199.66.103.41 (SD7/SD8_W) is one of our gateways that we send traffic from, destined to one of our underlying carriers. Given that, 4.55.18.134 in this example is going to be the carrier's IP address. For a case such as this, you can use Palladion to identify the ingress / egress carrier (Which, for this example would be Level 3 / Centurylink). Once the carrier is determined, a ticket should be opened with that carrier to investigate the packet loss.

Note: Multiple sections of the Media Summary may show packet loss. While packet loss may occur in multiple places, one leg will typically be the main source source and be repeated on subsequent hops. 

Jitter is another measure of instability that may coincide with packet loss.  An ideal jitter total is less than 100.  Click here for more information about jitter.


Verbiage for Client in matching scenarios:

We see a possible source of the behavior you have reported.  We will be opening a case with our underlying carrier for further investigation.  We will provide you with an update once they have completed their review.






In the Media Summary for this example, we can see the packet loss occurring between IP 50.205.85.186 and 64.27.39.177. Normally, when dealing with packet loss from the local site, you will see it between the user's gateway and one of our VoIP gateways. In this case, our gateway is 64.27.39.177 and the user/customer gateway is 50.205.85.186. Examples of gateways you'll see here include Edgewaters and User's home routers (like the router you'd get from an ISP such as xfinity). Take the IP address identified as the user gateway and search it in Nagios to determine if this is a gateway that's managed by Evolve IP (such as an Edgewater).


If this user is remote or at a location not behind an Edgewater, they will need to contact their ISP.  Provide the firewall guide.


Verbiage for Client in matching scenarios:

Presently, we see packet loss originating from your connection.  It is recommended that you open a case with your internet service provider for further investigation.  Enclosed, for added assistance, we have provided our recommended firewall guide.  At this time, we will be placing this case into a resolved status as we are unable to provided further assistance with the unmanaged connection.  






Analyzing Packet Loss from Behind an Edgewater

An Edgewater gives us visibility into a local network.  If choppiness is reported from behind an Edgewater, we can use OCOM and Nagios to identify packet loss.  If an issue with the local network is found, this may require a ticket with the carrier:

  1. For On-net circuits (managed by us), open a ticket with the carrier and keep our ticket open.
    1. You can go to the Edgewater in Nagios and Click on the Performance Graph at the Bottom of the Page to show the connection's Performance.
    2. On the left side is an example of how graphs will show Latency on the WAN connection to an Edgewater. In this case, just before 12pm there was a spike in packet loss. Ping times (top graph) were high earlier in the day but seemed to settle down by noon.
  2. For Off-net circuits (managed by the customer), advise the customer to contact their ISP, then close our ticket.
    • Verbiage for Client in matching scenarios:

      • Presently, we see packet loss originating from your connection.  It is recommended that you open a case with your internet service provider for further investigation.  At this time, we will be placing this case into a resolved status as we are unable to provided further assistance with the unmanaged connection.  Please do not hesitate to contact us should you need anything further.




Additional Troubleshooting Steps

Not all audio loss is caused by the carrier.  Sometimes it's due to local equipment or wiring.  Some troubleshooting may be necessary to isolate the source of the issue:

  1. Try to replicate the issue.
  2. Remove variables.
    • If using a headset, try speaker and handset
  3. Move the phone to another, known-good port.  If in the office, and no packet loss is showing on the Edgewater or carrier, it's possible there may be an issue with the cable, port or switch.
  4. For customers that have dedicated call recording servers, we can pull their recordings and listen to locate the audio loss (Note: Don't mention this to the customer, though.  This is just a troubleshooting step on our side) - List of customers that have dedicated servers: Orecx Call Recording Server List
  5. Based on the originating source of packet loss, your next steps may vary:
    1. Client Side
      1. Check for circuit errors, dropped packets
        1. If circuit is EIP responsibility, then open a ticket with the circuit provider
        2. If circuit is client responsibility, tell the customer they need to open a ticket with their ISP/WAN provider
      2. If circuit is clean,
        1. If Evolve owned or managed CPE, check equipment (CPU utilization, physical connections)
        2. If not Evolve owned or managed equipment, request customer to have their IT staff review
    2. PSTN Side - Open ticket with Carrier
    3. Evolve IP network - Document findings and elevate the ticket to the next Tier.
  6. When to send to Tier 2:
    1. Chronic, replicatable issues for which the client pushes back that the issue's on us or they don't believe the issue is from off our platform
    2. Issues that can't be resolved with Tier 1 resources (always present issue in the group chat and check with team leads before elevating to Tier 2)
    3. If the customer's listed on Orecx Call Recording Server List, you can mention this to Tier 2, that recordings can be pulled, but again, don't mention this to the customer.

See also:

Call Quality

 Call Quality Troubleshooting: Choppy or Static

 Documenting Call Examples in JSD

 What is Jitter?