The customer has reported a delay, an abnormal pause after dialing before the call begins or other delays in interaction.  This delay could be heard once or multiple times, on one call or over a series of interactions, and could be heard by either just the caller, just the callee, or on both sides

Information Gathering

Common Causes of Delay

Troubleshooting

     Analyzing SIP Messaging for Delays

     Additional Troubleshooting and Next Steps

Information to gather from the customer:

1. Circumstances of the call–Think in terms of Who, What, When, Where and How:
   1. Who: The parties involved, usually an Originating Party who called out to a Terminating Party
      1. Originating Party (calling number):
      2. Terminating Party (called number):
      3. Direction of issue: Did the delay happen for only one party (just the caller? just the callee?) or is the delay heard from both sides?
      4. Scope of the issue:  How many users are experiencing delays?
         1. Single user
            1. If remote, investigate network
            2. If in office, investigate physical layer
         2. Multiple users - Investigate Edgewater
         3. All users
            1. Site-wide - Investigate Edgewater
            2. Enterprise-wide - If in different locations, there could be an issue on the server or back end
   2. What: The context of the call.  Inbound?  Outbound? Internal? External? Is it only certain kinds of calls, or is it all calls?
      1. What is being experienced or reported: This could be a delay after dialing a phone number, before the call begins.  Sometimes it's a delay after dialing before any ringing is heard.  There can also be delays in calls being routed to the appropriate destinations, delays with softphones and application call pops, delays with hold music, etc.
      2. Endpoint: Method of placing and receiving call
         1. Deskphone:
            1. Manufacturer: Polycom, Yealink, Cisco or other?
            2. Primary Device or SCA?
            3. Hotel/Open Seating host
         2. Remote Office: For example, delays making an outbound call in Unity, the call takes too long to reach the user's remote office cell phone. Get call examples and test with your own cell. This may be an issue with the user's network that would require firewall settings
         3. Softphone
            1. UC-One
            2. Skype
            3. Teams
         4. Cell phone 
      3. Application: Was any call tracking application in use?
         1. Unity
         2. ECS
         3. Call center
         4. Receptionist
      4. Any additional details pertinent to the call (Was the user transferred or put on hold?)
         • Any error messages on the phone or application?
         • Headset?
   3. When: Was this a one-time occurrence, intermittent or persistent with every call?  Try to get a specific call example.
      1. Date:
      2. Time (including time zone): 
      3. When did the issue start?
         1. Today at a specific time
         2. Previous day
         3. Does this happen at a certain time of day?
            • If at certain times, could be due to call volume (connection bandwidth) or network issues (network saturation or QoS issues)
            • Any unusual weather conditions such as rain or electrical storms?  Could indicate bad wiring or electrostatic discharge (ESD)
         4. Has never worked since implementation
      4. When does the delay occur? 
      5. What is the duration of the delay?  Is it always the same amount of time?
      6. Frequency
         1. Every call?
            1. Check SIP messaging
            2. If none, investigate physical layer
         2. Intermittent?
            • Get call example and check SIP messaging
         3. Can it be replicated?
            1. On user's end?
            2. On our end?
   4. Where: Location of each party. 
      1. Remote or in an office behind an Edgewater? (Ask the user, "Are you in the office or remote/at home?")
         • Check phone in OCOM to see what IP address it's registering from.  If the IP can't be found in Nagios, it's probably remote.
      2. Does Evolve own the numbers involved or are they off-platform numbers owned by another carrier? (Some troubleshooting may be necessary to determine this)
   5. How: How was the call initiated?
      1. Direct call: Originating Party dialed Terminating Party directly
      2. Queued call: The call was presented to the Terminating Party through a call center.
      3. Transfer: The Originating Party reached a Terminating Party, then was transferred to a secondary Terminating Party.
2. Obtain a call example - Call Examples must be from within the past 24 hours so that if needed they can be investigated with underlying carriers.
   • Caller:
   • Callee:
   • Time/Date (including Time Zone):
   • Call Example Verbiage:

                   To facilitate troubleshooting the reported issue, a call example is required. 

                   Please provide us with a specific, recent call example (from within the last 24 hours) of the reported issue, so we can verify its routing and quality.

                   Please include the following information:

                  Caller number: 
                  Callee number: 
                  Date and time (including time zone): 
                  Issue description:

                  **************************Your ticket is now being placed on a HOLD status pending the requested information.

 Jump to top of page

A delay can occur for one of two major reasons:

• LAN/Physical Layer (Local Network or Device Hardware):
  • Outbound calls: After picking up the handset/headset, before hearing a dial tone
  • Inbound calls: Delays after answering a call, before being able to talk - If in office, investigate Edgewater
  • After pressing phone buttons, a delay between pressing the button and getting a response from the phone
  • Delays after a voicemail is received, before the indicator begins to light up
    • Example ticket INC-31166: VM pin was synchronized, phone was rebooted
  • Typical troubleshooting steps:
    • Reboot phone
    • Ping the phone's local IP and see response times.
    • If behind an Edgewater, try plugging the phone directly into a LAN port on the Edgewater.
    • For site-wide delay issues, investigate the Edgewater - Gather data re. up time, load and switch CPU (see higher tier if not sure how to do this.)
• WAN/Carrier (Delays with SIP/Latency on the Call):
  • After dialing a phone number, before it starts ringing
    • Example ticket INC-22889: Site-wide delays making outbound calls. They'd hear a dial tone right away but would experience delays after entering a phone number before it'd ring out. Call examples showed 403 Forbidden errors occurring right after the Invite, before finally going through to 183 Ringing. The 403 turned out to be coming from us. Bounced survivability in the Edgewater and delays stopped.
    • 
  • With audio and hold music
    • Delayed audio might be noticed on a call if the two parties are not hearing each other right away and end up talking over each other. For more information, see Call Quality Troubleshooting: Delayed audio / Talking on top of each other
  • With softphones and application call pops: Investigate network, get logs
  • In calls being routed to the appropriate destinations: For example, delays reaching voicemail or getting into a call center
  • Typical troubleshooting steps:
    • Check OCOM for jitter, latency and slow round-trip times
    • Look at SIP messaging and timing in OCOM. Any delays in when the signal's sent to the carrier and when the carrier responds? Look for delays after the invite. After the Try, you should be getting a Ringing message (180 or 183). Is this delayed after a few seconds? If so, you might need to investigate with the underlying carrier.
    • Check routing profile in OSSmosis as carrier interactions can cause a 503 error
    • 
    • These delays can occur because:
      1. SIP messaging is delayed - Look for timing delays in the call example.
      2. The call is struggling to find a path.

Troubleshooting

1. Find the call example in OCOM and document the call example and PCAP in the ticket
   1. OCOM
      1. Search by the offnet number.  This is the best way to make sure you get all the legs of a call.  Can search under "User Tracking" or "Calls."
      2. Calls from a softphone like UC-One or ECS will show as inbound.
      3. Is there any packet loss or jitter?  (Note: Delay is usually caused more by jitter and latency, as opposed to packet loss)
         1. If so, identify the source
            1. Carrier: Open a ticket with the offending carrier for investigation.
            2. Local network:
               1. If in the office, is there latency on the WAN connection of the Gateway/Edgewater? Check round-trip times in Nagios:
               2.  
                  1. If we manage the circuit, open a ticket with the carrier
                  2. If customer manages circuit, advise them to contact their ISP and close the ticket
                  3. Confirm physical layer
                     1. Reboot phone
                     2. Ping the phone's local IP and see response times
                     3. If behind an Edgewater, try plugging the phone directly into a LAN port on the Edgewater. Check survivability.
                     4.  If using headset, try with handset and speakerphone  
                     5. Replace ethernet cable
                     6. Move phone to a known-good location
                     7. Any issues with the local switch?
                     8. For site-wide LAN delays,  investigate the Edgewater - Gather data re. up time, load and switch CPU (see higher tier if not sure how to do this.)
               3. If remote, provide firewall guide and advise the user to call their ISP.
         2. If not, investigate local/physical layer.
            1. Reboot phone
            2. Ping the phone's local IP and see response times
            3. If behind an Edgewater, try plugging the phone directly into a LAN port on the Edgewater
            4.  If using headset, try with handset and speakerphone  
            5. Replace ethernet cable
            6. Move phone to a known-good location
            7. Any issues with the local switch?
            8. For site-wide LAN delays, investigate the Edgewater - Gather data re. up time, load and switch CPU (see higher tier if not sure how to do this.)

Additional Troubleshooting Steps

If no packet loss, jitter or SIP messaging has been found to account for the delay, you may have to investigate the physical layer:

1. Try to replicate the issue.
2. Remove variables.
   • If using a headset, try speaker and handset
3. Move the phone to another, known-good port.  If in the office, and no packet loss is showing on the Edgewater or carrier, it's possible there may be an issue with the cable, port or switch.
4. Check the Edgewater for latency or down survivability. 
5. For customers that have dedicated call recording servers, we can pull their recordings and listen to locate the delay (Note: Don't mention this to the customer, though.  This is just a troubleshooting step on our side) - List of customers that have dedicated servers: Orecx Call Recording Server List
6. Based on the originating source of packet loss, your next steps may vary:
   1. Client Side
      1. Check for circuit errors, dropped packets
         
         1. If circuit is EIP responsibility, then open a ticket with the circuit provider
         2. If circuit is client responsibility, tell the customer they need to open a ticket with their ISP/WAN provider
      2. If circuit is clean,
         1. If Evolve owned or managed CPE, check equipment (CPU utilization, physical connections)
         2. If not Evolve owned or managed equipment, request customer to have their IT staff review
   2. PSTN Side - Open ticket with Carrier
   3. Evolve IP network - Document findings and elevate the ticket to the next Tier.
7. When to send to Tier 2:
   1. Chronic, replicatable issues for which the client pushes back that the issue's on us or they don't believe the issue is from off our platform
   2. Issues that can't be resolved with Tier 1 resources (always present issue in the group chat and check with team leads before elevating to Tier 2)
   3. If the customer's listed on Orecx Call Recording Server List, you can mention this to Tier 2, that recordings can be pulled, but again, don't mention this to the customer.

Jump to top of page

See also:

Call Quality

 Call Quality Troubleshooting: Choppy or Static

 Documenting Call Examples in JSD

 What is Jitter?