User Experience Insight helps you understand and baseline the performance of large and distributed networks and identify and troubleshoot problems before users complain. UXI sensors tell you what the problem is from an end-user perspective. This gives you a starting point for where to investigate issues.

This document helps explain how the sensor arrived at a conclusion and what steps you should take to troubleshoot further. This document covers only the most frequent questions.

How you should use this document: You do not need to read every section; you can search the page for the specific error of interest.

The Status section of the sensor page displays “Offline”

Troubleshooting:

The term Offline also means “No Data” because the UXI cloud has not received any data, test results, or events from the sensor.

When you first power on the sensor, the sensor will:

This process usually takes 15 minutes but can take up to 45 minutes or more, depending on the connection speed.

When completed, the status light will reflect the sensor status.

The status light descriptions for G series sensors can be found here - https://www.arubanetworks.com/resource/user-experience-insight-sensors/

A steady white light or flashing white light means the device is operating well. If the status light/LED is orange, it indicates no connectivity to the UXI cloud through any of its configured or available networks. The device will appear offline on the dashboard when the status light/LED is orange.
​

F-series sensors and G-series sensors with cellular are shipped with pre-provisioned Twilio SIM cards activated and ready to go. However, even if the sensor has cellular capability, it does not mean the cellular coverage will be good. Generally, for each location you have sensors, it’s a good idea to have additional options for onboarding such as using an unrestricted ethernet port, using the UXI onboarding app for G series sensors or using Wi-Fi Easy Connect for G6 sensors. Please see the following document for more information on required reachable URLs for the sensor to function - https://help.capenetworks.com/en/articles/2351825-which-urls-do-i-need-to-make-accessible-for-my-sensor-to-function

For sensor models without cellular, you will need to onboard the sensors using an unrestricted ethernet port, using the UXI onboarding app for G-series sensors or using Wi-Fi Easy Connect for G6 sensors. Please see the document for more information on required reachable URLs for the sensor to function - https://help.capenetworks.com/en/articles/2351825-which-urls-do-i-need-to-make-accessible-for-my-sensor-to-function.

Please note – sensors without cellular do not send power outage notifications when unplugged or powered off.

Troubleshooting Summary:

The term “Offline” also means “No Data” because the UXI cloud has not received data from the sensor.

When fully zoomed in, any gap in test result data 10 minutes or more will result in these gaps on the status graph. Gaps do not indicate a sensor error.

The most common reasons for gaps in the status chart:

The sensor test cycle is described here – https://help.capenetworks.com/en/articles/3529400-user-experience-insight-sensor-test-cycle

When the sensor runs a test and the test fails, the sensor enters triage mode to find the root cause (network versus application issue). In an example like the one shown, there is no data for parts of this Status graph, likely because the sensor is spending time doing triage on another network.

To eliminate the gaps in the status chart, fix the issues on the other networks, fix the issues with the sensor configuration or reduce the number of tests.

Sensors that are cellular capable and have a valid cellular subscription are pre-provisioned with a Twilio SIM. There is no activation required.

Cellular is primarily used to simplify initial onboarding or communicating with our cloud when the local network loses connectivity to the UXI cloud. The sensor uses its configured ethernet or Wi-Fi to communicate back to the UXI cloud-first, but if communication fails, it will leverage the cellular connection. The list of connectivity URLs is here - https://help.capenetworks.com/en/articles/2351825-which-urls-do-i-need-to-make-accessible-for-my-sensor-to-function

If the sensor is not able to communicate with the UXI cloud and reach these URLs over Ethernet or Wi-Fi, the sensor can report issues notifications and receive configuration updates out-of-band via the cellular link, note that packet captures and screenshots for failed captive portals ARE NOT uploaded via the cellular link. Only cellular sensors are capable of reporting power outages.

Although cellular may be enabled on the device, the coverage depends on agreements between Twilio and the mobile service provider and physical surroundings of the device. When deploying the sensors, it’s best to have, as a backup option, an unrestricted Ethernet port available. G series sensors can use the onboarding Bluetooth app for or Android to get the initial configuration or G6 sensors can also use Wi-Fi Easy Connect. If using cellular for onboarding, it can take up to an hour for the sensor to get the initial configuration.

Also note – sensors without cellular do not send power outage notifications when they are unplugged or powered off.

The following help article describes the sensor packet capture. https://help.capenetworks.com/en/articles/1955490-how-does-packet-capture-work

There are two common reasons why a pcap might not be available in the triage menu.

To resolve the first issue, change the sensor pcap mode from pcap light to pcap full. Pcap light is the default and will only upload a PCAP file when an issue is first discovered. In PCAP full mode, a PCAP is uploaded every subsequent time the issue is discovered. The downside of PCAP full mode is it will use a lot more bandwidth to continuously upload these PCAP files. Therefore, it’s recommended to only be in PCAP full mode for a short time.

To resolve the second issue, make sure the sensor is connected to a network that can reach the UXI cloud. Packet captures are not uploaded via cellular.

“The Wi-Fi SSID is missing”

The sensor periodically does a Wi-Fi scan and reports the results in the Wireless Environment section of the sensor status page. When the sensor tries to connect to a wireless network that you have configured, it will check to see if this SSID is present in the Wireless Environment scan. If the sensor reports “The Wi-Fi SSID is missing”, this indicates the SSID was not returned in the scan.

On F-series sensors, the packet capture will not be useful to identify the problem as probe requests and responses are not captured. On G-series sensors an on-demand pcap should show the results of the probe requests and responses. One thing that can be tried is to set the SSID to hidden in the configuration in the UXI dashboard (even if the SSID is not hidden). This will let the sensor try and connect anyway and not rely on the Wi-Fi scan.

“The Wi-Fi SSID is present but is missing in the band you configured”

The sensor periodically does a Wi-Fi scan and reports the results in the Wireless Environment section of the sensor status page. When the sensor tries to connect to a wireless network that you have configured, it will check to see if this SSID is present in the Wireless Environment scan. If the sensor reports “The Wi-Fi SSID is missing in band”, this indicates the sensor is configured to test only 2.4 GHz or 5GHz for the network and based on the wireless scan the SSID is present but not in the band configured.

On F series sensors, the packet capture will not be useful to identify the problem as probe requests and responses are not captured. On G-series sensors an on-demand pcap should show the results of the probe requests and responses. One thing that can be tried is to set the SSID to hidden in the configuration in the UXI dashboard (even if the SSID is not hidden). This will let the sensor try and connect anyway and not rely on the Wi-Fi scan.

“Wi-Fi BSSID is missing”

The sensor periodically does a Wi-Fi scan and reports the results in the Wireless Environment section of the sensor status page. When the sensor tries to connect to a wireless network that you have configured, it will check to see if this SSID is present in the Wireless Environment scan. If the sensor reports “The Wi-Fi BSSID is missing”, this indicates the sensor is locked to test only a specific BSSID and based on the wireless scan the BSSID is not present. You can adjust the sensor config to remove the BSSID lock. It’s possible the AP was rebooted, down or did not respond to the probe requests.

"The Wi-Fi BSSID is present but not broadcasting the configured SSID”

The sensor periodically does a Wi-Fi scan and reports the results in the Wireless Environment section of the sensor status page. When the sensor tries to connect to a wireless network that you have configured, it will check to see if this SSID is present in the Wireless Environment scan. If the sensor reports “The Wi-Fi BSSID is present but not broadcasting the configured SSID,” this indicates the sensor is locked to test only a specific BSSID and based on the wireless scan the BSSID is part of a different SSID. You can adjust the sensor config to remove the BSSID lock. It's possible the AP was rebooted or reconfigured and the BSSIDs have changed.

"The case of the SSID is incorrect”

The sensor periodically does a Wi-Fi scan and reports the results in the Wireless Environment section of the sensor status page. When the sensor tries to connect to a wireless network that you have configured, it will check to see if this SSID is present in the Wireless Environment scan. If the sensor reports “The case of the SSID is incorrect” this indicates the SSID is present, but the SSID does not match the configured case. You can verify this in the wireless environment scan. This might also indicate your network config is not the same across your various sites and you should update your SSIDs to be consistent.

“Wi-Fi interface not associated to AP”

When the sensor runs a test and the test fails, the sensor enters triage mode to determine the root cause of the failure. One of the first checks in triage is a test to make sure the sensor is associated to the network. If the sensor is no longer associated, the error “Wi-Fi interface not associated to AP” is shown. This could be caused by a variety of factors such as roaming, the AP went down, the sensor was disconnected by the AP or congestion in the RF environment. This error is warning severity and only concerning if it happens consistently.

“Wi-Fi association failed”

If the sensor is unable to associate to the network, the sensor will report “Wi-Fi association failed”. This error code is often presented in conjunction with other error codes. For example, if there is also an error code that says the “WIFI SSID is missing” and another error that says “Wi-Fi association failed” the root cause is likely the “WIFI SSID is missing”. If the “Wi-Fi association failed” issue happens by itself and occurs intermittently, it may be due to RF interference. If the issue is persistent, it may be the SSID is using some capability not supported by the sensors.

“Wi-Fi authentication failed“

If the sensor is unable to authenticate to the network, the sensor will report “Wi-Fi authentication failed”. This error code is often presented in conjunction with other error codes. For example, if there is also an error code that says the “Wi-Fi pre-shared key is incorrect” and another error that says “Wi-Fi authentication failed” the root cause is likely the “Wi-Fi pre-shared key is incorrect” error. Double-check your network settings and re-enter the password if necessary.

“802.1X authentication failed”

This error indicates the sensor associated to the AP, however the sensor was rejected by the RADIUS server. A client doesn’t have much information on why it was rejected by the RADIUS server. You will need to check the RADIUS server logs to see the detailed reason behind the failure. The most common reasons are invalid or expired credentials. You can download the triage packet capture to review the 802.1x request, server and client exchange and typically will be able to see a failure, but the reason behind the failure would be present in the RADIUS server logs.

In the case of a configured 802.1x Ethernet port, the actual switch port itself to which the sensor is connected to may not be configured for 802.1x authentication.

“802.1X authentication timed out”

This error indicates the sensor associated to the AP, however the sensor was unable to complete 802.1x authentication before 60 seconds elapsed. This may indicate messages are not being properly forwarded to the RADIUS server. You will need to check the RADIUS server logs to see the detailed reason behind the failure. Looking at the triage packet capture, often this error presents a correct server and client certificate exchange, thereafter the EAPOL 4-way handshake is initiated and the initial EAPOL start packet is not responded to. In this case, the profile used by the sensor to auth is correct, however the RADIUS server is not responding to the 4-way handshake request.

“802.1X authentication failed with unknown CA”

The error “802.1X authentication failed with unknown CA” indicates the sensor does not trust the RADIUS server and therefore did not attempt to connect. The best ways to establish trust are to embed the RADIUS server cert in the #PKCS12 file or specify the RADIUS server certificate chain in .pem format and upload using the advanced option to specify the custom CA - https://help.capenetworks.com/en/articles/3322411-custom-ca You can also validate the required certificates from the sensor triage.

“Ethernet carrier is not present”

This error indicates the sensor is configured to test ethernet, but the sensor does not see any connection on ethernet. On the switch, you can check the ethernet port admin and oper states and duplex modes. You can also check the cable and verify the sensor is plugged in the expected port. You can also update the sensor configuration so the sensor no longer tests ethernet.

You can set a static IP for a sensor in the sensor settings under the Configure options when you add a network to a sensor. This option is only available for IPv4.

Unable to allocate static IP

This error indicates the sensor was unable to set the static IP address of the sensor. The reason is usually explained in the issue triage. To correct the issue, you will need to update your sensor configuration.

In the UXI dashboard, DHCP response time is the total time from when the sensor starts trying to get a lease to when the sensor receives the lease. This time could include retries in case of failures/errors. The DHCP response time includes all initial DHCP discover packets which are potentially not responded to. The sensor will do the full DORA process every time it joins a network. The sensor explicitly releases the IP when the sensor is finished testing the network by issuing a DHCP release packet before disassociating from the configured SSID (in the case of Wi-Fi testing).

"No response from DHCP server"

The sensor uses a Linux utility for DHCP, and this tool has an exponential backoff algorithm for retires when attempting to get a lease. If after a total of 60 seconds the sensor does not get a response then it is considered a DHCP failure with an error that says “No Response from DHCP Server”. You can confirm the DHCP issue in the issue triage and the pcap PCAP attached to the issue. If the device is testing two or more SSIDs which share the same subnet there may be a network isolation issue whereby the device’s initial DHCP lease from the first network is not fully released (or the DHCP release packet is not received) by the time the device is associated to and testing the next network. It is recommended to separate adjacent L3 networks with distinct routed L3 subnets rather than sharing L3 infrastructure between distinct SSIDs, especially when guest networks are involved.

In some cases, logs from the DHCP server will show the DHCP lease is being requested and issued to the sensor, but packet captures from the sensor could show that the DHCP offer is never being received by the device. In such a situation an Over-The-Air packet capture is recommended to determine where the DHCP offer is being dropped (it could be dropped by a firewall policy somewhere else in the network infrastructure).

High DHCP Response Time

If the sensor can get an address via DHCP, but the time it takes is larger than the threshold set in the dashboard, the dashboard will report a warning or error message that says “High DHCP response time”. You can adjust these thresholds and set them to what is appropriate after measuring the network behavior for a week or more. Thresholds are a rolling average calculation per sensor, per network and per test. It's recommended to adjust the threshold for the highest values based on your historical data and expand the time period if necessary.

To diagnose the root cause of DHCP issues, it’s usually recommended to request and examine an on-demand packet capture from the sensor status page. Once downloaded, the datagrams pcap PCAP will show you which part of the DORA process is failing or is taking a long time to complete. You should compare the timestamps of the datagrams pcap with the default pcap PCAP to ensure you are looking at the DORA process for when you are connected to a specific network.

High DHCP response times may be due to a misconfigured load balancer or broadcast messages not being forwarded correctly.

“Gateway is unreachable”

After getting a lease from DHCP, the sensor checks if it can reach the gateway using ARP. If there is no response, the sensor will report the error Gateway is unreachable. Usually this indicates an L2 forwarding issue.

“No connectivity past gateway”

If the network is configured for external connectivity, the sensor will check if it has external connectivity by checking http://cdn.capenetworks.io/auth for an expected response code, if the response differs from what is expected, then this issue is raised. The error “No connectivity past gateway” usually indicates there was a timeout reaching this URL. If the issue is transient, it may indicate an outage of your ISP. If the issue is persistent, it may indicate your network does not have internet connectivity and therefore you should update the network configuration in the UXI dashboard under the advanced settings and turn external connectivity to false. After updating this setting, it may take 90 minutes for this error to be resolved by the UXI cloud.

This error may also indicate the sensor is being blocked by a tool like Zscaler. For Zscaler, you will need to apply a bypass rule on your Zscaler instance for our required URLs.

“Connectivity URL resolution failure”

If the network is configured for external connectivity, the sensor will check if it has external connectivity by checking cdn.capenetworks.io/auth. The error “Connectivity URL resolution failure” usually indicates there was no DNS response for this URL. If the issue is transient, it may indicate an outage of your DNS. If the issue is persistent, it may indicate your network does not have internet connectivity and therefore you should update the network configuration in the UXI dashboard under the advanced settings and turn external connectivity to false. After updating this setting, it may take 90 minutes for this error to be resolved by the UXI cloud.

The DNS test is described here - https://help.capenetworks.com/en/articles/3498645-dns-resolution-tests

The DNS test is performed against the primary and secondary DNS servers. Primary and secondary are determined by the order of the nameservers returned in the DHCP lease.

“Primary DNS failing”

This error indicates the DNS test failed only for the primary DNS server. If the issue is transient, it may indicate a brief network or RF issue. If the issue is persistent, it indicates a network configuration issue. The error is only reported after a timeout of 60 seconds, which includes multiple attempts.

“Secondary DNS failing”

This error indicates the DNS test failed only for the secondary DNS server. If the issue is transient, it may indicate a brief network or RF issue. If the issue is persistent, it indicates a network configuration issue. The error is only reported after a timeout of 60 seconds, which includes multiple attempts.

“DNS resolution is failing”

This error indicates the DNS test failed for the primary and secondary DNS server. This error also indicates DNS also fails when no specific DNS server is specified. If the issue is transient, it may indicate a brief network or RF issue. If the issue is persistent, it indicates a network issue.

“Unable to resolve hostname”

This error indicates the DNS server was reachable, but the server did not provide a resolution. You can view the response in the issue triage. You will likely see no answers to the query. This indicates a DNS server issue. Take note of the nameserver that provided the response.

“High DNS lookup time”

This error indicates the DNS test is successful in under 60 seconds, but the result is higher than the threshold setting. You can update the threshold under settings -> thresholds. Thresholds are a rolling average calculation per sensor, per network and per test. It's recommended to adjust the threshold for the highest values based on your historical data and expand the time period if necessary. If this issue is constant, you can also validate the DNS measurement by requesting an on-demand pcap.

Read the following help article to learn how to record a captive portal - https://help.capenetworks.com/en/articles/1927431-captive-portal-setup

To test if a captive portal is present, every test cycle the sensor will check if it can reach http://cdn.capenetworks.io/auth and get “Success” in the result.

Note that this URL cannot be changed (even if you change the DNS lookup URL). The reason is this page has specific text for the sensor to get and parse to verify it successfully reached the URL. If a captive portal is present, the sensor should be redirected to a captive portal when it browses to this URL.

There are three possible outcomes:         

When a captive portal test is run, the first step is to load the captive portal webpage. The error "Captive portal landing page inaccessible” indicates the captive portal web page did not load. To diagnose the error, look at the datagrams pcap PCAP to see what captive portal server responded to the request, then investigate the captive portal server or network issue.

Once the sensor loads the captive portal page, the sensor will begin to replay the captive portal script step-by-step. The error “Captive portal login failed” indicates the captive portal script did not complete successfully and failed at one of the configured steps. This can also happen if all the captive portal steps have been completed but the sensor is redirected back to the initial login page. Triage indicates the step where the captive portal script is not working. In addition, triage attempts to capture a screenshot of the issue. Screenshots are only available if the sensor can upload over wired or wireless. Screenshots are not uploaded over cellular.

If all steps were executed successfully but afterward the sensor was not able reach cdn.capenetworks.io/auth the sensor will report the error “No connectivity past captive portal”

The captive portal should only contain basic commands like click and type, and the credentials should be static and be able to be used by multiple sensors. You may need to create a service account specifically for the sensors to authenticate.

One common issue for captive portal issues is missing steps in the captive portal script. For example, a script may contain steps to enter a username and password, but there is still another button to click on the page the sensor was presented with that is not in the recording. Another possible situation is that the credentials used to authenticate to the captive portal have expired. So the sensor goes through all the steps correctly, but the login fails.

To troubleshoot these issues, it’s usually best to examine triage for where the test is failing and if necessary record the captive portal again using the steps outlined in the help article. It’s also advisable to clear the browser cache before making the recording.

If the sensor successfully runs the captive portal script, you may see a threshold warning that says “High Captive Portal Load Time”. This error indicates the sensor able to successfully run the captive portal script, but the time taken to load the captive portal and complete the login was higher than the desired threshold. To resolve this error, update the captive portal thresholds. If the error is persistent, download an on-demand pcap and examine the datagrams to see which captive portal server is taking longer to respond to the request.

Captive Portal on Ethernet

A captive portal can be assigned to a wired network. There are no known limitations (beyond the normal captive portal limitations). However, this has not been extensively tested. Captive portal recordings on Ethernet must also be recorded using the Aruba recorder Chrome plugin and can only be added by engineering via support.

"The internal or external test is grey"

If a test is a grey, it usually indicates “no data”. There are many reasons why the test may not be running.

"Internal service is unavailable” or “External service is unavailable”

The “Service is unavailable” error message is often the result of a failed TCP or ICMP ping.

You can validate this in the issue triage.

If the issue is transient, this error may indicate an application issue.

If the issue is persistent, this error may indicate an application issue or the test is misconfigured.

To troubleshoot:

“SSL Error”

When the sensor performs a webserver test and the option to validate the SSL certificate is enabled, the sensor will check that the SSL certificate returned is trusted and valid. If the certificate is not trusted or valid, an SSL error is returned. In the issue triage, the curl output can help you identify the specific reason for the SSL error.

If you have something on the network that performs SSL inspection/decryption, you can provide a trusted certificate to support. The preferred format is base64 encoded .pem, but if it's in another format it can be converted. The certificate is assigned per network, so any sensor testing that network will use that certificate. Also note the URLs required for the sensor to reach the UXI cloud should not require SSL inspection: https://help.capenetworks.com/en/articles/2351825-which-urls-do-i-need-to-make-accessible-for-my-sensor-to-function

You can optionally adjust the webserver test template to disable SSL validation. After making this change it may take up to 90 minutes for the SSL error to no longer be reported.

"HTTP timeout"" or "HTTPS timeout"

This error indicates the sensor attempted to run an HTTP GET but the request timed out after 25 seconds. You can validate the timeout in the issue triage and triage packet capture. If this issue happens intermittently, it may be due to RF interference or other wireless factors. If the issue is persistent, double-check the test configuration URL. One possible reason for this issue is the application has determined the sensor is a bot and has decided not to respond to the request. Consider applying the rate limit and frequency options to the test template, simplifying the test to check only TCP connectivity using the generic test template, or expanding the test to be a web-based interaction with web app testing.

"Unexpected HTTP status code" or "Unexpected HTTPS status code"

This error indicates the sensor attempted to run an HTTP GET but the response was not HTTP response codes 200 or 204. You can validate the response code received in the issue triage. Typically this issue is an application issue. If the issue is persistent, double-check the test configuration URL. You may also consider expanding the test to be a web-based interaction with web app testing.

"High packet loss", "High latency" or "High jitter"

These issues are examples of threshold issues. Thresholds are successful tests but the results violate the threshold settings. Thresholds are a rolling average calculation per sensor, per network and per test. It's recommended to adjust the threshold for the highest values based on your historical data and expand the time period if necessary.

Latency, jitter and packet loss are all measurements from ICMP or TCP Ping tests.

If these threshold issues happen across only one service, it may indicate an application issue. If this issue happens intermittently or across multiple applications, it may be due to RF interference or other wireless factors. To troubleshoot persistent threshold issues, double-check your test configuration. If testing multiple TCP ports, make sure all the port numbers in the test template are correct. If the issue is ongoing, you can also request an on-demand packet capture from the sensor to validate the issue.

"Low internal VoIP MOS" or "Low external VoIP MOS"

The following documents describe the VoIP MOS test template.

These issues are examples of threshold issues. Thresholds are successful tests but the results violate the threshold settings. Thresholds are a rolling average calculation per sensor, per network and per test. It's recommended to adjust the threshold for the values based on your historical data and expand the time period if necessary.

The VoIP MOS results range from 1 to 4.4 and is calculated from the average latency, jitter and packet loss of TCP packets tagged with the appropriate DSCP/TOS values for voice. The formula used is based on ITU-T Recommendation G.107.

If this issue happens intermittently or across multiple applications, it may be due to RF interference or other wireless factors. To troubleshoot persistent threshold issues, double-check your test configuration. If the issue is ongoing, you can also request an on-demand packet capture from the sensor to validate the issue.