QoS Benchmark of 3G/4G Mobile Data Services
This is the 2nd series of white paper that provides a brief and Data specific assessment of the current networks with respect to the customer experiences while using the 3G & 4G mobile network in Islamabad. This time we have added the highly anticipated LTE networks by Zong and Warid as well. This QoS audit is also important as most operators have completed their initial roll-outs and optimizations, subscriber base is greater than it was 03 months back when first audit was conducted. It compares improvements/degradations for each operator about maintaining their services for the end users.
This QoS audit is also important as most operators have completed their initial roll-outs and optimizations, subscriber base is greater than it was 03 months back when first audit was conducted.
Principally, periodic benchmarking or audit of the QoS evaluation is important. It helps to evaluate customer’s experience and secure a competitive edge in the market. The QoS standards are setup to make sure that the costumers are provided satisfactory level of services and ensure that operators meet minimum acceptable standards of service.
Regulator has its role to emphasize on the quality of the services, as mentioned in the QoS parameters in their NGSMA licenses to be met by the operators. The next QoS Audit is expected to be jointly undertaken with PTA and industry partners in more detail for other cities as well. With our results and feedback from the end users, the networks irrespective of the operator can improve further where required. The mobile network operators have to ensure quality of service for which regular performance survey and measurements of quality parameters of their networks is conducted on regular basis.
1.1 Network Audit
The objective of the network audit is to understand and to indicate the potential issues faced by the network and enable operators to improve and to maintain a good balance between coverage, capacity and quality. QoS parameters are universally accepted Key Performance Indicators (KPI) that serves as the guiding principles for professionals to analyse and keep network services within specified threshold for the desired QoS criteria.
This QoS Audit report, as mentioned earlier, is based on the same drive route in Islamabad and for the key performance indicators mandated by Pakistan Telecommunication Authority (PTA) in its NGSMA license for comparative consistency. The QoS benchmarking was performed using standard network audit tools and commercial devices in two phases using the following standard procedure for such an exercise.
- Log Collection. >> 3G and 4G
- Post Processing. >> 3G and 4G
- Analysis and comparison of the findings
1.2 Test Execution
The QoS Audit was conducted during the last week of February, 2015. For 3G network drive test was performed for two operators at a time. 3G Logs were collected for Zong and Mobilink during afternoon hours, the first day; and the same route and time was selected for Ufone and Telenor. For LTE though, log collection was done during the morning hours for Warid and Zong networks. All the tests were conducted in mobility and the area covered the sector of F-6, F-7, and F-8 and Blue area in Islamabad. These areas are considered to have a good coverage.
The network assessment can be quite different for other cities. Our goal here is to make a more objective and technically accepted assessment to educate interested professionals, regulator, vendors and consumers about the current operational state of the 3G and 4G networks in Islamabad. The figure shows the drive route on which tests were conducted.
2. QOS Methodology, Technology & KPI’s Explained
2.1 QoS Methodology
In our audit report we monitored the quality of data related services only. Please note the following scenario/comment for interpreting the results;
- Two Samsung Galaxy S4 i9505.
- The devices were locked in the specified network mode preference.
- The tests were performed in mobility over a specific route.
- Our findings and conclusions may not be 100% consistent to the consumer experience.
2.2 Operators Spectrum & Technology
While, we have provided this in our earlier report but for those who have not read our previous report, we will briefly explain the spectrum available with the operators and the technology deployed.
3G (Ufone and Telenor): Ufone and Telenor have 5MHz bandwidth in 2100 MHz spectrum that enables them to implement Rel-7 (HSPA) of UMTS Technology. Under ideal conditions it can achieve a peak data rate of 21 Mbps (64 QAM) for downlink and 11 Mbps (16 QAM) for Uplink.
3G (Zong and Mobilink): Zong & Mobilink have invested more in the spectrum and have won 10 MHz of bandwidth each in 2100 MHz spectrum that allows them to deploy Dual Carrier HSPA which can provide them with a peak data rate of 42 Mbps in downlink and 11 Mbps in uplink.
LTE (Warid and Zong): In addition to Spectrum in 2100 Mhz for 3G, Zong also acquired 10 MHz in 1800 Band that is the spectrum widely used for LTE deployment worldwide. Warid also has deployed LTE in the same band but they have done so in limited bandwidth (reportedly from industry sources that have made use of only 3 MHz for their LTE deployment). They have re-farmed their existing 1800MHz band and have LTE with 2G services. Their performance of the LTE Data has been included in the current report.
The table below shows the Spectrum used by operators in Pakistan for 3G/4G deployment from the NGMSA auction and the refarmed spectrum.
2.3 QoS Key Performance Indicators
To analyse this QoS, we have focused on the following KPIs while collecting the logs over the drive route. These KPIs are specifically collected for conducting audit of their Data Services (previous QoS audit also included voice services).
- 3G Data Throughputs
- Downlink Throughput
- Uplink Throughput
- 4G Data Throughputs
- Downlink Throughput
- Uplink Throughput
- 4G Service Coverage Area
- RSRP: Received Signal Received Power
- RSRQ: Received Signal Received Quality
- RSSI: Received Signal Strength Indicator
LTE Service (KPIs): *In cellular networks, when a mobile moves from cell to cell and performs cell selection/reselection and handover, it has to measure the signal strength/quality of the neighbour or adjacent cells. In LTE network, a UE measures two parameters on reference signal:
RSSI – Received Signal Strength Indicator: The carrier RSSI (Receive Strength Signal Indicator) measures the average total received power observed only in OFDM symbols containing reference symbols for antenna port 0 (i.e., OFDM symbol 0 & 4 in a slot) in the measurement bandwidth over N resource blocks.
The total received power of the carrier RSSI includes the power from co-channel serving & non-serving cells, adjacent channel interference, thermal noise, etc. Total measured over 12-subcarriers including RS from Serving Cell, Traffic in the Serving Cell
RSRP – Reference Signal Received Power: RSRP is a RSSI type of measurement.
It is the power of the LTE Reference Signals spread over the full bandwidth and narrowband.
A minimum of -20 dB SINR (of the S-Synch channel) is needed to detect RSRP/RSRQ
RSRQ – Reference Signal Received Quality: Quality considering also RSSI and the number of used Resource Blocks (N) RSRQ = (N * RSRP) / RSSI measured over the same bandwidth. RSRQ is a C/I type of measurement and it indicates the quality of the received reference signal. The RSRQ measurement provides additional information when RSRP is not sufficient to make a reliable handover or cell reselection decision.
In the procedure of handover, the LTE specification provides the flexibility of using RSRP, RSRQ, or both.
It must to be measured over the same bandwidth:
- Narrowband N = 62 Sub Carriers (6 Resource Blocks)
- Wideband N = full bandwidth (up to 100 Resource Blocks / 20 MHz)
3. Quality of Service (QOS) Audit
It should be noted that the level of reporting of service quality may differ or will not be exactly comparable with consumers own experience or by operators own engineering team.
While, we have provided clarification for 4G KPI’s, for details regarding 3G KPI’s please refer to our previous white paper “Pakistan’s first 3G Mobile Service Benchmarking” on Phone World website. Since, LTE was launched recently and not included in our previous report therefore we have provided some additional information on its parameters above that are generally used to assess the data rates of downloads and uploads over the route.
3.1 3G Data Performance
For data performance analysis large files were downloaded via dialup connection. The transfer rates recorded using the tools varied according to the changes in factors like RSSI, RSCP and CQI for 3G; and RSSI, RSCP and RSRQ determined for 4G. FTP was used to download the files. Below is a brief interpretation of throughput for each network. The values are all recorded in mobility.
The graphs above shows the throughput values of Mobilink network. The first graph shows the through-put obtained on Oct 2014 while the second graph shows the through-put obtained on Feb 2015. During the first benchmarking exercise there was no data bundle activated because Mobilink offered free trail to its users trying to entice customer to use data.
During the 02nd benchmarking exercise Mobilink has officially launched its 3G services data bundles and the data users have grown in addition to their appetite for using data on the go. During the second run large number of users in the same sector may have contributed to low RSCP and CQI which eventually resulted in low throughput especially in the beginning of the route. The max value achieved was 16.6 Mbps (15Mbps @ Oct’14) while the average value achieved was 2Mbps (3Mbps @ Oct’14).
Ufone showed more consistency from the beginning to end unlike the previous audit. Apart from the initial period it showed good behavior throughout the route. The max throughput achieved was 15.19Mbps (16.9Mbps @ Oct’14) and average values achieved was 2.12Mbps (6Mbps @ Oct’14). The declining average throughput by more than 3 times should be of concern to Ufone when compared with just 3 months back. However, even this value is equivalent to Mobilink throughput that has twice the spectrum, which shall be of concern to Mobilink as well.
The Peak throughput achieved for Telenor was 14.1Mbps (18Mbps @ Oct’14) while the average value achieved is 4.0Mbps (5Mbps @ Oct’14). Some of the values achieved for throughput are low because of low RSCP which resulted in low CQI. The uneven rise and fall of the graph indicates the continuing communication between UE and Node-B to manage assigning of the respective CQI to avoid disconnection and continuous through-puts, which is requires network to be properly optimized
The graph for Zong shows an overall consistent performance. The network behaviour was very good although the max and average values are less than the one measured last time which may have occurred due to an increase in customers. The highest peak achieved was 18.5Mbps (21Mbps @ Oct’14) while the average value achieved was 5.28Mbps (6 mbps @ Oct’14).
Also, it must be encouraging for Zong and its subscribers both to note that the drop in the average throughput was not as steep as is seen for other operators.
3.2 Average Downlink (DL) Throughput Comparison for 3G
For the calculation of downlink throughput, files were downloaded from the server and its downlink rate was measured. Since, it was in mobility, the CQI, RSCP and RSSI changed from location to location due to change in noise levels and possibly other available traffic at that time. The chart shows a summarized comparison of the average DL data throughput performance of each operator during the previous and the current testing.
It can be seen that while for all operators the throughput has seen downward trend but the decrease has been nominal for Zong and Telenor that takes the lead in achieving higher average downlink throughput of 4 & 5 Mbps respectively. While, Mobilink throughput decreased as compared to last QoS audit but the most significant degradation was that for Ufone, that showed a drastic decrease and achieved around 2.12 Mbps from its previous value of 6Mbps.
3.3 Average Uplink (UL) Throughput for 3G
For the calculation of uplink throughput, again specific files were uploaded on the server and its uplink rate was measured. Due to mobility and communication between user and Node-B, it had to force the Node-B and the UE to select the best possible modulation to provide best achievable throughputs. It is the inherit property of the network to schedule data rates among the users judicially.
For uplink throughput evaluation large file were uploaded to the FTP server from UE throughout the drive route. The graph compares the average upload through-puts that were achieved during the drive test.
From the graph it can be observed that while uplink through-put for all operators decreased, Mobilink was able to achieve the best uploads with an average of 1.5 Mbps, while Telenor has 2nd best throughput value of 1.47 Mbps. Zong witnessed a very drastic decrease in its uplink throughput of over 130%, which is a serious issue for the operator to support its active data users. Ufone on the other hand achieved an average throughput of 1.2 Mbps which is quite lower as is the case for their downlink as well than the throughput achieved previously.
3.4 CQI Value Comparison for 3G
Channel Quality Indicator (CQI) is the value calculated collectively by the User Equipment and Node-B indicating how fast or slow the UE can communicate with the network. In downlink and uplink the CQI values ranges from 0 to 30, where 30 indicates the best quality channel and 0 or 1 indicates the worst quality channel. This parameter is checked mainly for data services. The figure shows the availability of good CQI i.e. between 18 and 30 for each operator in terms of percentage. For ease of understanding for non-technical managers, the higher the percentage an operators remain within the given limit of 18 to 30 the better is the network performance for Data Performance.
It is clear from the graph that each operator has improved its good CQI for the proposed drive route. It also adds to the better data rates achieved for each operator as compared to its previous data rates in October 2014.
For detailed distribution we have added break down of the CQI values for each operator below, which is self-explanatory.
The CQI for Mobilink has improved and shows higher CQI assignment over the route as seen from the bar chart when compared with line chart. Similarly for Mobilink the CQI distribution is almost the same but comparatively have less contribution from good CQI values.
Also, as is seen from the graph that overall for Zong the average CQI lies in good region which is clearly supporting the good downloads as its Lower CQI has very low contribution to the network.
Telenor had further improved its CQI values and therefore resulted in being the second best network in terms of downloads.
Similarly, Ufone has also improved its CQI assignments which are evident from the graph but they still need to improve upon their resources for better data rates than before.
*For further details about CQI please refer to the previous white paper.
3.5 Average Throughput Comparison for 4G
4G data throughputs were performed on the same route. The summarized Upload and Download throughputs are shown in the graph. The values are less than 10 Mbps for both uploads and downloads which should have at least been achieved considering that there are not many users on the 4G network at the moment. This low Data throughput could be due to lesser number of e Node-Bs in the sector.
For testing the Data Through-put, files of around 800 MB were processed more than twice during the testing. Considering the fact that these networks are still getting optimized for better performances; such values are acceptable for a common user. One more thing that should be taken into consideration is that the above readings are taken from mobility. So it should not be compared to stationary values as those values are very high in comparison to the mobility.
Other factors adding to the low data rates are the number of hops from server to server and the resources available for the files to get transferred. For Example while using HTTP transfers, the site used for transfers may have issues or may have too many users to support at the same time. So the site schedulers reduce the resources for each user to keep a common rate for each one of them. This issue does not arise during mobility because on closed traffic signals or other short breaks, an increase in Uploads and Downloads was seen which means that the rates increased during stationary period.
Secondly, it is important to note that both Warid and Zong may have a reasonable customer base that are heavy users in Islamabad. Therefore, the transferring data may be due to OTA updates, emails, web-surfing, streaming, active navigation etc., and most importantly internet hotspots. So both the operators, have to allocate their network resources to enable equivalent rates for each user. In this case the network assigns rates in a way that people near or away from the Node have acceptable rates in that particular sector. So at this stage the RSRP, RSCQ and RSSI and the UE communication begins to decide the best Resource Block for that particular customer.
Also, the throughput are a bit different than we see for 3G networks where the downlink throughputs are greater than the uplink throughputs. The unusual difference between upload and download could also indicate towards requirement for better optimisation.
The purpose of this QoS benchmarking was to compare the existing four 3G and two LTE operators in Pakistan. This QoS benchmarking reports the comparison of the five operators by using standard KPIs for the most popular services like data download and upload.
Other factors adding to the low data rates are the number of hops from server to server and the resources available for the files to get transferred.
After performing the second benchmarking we were able to have an overview of the improvements that have been made and places where things can be further enhanced. From the summarized graphs of 3G throughputs we can observe that Zong is best in downloads with Telenor still remains second best choice in terms of throughputs. While, Mobilink has the best upload through-put. Though both download and upload are showing lower values than the first runs yet their CQI are distributed evenly to maintain better rates throughout the route. For 4G LTE Zong took the lead again and achieved higher throughput in both Uplink and Downlink as compared to its competitor, Warid.
During the first benchmarking there were not many active users in the sectors therefore higher data rates were observed. But now with time more and more users are joining to consume what 3G and 4G networks can offer. Overall data through-puts are getting shared among them. Even now the networks are still going through optimization and it will keep on improving further. The LTE network still need improvement and we can expect better results in the near future.
The unusual difference between upload and download could also indicate towards requirement for better optimisation
Last but not the least the significance of QoS survey can never fade away. As it is one of the only impartial and transparent ways to identify the operator services. It not only helps the end user but the operator also become aware of their network performance. So the objective is to keep a healthy environment between users and operators. Additionally, Regulator plays a vital role; to voice and protect customers’ concerns and complaints, to keep and maintain competition between the operators and to achieve their goals.
QoS Benchmark of 3G/4G Mobile Data Services : Download