Pakistan’s first 3G Mobile Service Benchmarking
In order to achieve a sustainable competitive upper hand in the telecom industry, customer satisfaction is important which makes the working of the Quality of Service (QoS) evaluation vital. The QoS standards are setup to make sure that the costumers are provided satisfactory level of services and ensure that they meet the minimum acceptable standards.
This article provides an objective assessment of the current network status and resultant customer experiences while using the 3G mobile network in Islamabad and is evaluated in terms of both voice and data connectivity. The scope of current Quality of Service (QoS) audit does not include the much anticipated LTE service or 4G network (as branded) that was recently launched by Zong in seven cities. As the operators are still in the process of deployment and optimisation of their current 3G networks and there shall be drastic changes in the QoS over the next few months. Still, customer first experience is very important and their network QoS on continued basis is equally essential to achieve a sustainable competitive edge in the market.
Further, one of the key role and focus of the regulators is and should be on the end user satisfaction and for this purpose Pakistan Telecom Authority (PTA) has also provided some specific QoS parameters in their NGSMA licenses to be met by the operators. To ensure deployment of new services to fulfil customer demands with consumer satisfaction, operators also need to invest both time and money on the constant development and optimisation of their networks. The objective of the network audit is to understand and to indicate the potential issues faced by the network, which will enable operators to improve and to maintain a good balance between coverage, capacity and quality.
The exercise was conducted over a period of four weeks between August-September 2014 at peak hours during weekdays, to check performance and quality of the newly deployed licenced 3G operators Zong, Mobilink, Ufone, and Telenor. Warid was left out as it had not deployed 3G network. All the tests were conducted in mobility and the area covered the sector of G-6, F-6, F-7, F-8 and Blue area in Islamabad. These areas are considered to have a good voice and data traffic users.
The figure shows the drive route on which tests were conducted. This graph was taken from idle mode and the places where the bins are distant are due to GPS not synching at those points. Bins are the data-points that the drive testing tool writes for that specific time. It records the information from the GPS, the device and respective network.
Our goal of this survey is to educate readers belonging to operators, regulator, vendors, professionals and consumers as it provides a neutral and objective assessment of the current operational state of the 3G networks in Islamabad i.e. considered a model city. Therefore, the network assessment could be quite different for other cities and that will be provided from time to time.
Quality of Services depends upon various factors which may not be limited to voice, data or service area. Our methodology used to monitor the quality of services includes the major service coverage, voice and data KPI’s. Please note for interpreting the results;
- Particular handsets of CAT 14 were selected.
- The tests were performed in mobility over a specific route.
- The findings from the survey should not be treated as recommended best buys.
- Value Added Services (VAS), price plans, customer care and support are not presented in this article.
- Our findings and conclusions may not be 100% consistent to the consumer experience.
The User Equipment (UE) or mobile/cell-phone is the interface between the user and the 3G network. Keeping in view the numerous facilities and applications that it can perform, it was decided that it should be called user equipment rather than mobile or cell-phone. Since, there are 3G WI-Fi devices which are not cell-phones but consumers use them for their mobile/cell-phones also known as smartphones.
Apart from the technology releases of the network there is also breakdown among the User Equipments that depends on their data throughputs as shown in figure below. During the QoS audit category 14 devices were used which means that the maximum throughput that could be supported on such a device was 21 Mbps.
To analyse the QoS, the focus on the following KPIs is mandatory while collecting the logs over the drive route. These KPIs are for Coverage Area, Data throughputs and Voice calls.
· RSCP: Received Signal Code Power
· CQI: Channel Quality Indicator
· CSSR: Call Setup Success Rate
· CDR: Call Drop Rate
· CSR: Call Success Rate
· Downlink Throughput
· Uplink Throughput
It should be noted that the level of reporting service quality may differ or will not be exactly comparable with consumers own experience.
Service coverage of mobile networks is based on Received Signal Code Power (RSCP) which refers to the ability of a network to achieve a minimal signal strength of -100 dBm or higher. Higher the RSCP better the network performance. In order to achieve good network coverage the RSCP should be having greater value than -100 dBm. On the average -45 dBm is the best possible signal strength achievable while -100 dBm is considered as the poorest. For simple comparison, the RSCP graph was divided into three categories namely Good, Average and Bad. The Good RSCP lies between -45 dBm to -65 dBm, the average RSCP lies between -66 dBm to -95 dBm while -96 dBm and lesser is considered as bad RSCP.
The graph below is the coverage plot that shows the coverage of the networks on the course of the drive route. It was observed that three operators i.e. Mobilink, Ufone and Zong provided good service coverage on the drive route area, while Telenor provided an average service coverage.
The average RSCP graph of the network during the whole drive route shows that Zong, Ufone and Mobilink has greater than -61 dBm value considerd as very good value. Telenor on the other hand achieved -82 dBm on average.
Since, PTA recommends that the RSCP values should be greater than -100 dBm, therefore it can be said that all the four 3G mobile networks do satisfy the regulatory conditions or the minimum limit set by the regulators.
Channel Quality Indicator (CQI) is important parameter for service quality. It is the value calculated collectively by the User Equipment and Node-B indicating how fast or slow the UE can communicate with the network. In downlink and uplink the CQI values ranges from 0 to 30, where 30 indicates the best quality channel and 0 or 1 indicates the worst quality channel. This parameter is checked mainly for data services.
The network transmits data with different transport block size depending upon the CQI values which has been reported by the UE. If UE reports high CQI value the network transmits the data with large transport blocks whereas if the UE reports lower CQI values the network transmits the data with small transport blocks. In case of error, if the UE reports low CQI value and the network sends it large transport block then it is highly probable that the UE will fail to decode the block.
The figure below shows the CQIs that were reported by the UE during the drive test. It was observed that overall for each operator average CQI lies in good region i.e. above 18 CQI value.
The average CQI received from Telenor was the best among other operators and was 24. Whereas, Ufone and Zong received an average CQI of 20, Mobilink recorded a value of 21. The figure displays the average CQI values of the four networks.
As seen the Average CQI values recoded are quite good throughout the drive route.
Voice performance involves multi KPIs based on its test scenarios. The test performed only checked for the call terminating and releasing cycle. For short calls the conversation time was kept for two minutes. For long calls the call in conversation were kept for the whole route. It helped in evaluating the performance of the voice calls of each operator. In this audit a call is considered successful when the call is setup successfully in the standard time period with the conversations starting and ending successfully till released. Based on this process call performance is evaluated. The three main KPIs that are used for voice performance are: Call setup success rate (CSSR), call success rate (CSR) and call drop rate (CDR).
Keeping in view the timelines and log processing time required for all the operators the max amount of short and long call(s) were attained for the duration of the route. Approximately, 30 calls of 2 minutes each were made. All the calls were able to terminate and release calls successfully and completely. Which means the CSSR and CSR was excellent i.e. 100% while CDR was 0%. These terms are defined in the following paras for convenience.
Call setup success rate measures network capability of setting up a call successfully in a specified time period. This KPI is the ratio of total number of successful call established to the total number of call attempts.
The CSSR observed for each operator in this QOS is 100%. The graph below shows the result obtained for CSSR.
Call setup rate measures the calls that were successfully setup and normally terminated. This KPI is evaluated by both CSSR and CSR to give an overall end to end view of the total calls successfully established. Note that high call success rate is desirable.
The CSR observed for each operator is 100 %. Below is the graph for CSR.
Another common KPI used worldwide is Call drop rate (CDR) which measures the call drops during the sessions. As it can be clearly seen from the above graph that call success rate is 100% so no CDR was observed during the session.
3.3 Data Performance
Data performance or data throughput is one of the most important factors from user’s perspective especially for 3G/4G network. As discussed already the scope of the existing audit is restricted to 3G network only (4G network audit will be conducted shortly as it was not available at the time of this audit). Since, Pakistan is one of the rising countries with increasing smart-phone penetration, more and more consumers are getting addicted to these devices. Currently, smartphones are exhaustively used for the following but their use in future will include other applications as well;
- Emails configured on their devices;
- Texting application installed;
- Social applications for sharing large picture, audio and video files
The use of data services on the move, where there is no restriction of being tied to just Wi-Fi, will essentially increase the usage of data over respective networks. So being independent of a stationary access point the user is mostly devouring the throughput while on the move. Apart from social applications the OEM of these Smart-Phones also provide updates for their expensive devices. Popularly iPhones, Samsung Nokia, Sony and HTC provide their customers with Over the Air (OTA) updates. These updates are usually larger than 500MB that cannot be downloaded unless they have interruption free service to update their devices. The consumers usually prefer to compare the amount of data rates they usually get on their devices specially with friends & family. As professionals a more authenticated and scientific comparison of these data services is important. So to check and report about the available quality of the recent 3G networks, each of the operators is tested for downloading and uploading large files. These large files were transferred in a continuous stream of downloading and uploading around 4GB and 2GB data respectively. The purpose was to force the nodes and devices to maximum throughput achievable.
During the audit process the largest data packages were checked for the operators and that too, over the entire drive route as per industry practice. Due to large data throughputs for Uplink and Downlink, heavy data download was recorded for analysis and reporting. This data helped measure the actual throughput for each operator. The log collection was completed in more than ten rounds and identified blind spots, weak areas and worst places for each operator. In the following figures and graphs the reader will observe the operators having achieved their best possible data rates. Technically, the date rates are measured in bits per second (bps) but one can observe that for general public or user throughputs are shown in Bytes per second (Bps). Please note that bytes is just 8 times multiple of bits.
For data performance analysis large files were downloaded as mentioned before via dialup connection. The transfer rates recorded using the tools varied according to the changes in factors like RSSI, RSCP and CQI. FTP was used to download the files. Below is the analysis of throughput for each network.
Please keep in mind that the following values are all recorded in mobility.
The graph below shows the throughput values of Mobilink network. The values were recorded using free 3G internet. Due to free trial of the internet it is expected to have a large number of users in that sector. It can be observed from the graph that rates started rising after around 15 minutes. The large number of users in the same sector contributed to low RSCP and CQI which eventually resulted in low throughput in the beginning of the route. The highest peak achieved was 15mbps while the average value achieved was 3mbps.
In the beginning the throughputs remained consistently low. However, moving into better coverage area one can observe the rise in the graph against time. Apart from the initial period it showed good behaviour in throughput values specially during the latter part of the route. The highest peak achieved was 16.9 Mbps and average values achieved were 6 Mbps.
Looking at the graph below it can be observed that overall the values are satisfactory. The Peak throughput achieved was 18 mbps while the average value achieved is 5 mbps. Some of the values achieved for throughput are low because of low RSCP which resulted in low CQI. The rise and fall of the graph indicates the communication between UE and Node-B, and the assigning of the respective CQI to avoid disconnection and continuous throughput.
The graph for Zong shows a consistent behaviour overall. The trend shows that overall the network behaviour is very good. The highest peak achieved is 21mbps while the average values achieved is 6 Mbps.
To get an overall overview of the data performance of each network operator, a comparison has been shown in the graph below. The graph compares the average download throughputs that were achieved during the drive test.
It can be seen that Zong and Ufone takes the lead in achieving the average downlink throughput which is 6 Mbps. Telenor and Mobilink on the other hand were able to achieve 5 Mbps and 3 Mbps respectively.
3.3.3 Uplink Throughput
For the calculation of uplink throughput specific files were uploaded on the server and measured its uplink rate. Since, in mobility therefore the CQI, RSCP and RSSI changed from location to location due to change in noise levels and possibly other available traffic at that time. So it had to force the Node-B and the UE to communicate and select the best possible modulation to provide best achievable throughputs. It is the inherit property of the network to schedule data rates among the users judicially.
For uplink throughput evaluation large files were uploaded to the FTP server from UE throughout the drive route. The graph compares the average upload throughputs that were achieved during the drive test.
From the graph it can be observed that Zong takes the lead in upload throughputs and achieves average throughput of 2.3 Mbps, while Telenor has 2nd best throughput value which is 2.15 Mbps.
Following is the summary regarding the values achieved by each operator viz-a-viz PTA defined parameters for the mobile operators as per their license obligation. The PTA column is the minimum value required by each operator. It is observed that all the 3G operators do fulfil the minimum standards defined for its consumers.
The above mentioned audit is for the existing four 3G operators in Pakistan (it does not take into consideration recently launched Zong LTE services). This article reports the comparison of the four operators by using standard KPI for the most popular services like Data download, upload and CS Calls both MT and MO. The results showed that all four networks in their present state performed well in achieving their regulatory targets in voice calls. All the networks showed 100% Call Success rate with no block and drop calls. The networks were mainly tested for the newly announced data packages that the Telecom customers in Pakistan will experience on 3G. Again all the networks show good performance of download and upload data throughputs, with Zong and Telenor achieving slightly better data rates along the routes.
Finally, it should be distinguished that QoS survey should be conducted on a periodical basis. The results may vary from the actual conditions as the networks are mainly going through the optimization and looking for further improvements beyond their regulatory compulsions specially to manage the increase in traffic by users. Currently, the QoS services are monitored by the operators and the results they obtain are kept classified. To reduce customer confusion and complaints about the services to increase their satisfaction, the regulators should conduct the assessment of networks and services. This could be achieved by either planning regular benchmarking studies or subcontracting benchmarking works to third parties, and then report some relevant outcomes and findings to the public for the development of the industry as well as consumer awareness.
DOWNLOAD THE FULL VERSION HERE