France Telecom Orange Discussion Papers on Bill & Keep The Technical Impact of Mandatory Bill And Keep (BAK): BAK would imply high involvement of NRAs in controversial network operation issues
This short paper explains that if NRAs impose mandatory BAK the result will require an increase in regulation, rather than less regulation. If operators are forced to interconnect on a bill and keep basis, complex and numerous disputes will arise. Operators are currently obliged to interconnect under current electronic communications regulation in Europe, but are able to charge a regulated interconnection fee. Voluntary BAK exists between peers among Internet carriers world-wide but are less common than transit paying arrangements. However these internet carriers are not mandated to interconnect and interconnect on voluntary BAK basis. Mandatory BAK, on the contrary, has never been observed on a large scale. In this paper we develop concrete examples showing that mandatory interconnection and BAK does not simplify the process of interconnection and payment and therefore cannot lead to improvements in efficiency. Instead, mandatory BAK would require regulatory intervention to resolve disputes between operators.
1) Anyone can interconnect Apart from current operators in the sector who are already interconnected, other actors (typically not from the communications sector, eg private companies) will be interested to take advantage of interconnection because of the free access to networks under mandatory BAK. Due to the availability of protocols like ISUP1 there is no longer a technical barrier to interconnection. For example, in France 800 actors have notified ARCEP as providers of electronic communications networks and services and can therefore ask for a connection, even if some have no clear activity in the sector, while at the moment, fewer than 200 are connected to the France Telecom network. Telecommunications operators have already seen many private companies requesting interconnection not for the purpose of selling public telephone services on the market, but rather to cover their own needs. Even though they have very asymmetrical traffic profiles, Broadcasters have asked for BAK interconnection. In this way, any large company outside of the telecoms sector can request interconnection, if necessary by creating an ad hoc subsidiary, in order to be granted a BAK status and thus benefit from free telecommunication services from network operators. Interconnection would enable such an actor to originate and send traffic from a virtual private network (VPN) without bearing any of the cost of the network infrastructure used to transmit the call. Wireless operators of all sorts and activities could ask for interconnection, content providers, content distributors, of all kinds and sizes, producing all sorts of traffic and volume could ask for interconnection including IPTV providers and VOD providers.
ISUP defines the protocol and procedures used to setup, manage and release trunk circuits that carry voice and data calls over the public switched telephone network. ISUP is used for both ISDN and nonISDN calls.
Due to the double obligation of interconnection and BAK, new candidates for interconnection will bring traffic but not the financial resources necessary to maintain and develop the network infrastructure, generating network congestion and, consequently, quality problems. Due to network over-burdening and lack of investment, the regulator will have to issue a list of criteria for a company to be connected in order to limit the problem. An economically sound solution could be based on a minimal interface capacity or on a guarantee of symmetrical arrangements. But this will lead to litigation on the grounds of discrimination or regulatory capture. BAK can lead to arbitrage if the same terms and conditions are not applied to all operators. It was the case when BAK existed between the French mobile operators (up to 2004). Some fixed operators disguised fixed-to-mobile traffic into mobile-to-mobile traffic in order to benefit from the free BAK agreements between mobile operators. As a consequence these gateways generated local overload and an inefficient usage of the frequency spectrum and regular users of the radio spectrum were disadvantaged.
2) Who will decide where the point of connection is located? In the context of mandatory BAK the question of where the physical point of interconnection should be remains an open question. Some interconnection points are highly connected hubs with high direct link capacities to all national and international routes, while others are only connected to major national and international routes through congested intermediate nodes and links. Obviously, interconnection seekers will request to be interconnected to the highly connected hubs while interconnection providers will propose the intermediate hubs if they consider that Bill and Keep interconnection is not an equitable deal for them. It is then not clear who should build the infrastructure and who should cover the direct cost of interconnection. With no return value from the interconnection point or the transmission and switching equipment, there is no rationale for investing in the network. The tendency will be to minimize costs or to transfer the cost to the others in a form of hot potato routing as a call is transferred as quickly as possible from one network to another to minimise use of an operators own network. Without defined operational processes the mandatory interconnection and BAK mechanism will generate conflicts which the regulator will be frequently requested to solve. In the meantime, no satisfactory service will be available for customers. 3) Who will decide the capacity of the interconnection? When two interconnected operators cannot use price to adjust their interconnection agreement they use interconnection capacity as a negotiation tool. The access seeker will ask for the maximum, but, without incentives, the access provider will offer the minimum. This will lead to disputes and to congestion at the connecting point; however, congestion can and will spread throughout the networks through the following phenomena: (1) when a direct route is congested, routing algorithms try indirect routes, hence the average number of links and nodes per communication increase, this inflates the amount of traffic to be carried by network elements and produces new congestion, which in itself implies even more indirect and inefficient routes and so on, (2) in a congested network, calls or packets are lost and are thus repeated at the source of the traffic until they reach their destination, therefore overall traffic increases.

In a congested network, it is extremely difficult to identify the original cause of congestion. It is very likely that increasing capacity somewhere will generate congestion elsewhere with no improvement of end to end performance for customers. It is difficult therefore, to define where capacity provisioning would be necessary. If mandatory BAK eliminates price as an adjustment factor, the only remaining adjustment factors are quality and capacity. This phenomenon was very common in the bilateral national agreements related to international trunk groups. When an operator disagreed with a proposed tariff, considering that it was not equitable due for instance to the unequal volumes of exchanged flows, the consequence was often a reduction in interconnection capacity. IP traffic on core networks continues to grow by around 40% per year on average. Therefore, it is necessary to continually invest to guarantee a satisfactory interconnection capacity and to adjust the necessary technical resources. Without financial compensation for interconnection, the system will lack any incentive to invest. We are already facing this situation in France: broadband access is sold at the best capacity the (existing) line can offer. This results in cheap flat rate prices, but if these retail flat rates were combined with BAK then there would be no economic incentive to bring higher broadband capacity to customers with low bandwidth eligibility. On the other hand, with positive MTRs, mobile coverage was achieved without any government intervention: installing a new base station clearly and automatically meant more revenue.
4) Routing and metering problems occur when different traffic flows have to be identified creating extra cost Today, several large operators are pure transit network operators2. They face network costs but with mandatory BAK they would not earn any revenue. Therefore, these activities and the corresponding resources, critically necessary for the service to the customer, will disappear. Even if transit companies are exempted from BAK obligations, they will be in competition with the transit part of end to end operators providing access and termination. If the latter have a general obligation of BAK, their transit service would be available for free, this will kill the business model of transit operators. It could be said that a solution would be that integrated operators have the right to price transit while being obliged to propose termination for free. However this leads to two types of problems: one economical, where the frontier between transit and termination is unclear. This problem is as controversial and in the end identical to the question of defining relevant costs for termination prices. This point is addressed in the specific economic paper.3 one practical related to routing and metering questions which will be developed below. The same interconnection point will serve for transit traffic and for free termination service. In theory, only traffic flows that are addressed to customers located on the last segments behind the Point of Interconnection would "benefit" from BAK. But the issue arises of what to do with

Transit is necessary to complete a call notably on long distance: for example, thanks to the Transit operators, the European Internet users can access an USA websites. 3 This point is addressed for instance in an economic paper by Professor Mason.
traffic flows addressed to other destinations, as these could be simply rejected or rerouted to their destinations, as far as the system is able to differentiate the traffic and the transit flows. An operator that transmits a flow does not know if the flow is related to transit or termination, so he does not know if it is free or not, creating a source of conflict. With the internet, the only thing that the routing tables know is that the use of an interconnection point will bring the traffic closer to its destination, with no distinction of the flows because the packets are aggregated. Then it would be extremely difficult to have different prices depending on the flow; so, because the transit activity must be compensated, the termination one must be compensated as well. Filtering calls has a cost, and there is little incentive for an operator to engage additional computing or network resources to process traffic flows for which it is neither the source nor the destination. Rejecting traffic is always the source for many disputes (cf the problem of "phantom traffic flows" in the USA). In the case of rerouting, fairness would imply that operators who submit the rerouted traffic flows cover the cost of this rerouting. To calculate these costs, a traffic metering process must be set up to count how many calls, sessions, minutes or bits are sent by a given operator to a given destination. Then the apparent advantage of lowering interconnection transaction costs that is put forward by BAK advocates suddenly disappears. Even with BAK, a sophisticated metering process must exist at the interconnection points; therefore we cannot expect any savings on the transaction costs. In any case, interconnection traffic metering is still necessary for destination-paid traffic flows (like 0800 numbers) and all value added traffic flows. If this extra cost cannot be compensated, no positive discrimination will be achieved between the two types of flows leading to disputes on the definition and the volume of the termination and transit traffics. In the end, ultimately, the transit activity could disappear.

5) SPAM will increase If termination is essentially a free of charge service through BAK, traffic will increase even more due to unsolicited calls creating spam for consumers. It would surely be a nightmare for all customers if, as for their email box, most of the phone calls they received, day and night, were unsolicited. Moreover, customer voicemail or answering machines would be rendered totally useless, as it is much more difficult to browse through an even lightly filled vocal mailbox than it is through an email spambox. As for vocal or multimedia content filtering, supposing it conforms to legislation, and that prior consent from the user is obtained, it would be incomparably more difficult and costly to develop and deploy than email text-based filtering.
6) Conclusion: mandatory BAK will lead to poor performance for customers and to high levels of litigation We can see that conflicts will arise at each level of the interconnection process: due to congestion problems, with list of interconnection criteria to be defined by NRAs, points of interconnection to be defined, interconnection capacity to be allocated to be defined, transit activity to be protected,
quality to be maintained despite the lack of revenue/investment, Spam Network congestion will occur due to the increase of interconnected companies and the lack of resources to invest in the necessary equipment to upgrade the network. As previously seen to minimize this drawback, limits will be necessary and the regulator will have to define criteria to manage interconnection disputes. The operational process, from the interconnecting point issue to the level of capacity and quality offered to the interconnected parties, will also have to be defined in order to avoid the likely disputes between the stakeholders. The regulator will have to find alternatives for the lack of motivation from the network operators to invest in the networks. The lack of incentive to invest under BAK persists and it may be up to the regulator to determine alternative solutions to encourage investment. Building, maintaining, upgrading networks has a cost. Interconnection obligations as well as quality obligations have a cost. As long as interconnection is mandatory, the recovery of the consequential costs is necessary. The lack of a well balanced cost recovery mechanism will lead to arbitration using other levers such as capacity and/or quality, which will lead to frustration, complaints and legal procedures. In this context, no party will be satisfied, the network operators will be restrained in their network development, the interconnected parties will suffer from poor quality, low capacity and the regulator will face multiple complaints and disputes. Ultimately consumers will suffer as during the disputes, service will not be provided to the customer or very poorly, because disputes will concern how to technically operate the service. And when interconnection occurs; it is very likely that a vast majority of traffic will be junk traffic, including unsolicited spam.

EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 1 / 22
For example, broadcasting applications may require not only real-time decoding but also real-time encoding. If real-time processing is not critical, it may be possible for video to pass through the encoder more than once (i.e. "two-pass encoding") which increases the encoding efficiency significantly. Another example is video conferencing which requires very low delay. This requirement is contradictory to using frame stores (e.g. "B-frames") which may improve coding efficiency but introduce unwanted delay. Other requirements may involve error robustness, scalability and randomaccess editing. This article focuses on the performance aspects of internet video codecs and neglects evaluation of other codec characteristics.

General considerations

It is evident that a magic formula allowing us to determine the coding performance does not exist. The performance of a codec needs to be evaluated by physically measuring it using a suitable methodology. Some typical questions relating to codec performance and quality are as follows: How does video codec A compare with video codec B (e.g. how does Windows Media compare with RealVideo)? What is the quality improvement of generation Y in comparison to generation X for the same codec standard (say, Windows Media)? What is the difference in codec performance between implementation C and implementation D for the same codec standard (e.g. MPEG-4)? For a given bitrate available in the transmission path, which codec gives the best quality? Conversely, given a required level of quality (e.g. "transparent" 1 quality), what is the required bitrate for different codecs? Which codec performs best for "my" content (e.g. sport events, films, news bulletins, etc) Which codecs gives the most consistent quality across a range of different content? We will attempt to provide some informed answers to the above questions in the summary section of this article. Broadly speaking, evaluation methodologies are either objective or subjective. The former use mathematical models to mimic the behaviour of human visual systems, or can be based on feature extraction from a bitstream. The latter use a group of subjects (assessors) who are presented with decoded video pictures and have to judge the perceived quality using a tailored evaluation methodology. Objective tests can be used for quick and cost-efficient assessment of media quality. They are particularly useful for assessing the progress in codec design for a particular algorithm. However they have several drawbacks. They are strongly dependent on the type of codec used and the parameters chosen: they have very limited correlation with the subjective test results, especially at lower bitrates where distortions are high. Subjective methodologies are more time consuming, require more effort and are more costly than objective methods. But subjective methods generally give reliable and accurate results if correctly applied. Objective methods are not capable of providing the full truth about codec quality. If a conclusive decision is needed, we can only rely on real eyeballs. The process of evaluating codec performance subjectively can be summarized in the following simple steps:

1. Transparent quality is the quality of the codec which is indistinguishable from the uncompressed source quality.
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 2 / 22
1) Select the codecs under test and define their parameters (pre-filtering, buffering, key frame distance, etc); 2) Select the test sequences (uncompressed); 3) Define the coding conditions: bitrates 2, format resolutions; 4) Compress the test sequences to produce coded representations of the test sequences; 5) Select the evaluation methodology and establish a reproducible test environment; 6) Organize test sessions, invite the test subjects (assessors), present them with the decoded test sequences and ask them to determine the quality as they perceive it; 7) Collect the evaluation results, perform some statistical analysis on the voting data and remove inconsistent subjects; 8) Publish a test report.

The rationale for SAMVIQ

The television and multimedia domains differ significantly in a number of aspects, which may justify using different assessment methodologies [5]. Some of these differences are summarized in Table 1.
Table 1 Main differences between digital TV and multimedia domains

TV domain

Types of codecs MPEG-2

Multimedia domain

Open standards (e.g. MPEG-4, AVC) Proprietary codecs (e.g. Win-
The table shows that the multidows Media a, Real Video, media domain offers a large QuickTime) choice of parameters and uses Fixed (720 x 576 CIF, QCIF, SubQCIF, SIF, VGA, a variety of proprietary and Image format pixels for active SVGA, etc. standardized decoders and area) players, as opposed to television where the system parame- Rate of image Fixed (25 Hz) May vary from 0 to 30 Hz ters do not vary so much. refresh (frame Whereas in television the rate rate) of temporal refresh (update) of Standardized Various the image data is fixed, it can Decoder type vary significantly in multimedia Display type TV PC, PDA, mobile phone video sequences. In addition, a. Windows Media is in the process of being standardized within the SMPTE in the case of television, the and may become an open international standard known as VC-1. assessor's judgment of images is based on one axis of perception only: spatial sharpness of the images (i.e. quantization). In multimedia, however, the rate of updated video data is not constant. Thus, the assessors combine perception of the sharpness axis with perception of the fluidity of images. Because of this two-dimensional perception (i.e. fluidity and sharpness), it is far more difficult to evaluate the quality of multimedia images than those in the TV domain. Experience shows that this combination strongly influences the final subjectively-perceived picture quality. Furthermore, multimedia image formats may vary with the types and characteristics of the transmission network, whereas in television these are fairly constant. Another important difference between TV and multimedia is the viewing distance. In multimedia the viewing distance may vary significantly. In TV the viewing distance used for subjective evaluation is

2. Assuming constant bit rate (CBR) codecs in which the encoder produces a bitstream at its output which has a constant bitrate (using a suitable buffer size in order to average out the bitrate fluctuations).
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 3 / 22
well defined and can be between 4 H and 6 H (where H is the display height). In multimedia the viewing distance depends on both the image size (a combination of image and display formats) and the punctum proximum 3 which varies from one user to another. Consequently, almost any viewing distance can be considered for evaluations. In multimedia there is also a practical difficulty: multimedia images cannot be recorded directly on tape they are only available as data files. As direct stream does not allow for repeatable playout of the sequences, it is not possible to use traditional ITU-R Recommendation BT.500 (e.g. DSCQS) for the multimedia tests. Because of all the reasons given above, a new evaluation approach specifically designed to assess multimedia was patently necessary.
Conventional video evaluation approaches
ITU-R Recommendation BT.500 [6] is a reference document for conventional television-centred subjective evaluations. It proposes several subjective test methodologies. The most important subjective methods used for television assessments are as follows: DSIS: Double Stimulus Impairment Scale; DSCQS: Double Stimulus Continuous Quality Scale; SSCQE: Single Stimulus Continuous Quality Evaluation; SDSCE: Simultaneous Double Stimulus for Continuous Evaluation. BT.500 contains the test methodologies used for both quality assessments and impairment assessments 4. The most commonly used is the DSCQS method in which an assessor is presented with a pair of images or short video sequences A and B, one after the other, and is asked to give A and B a quality score by marking on a continuous line with five intervals ranging from Bad to Excellent. For each pair of sequences, one is an unimpaired reference sequence, and the other is the same sequence, modified by the coding system under test. The order of the two sequences is randomised, so that the assessor does not know which is the original and which is the impaired sequence. The result of the evaluations is a "Mean Opinion Score", which indicates the relative quality of the impaired and reference sequences. Table 2 gives some details of the BT.500 methodologies, in comparison with the SAMVIQ methodology, which is now described in some detail.

The SAMVIQ methodology

SAMVIQ has specifically been designed for multimedia content. It takes into account a range of codec types, image formats, bitrates, temporal resolutions, zooming effects, packet losses, etc. The SAMVIQ methodology was submitted to ITU-R 6Q in 2003 and has achieved the status of a Draft New Recommendation [7]. This section gives a broad outline of SAMVIQ; a detailed description is given in Appendix A. Compared to BT. 500, a major difference is in the way video sequences are presented to the assessor. In SAMVIQ video sequences are shown in multi-stimulus form, so that the user can choose the order of tests and correct their votes, as appropriate. As the assessors can directly
3. Punctum proximum is defined as the nearest viewing distance, subjectively determined by the viewers eyes, for optimum accommodation to a given display. 4. Quality assessments are defined as those that establish the performance of systems under optimum conditions. Impairment assessments are used to study the systems subjected to non-optimum conditions such as error-prone transmission and emission.
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 4 / 22
VIDEO COMPRESSION Table 2 ITU-R BT.500 and SAMVIQ

Parameter

Explicit reference Hidden reference High anchor Low anchor Scale Sequence length Picture format Two simultaneous stimuli Presentation of test material
Yes No No No Bad to excellent 10s All No I: Once II: Twice in succession Only test sequence
No Yes Yes Yes Bad to excellent 10s All No Twice in succession
No No No No Bad to excellent 5 min All No Once
Yes No No No Bad to excellent 10s All Yes Once

SAMVIQ

Yes Yes Hidden reference Yes Bad to excellent 10s All No Several concurrent (multi-stimuli) Test sequences and reference

Voting

Test sequence and reference

Test sequences

Difference between the test sequence and the reference simultaneously shown No
Possibility to change the vote before proceeding Continuous quality evaluation Minimum accepted votes Assessors per display Display
Yes (moving slider in a continuous way) 15 One or more Mainly TV

15 One or more Mainly TV

15 One Mainly PC a
a. B/VIM is in the process of studying whether SAMVIQ can also be applied to standard television displays, rather than solely PC displays.
compare the impaired sequences among themselves and against the reference, they can grade them accordingly. SAMVIQ is based on random playout of the test files. The individual assessor can start and stop the evaluation process as he wishes and is allowed to determine his own pace for performing the grading, modifying grades, repeating playout when needed, etc. With the SAMVIQ method, quality evaluation is carried out scene after scene including an explicit reference, a hidden reference and various algorithms (codecs). There is no continuous sequential presentation of the sequences as in the DSCQS method, where the assessor can make errors of judgement due to a lack of concentration. As a result, SAMVIQ offers higher reliability, i.e. smaller standard deviations.
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 5 / 22
In SAMVIQ there is only one assessor at a time, which alleviates a "group effect". Both an explicit and a hidden reference are used. The explicit reference is an uncompressed version of the original sequence and allows the assessor to determine a near-absolute measure of video quality 5. A hidden reference is technically identical to the explicit reference but is not readily available to the subject. It is actually hidden among other stimuli and the subject should be able to identify it. In SAMVIQ the hidden reference is mandatory. The SAMVIQ method provides an overall quality score for relatively short multimedia sequences. The duration of a sequence is typically in the range of 10 to 15 seconds in order to give the subject sufficient time to formulate a stable grading. The content of a sequence has to be homogeneous. A large quality range is required to stabilize the assessor's quality scores; otherwise, when the quality range is reduced, assessors try to discriminate among the quality of the sequences even if the differences are not perceptible. Therefore, the reliability of results decreases, as the quality of the codecs tested is similar. SAMVIQ includes improved rejection criteria (compared with those used in BT.500). The multimedia image quality is to be assessed on a multimedia screen and platforms, and not on conventional TV displays, in order to avoid the artefacts due to interlace and flicker. Similar to all other subjective methodologies, SAMVIQ requires careful consideration of the test arrangements. If these arrangements are not scrupulously adhered to, the results of the evaluations may not be as expected.

Subjective evaluations of internet video codecs Phase 2
Phase 2 evaluations were performed by Project Group B/VIM during 2002 and 2003. The following four codecs were included: Windows Media 9 Microsoft; RealVideo 9 RealNetworks; MPEG-4 Envivio implementation; QuickTime 6 Apple.
As in Phase 1, the following bitrates were used for the QCIF and CIF resolution formats: QCIF format CIF format 56 kbit/s 256 kbit/s 128 kbit/s 500 kbit/s 256 kbit/s 700 kbit/s 500 kbit/s 1400 kbit/s
The test sequences used were taken from the Phase 1 viewing tests and are listed in Table 3 and shown in Fig. 1. The sequences represent typical broadcast programmes and are fairly critical but not unduly so. Some of the sequences, however, contain difficult scenes (fast movements, details, colours) that may challenge the performance of the codecs under evaluation. The duration of the sequences was typically set to 10s. Two organizations performed the subjective tests: NRK (the Norwegian public broadcaster) and France Telecom R&D (FTRD). Each site organized a test panel consisting of 15 to 20 subjects. Generally, half of the subjects were experienced while the other half were not regularly involved in this kind of test but showed some interest in video evaluations and were subjected to some initial training.
5. By comparison, DSCQS and DSIS are capable of establishing a quality level relative to a reference sequence.
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 6 / 22
For the viewing and lighting conditions, the tests used the data given in ITU Recommendation BT-500.11. Each of these codecs was tested according to the parameters listed in Table 4.
Summary of the Phase 2 codec evaluations
Detailed results of the B/VIM video evaluation tests are given in Appendix B. In the following, some analysis of these results is performed by responding to the questions raised in the introductory section.
Figure 1 Test sequences used Table 3 Video test sequences used in Phase 2
Basket Kayak Entertainment Flower Garden

MPEG RAI RAI MPEG

Sport footage with vigorous movements and extensive details Sport footage with background panning motion Concert footage with camera motion and details Detail and colour rendition
How does video codec A compare to video codec B?
In the Phase 2 evaluations, we compared four codecs: Real Networks 9, Windows Media 9, Envivio MPEG-4 and QuickTime 6. For the CIF image format, RealNetworks 9 is the only codec that reaches transparency level at 1.4 Mbit/s. Windows Media 9 is next best

Table 4 Parameters of the codecs under test

Channel type

Nom. bitrate (kbit/s)

700 1400

Net bitrate (kbit/s)
4010% 10010% 20010% 40010% 56010% 116010%

Audio (kbit/s)

Video (kbit/s)
Frame rates and Formats QCIF (176 x 144)
6.25 12.25 6.25 12.AVI RGB 24 @ 18 Mbit/s AVI RGB 24 @ 68 Mbit/s

7 / 22

CIF (352 x 288)
Modem/PSTN Dual ISDN DSL/Cable 1 DSL/Cable 2 DSL/Cable 3 Cable 1 Reference
8 mono 20 mono 32 st music 48 st music 64 st music 128 stereo
3210% 8010% 16810% 35210% 50010% 103210%
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens
but falls 10 points short. It is interesting to see that the quality range of the four codecs at 250 kbit/s is about 10 points but this range increases steadily to some 35 points at 1.4 Mbit/s (which is equivalent to the difference between RealNetworks 9 and QuickTime 6 at that bitrate). This conclusion applies to both CIF and QCIF.
What is the quality improvement between one generation and the next of the same codec standard?
The new generation of codecs Example 1 (e.g. WM9) is not always better Windows Media 9 vs. Windows Media 8, CIF, the same laboratory than the previous generation (FTRD) ones (WM8). Sometimes, the reverse is true: WM8 performs 250 kbit/s 500 kbit/s 700 kbit/s 1400 kbit/s better at 500 and 700 kbit/s than WM9. Similarly, RealNet- WM 8 works 9 is slightly worse than WM 70 RealNetworks 8 at 250 kbit/s but similar at higher bitrates. The QuickTime codec has the same difficulty: the grades of the new version of QT6 are consistently lower at all bitrates than those of the older version (at 1.4 Mbit/s this difference amounts to 15 points).
What is the difference in codec performance between different implementations of the same codec standard?
Some evidence is available from the Phase 1 evaluations: both QuickTime 6 and Dicas Mpegable implemented the MPEG-4 Part 2 video standard. Dicas is better for both CIF and QCIF for all bitrates but the difference is relatively small (about 5 points). In addition, Dicas seems to render Flower Garden (which is critical for colour rendition) significantly better than QT6. In Phase 2 the Envivio codec and the QT6 codec, both based on the MPEG-4 algorithm, can be compared. Our results show that Envivio performs better for both CIF and QCIF at all bitrates. The difference increases with bitrate and reaches 10 to 15 points at 1.4 Mbit/s.

Acknowledgements

This report would not have been possible without the kind support of those who devoted much of their precious time and effort to the underlying work. The bulk of the work was performed by the following EBU member organizations and their staff: France Tlcom R&D Institut fr Rundfunktechnik (IRT)
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 10 / 22
Norwegian Broadcasting (NRK) Radiotelevisione Italiana (RAI) CRIT Many thanks to all of them.

Acronyms used

CIF QCIF SubQCIF VGA SVGA Common Image Format (352 pixels/line, 288 lines per picture, 30 pictures per second) Quarter CIF (176 x 144) Subquarter CIF (128 x 96) Video Graphics Array (640 x 480) Super Video Graphics Array (800 x 600)
Appendix A: The SAMVIQ methodology

Test material

The choice of material is crucial to the success of the tests and is far from being a simple matter. It is recommended that we should use a variety of unprocessed, ordinary broadcast programme sequences, addressing different quality aspects (e.g. codec artefacts, motion portrayal, colour rendition, sharpness, etc. The sequences should be chosen not to stress or indeed break the codecs tested. In selecting a range of test sequences it is important to achieve some balance between being not critical enough and being too critical. In the former case, all codecs would appear to be very good. In the latter case, all codecs would appear to be bad. The length of the sequences should typically not exceed 20s to avoid fatiguing the observers and to reduce the total duration of the tests. The test sequences should normally be different from those used by the manufacturers in optimizing the coding algorithms.

Training phase

The training session is an integral part of the SAMVIQ methodology. It is absolutely essential to train the subjects (assessors) in a special training session in advance of the tests proper. The appropriate training helps to obtain more reliable and more consistent results. The subject should be handed a written instruction sheet. As the same instructions should be used in all laboratories involved in the measuring campaign, there should be no statistically inconsistent results from one laboratory to another. The purpose of the training phase is to allow the subject to achieve the following objectives: to become familiar with the kind of artefacts of the compressed sequences to learn how to use the test equipment (user-interface) and the grading scale.

EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 11 / 22
The subjects should be told that the hidden reference is included in the tests but should not necessarily score it 100. They should use the full range of the continuous scale, as they find it appropriate.

Viewing conditions

The tests follow the conditions given in Rec. BT.500-11 for the ambient light, colour of the walls, type of monitor, etc. It is of paramount importance to create viewing conditions that can be reproduced in other laboratories around the world. Influence of the laboratory setup should be minimized.

Test organization

Test sessions are organized such that one scene follows the other (see Fig. 2). Only one image format (e.g. CIF or QCIF) is considered per session. The number of algorithms is limited to ten per scene, e.g. five algorithms for Codec 1 and five algorithms for Codec 2. For a scene it is possible to play and grade any sequence in any order. Each sequence can be played and assessed as many times as the assessor wants the last grade remains recorded. Each algorithm must be played out and viewed completely in each scene at least once. Grading of an algorithm can only be made after at least one complete viewing of that algorithm.
Explicit reference Ref Hidden reference F Algo.1 Algo.2 Algo.n

Scene 1

Algo.3
A B Corresponding access buttons

Scene 2

Explicit reference Ref

Hidden reference C

Algo.1

Algo.2

Algo.n
G F Corresponding access buttons Algo.1 Algo.2

Scene 3

Hidden reference B
Corresponding access buttons

Scene k

Hidden reference E
C D Corresponding access buttons
Figure 2 An example of the test organization in SAMVIQ
From one scene to the next, the sequences are randomised. This prevents the assessors from attempting to vote in an identical way according to an established order. Nevertheless, within a test, the algorithm order remains the same to simplify the analysis and presentation of results. Only the corresponding access from an identical button is randomized. The assessor is allowed to proceed to the next scene only after the evaluation of the previous scene was accomplished successfully. To finish the test, all the sequences of all the scenes must be scored.

The SAMVIQ interface

A typical SAMVIQ interface is shown in Fig. 3 7: Seven anonymous algorithms plus an explicit reference for five scenes are to be evaluated and scored. The slider is directly implemented on the
6. The lower the standard deviation, the higher is the consistency of results. 7. This example is taken from France Telecom's software package.
EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 13 / 22
screen. In the screenshot, some algorithms have already been evaluated their scores are written under the corresponding access button and the explicit reference is currently under evaluation. The following buttons for controlling the playout of the test sequences are shown on this screenshot: Selection of algorithm (A, B, C, , Explicit Reference). Play, Stop, Previous scene, Next scene, End. The "Ref button" (bottom left) represents the Explicit ReferFigure 3 Interface that implements the SAMVIQ method ence, while the other buttons (Courtesy: France Telecom R&D) (A, B, C) represent the algorithms including the hidden reference and the low anchor. On the right side, a slider is present in order to allow the assessor to grade the quality of the test item according to the continuous quality scale used.

Grading scale

The assessors are asked to assess the overall picture quality of each presentation by inserting a slider mark on a vertical scale. The scales provide a continuous rating system to avoid quantizing errors, but they are divided into five equal lengths which correspond to the normal ITU-R BT.500 five-point quality scale. The associated terms categorizing the different levels are the same as those normally used; but here they are included for general guidance. The grading scale is continuous and is divided in five equal portions, as follows: Excellent (80 to 100 points) Good Fair Poor Bad (60 to 80 points) (40 to 60 points) (20 to 40 points) (0 to 20 points)
The lowest quality perceived should be scored "0" (bottom of the scale) and the highest quality should be marked "100" (top of the scale).

Viewing Distance

SAMVIQ does not require any specific viewing distance range. Each assessor adjusts his own optimal viewing distance according to his preference for comfortable viewing. Especially for small images, the viewing distance depends on both the image size (a combination of image and display formats) and the punctum proximum (see Footnote 3. on page 4) which may vary from one user to another.

EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 14 / 22

Rejection criteria

All the assessors who have taken part in the evaluation process must be screened in order to establish the consistency of their scores. Inconsistent assessors who produced unstable or even contradictory scores are discarded from the final statistics. Compared to BT.500, SAMVIQ developed more accurate and reliable rejection criteria [8]. In SAMVIQ (as in DSCQS), all sequences including the hidden reference, low anchor and encoded sequences are considered. The rejection criteria use the Pearson decision criterion which is based on a correlation "r" of individual scores and corresponding mean scores from all the assessors. The Pearson algorithm assumes a linear relationship between the quality scale and score range of assessors. If this relaFranc Kozamernik graduated from the Faculty of Electrotechnical Engineering, University of Ljubljana, Slovenia, in 1972. He started his professional career as an R&D engineer at Radio-Television Slovenia. Since 1985, he has been with the EBU Technical Department and has been involved in a variety of engineering activities covering satellite broadcasting, frequency spectrum planning, digital audio broadcasting, audio source coding and the RF aspects of various audio and video broadcasting system developments, such as Digital Video Broadcasting (DVB) and Digital Audio Broadcasting (DAB). During his years at the EBU, Mr Kozamernik has coordinated the Internet-related technical studies carried out by B/BMW (Broadcast of Multimedia on the Web) and contributed technical studies to the I/OLS (On-Line Services) Group. Currently, he is the coordinator of several EBU R&D project groups including B/AIM (Audio in Multimedia), B/VIM (Video in Multimedia) and B/ SYN (Synergies of Broadcast and Telecom Systems and Services). He also coordinates EBU Focus Groups on Broadband Television (B/BTV) and MultiChannel Audio Transmission (B/MCAT). Franc Kozamernik has represented the EBU in several collaborative projects and international bodies, and has contributed a large number of articles to the technical press and presented several papers at international conferences. Paola Sunna was born in Italy in 1971. In 1997 she received a Degree in Electronic Engineering from the Politecnico of Turin. Her thesis was on objective video quality assessments in the MPEG-2 domain. Since then, Ms. Paola has been working at RAI CRIT (the Centre for Research and Technological Innovation of the Italian broadcaster RAI); her activities are mainly focused on full lab testing of video quality assessment schemes for broadcasting, webcasting and 3G applications. Since 2002, she has been the chair of EBU project group B/VIM (Video in Multimedia) that has defined the new subjective methodology for multimedia quality evaluation SAMVIQ. Emmanuel Wyckens was born in 1969 and graduated in Electronic Engineering from Valenciennes University (France) in 1994, having specialized in images,. In 1998, he joined France Telecom R&D and became involved in the development of digital processing algorithms for improving the video quality in MPEG-2 codecs. Mr Wyckens current activities are in the field of subjective video quality in multimedia (streaming and videoconferencing) applications, and in the standard and high-definition television domains. He also participates in the work of EBU project group B/VIM (Video in Multimedia). Dag Inge Pettersen received a degree in Electronic Engineering from HiBU in Kongsberg, Norway in 1993. In 1998 he received his MSc in signal processing from the Norwegian University of Science and Technology (NTNU) in Trondheim, Norway. Mr Pettersen joined NRK, the Norwegian public broadcaster, in 1999 and works mainly as a technical advisor in DAB and other digital broadcast technologies. He also works as a software developer in NRK.

EBU TECHNICAL REVIEW January 2005 F. Kozamernik, P. Sunna and E. Wyckens 15 / 22
tionship is not supposed to be linear, the Spearman rank correlation may be applied. In practice the Pearson and Spearman correlation results are very close indeed. By taking into account the Spearman rank and Pearson correlation results, an assessor may be discarded if "r" is less than the correlation threshold, which is normally set to 0.85.
Presentation of the results
The results of assessments should be presented in a standardized SAMVIQ form, so that they can be compared among the different laboratories. The EBU plans to establish a standard evaluation protocol which will include the following information: Test configuration; Test sequences; Type of picture source and display computer monitor (screen size, make and model number of displays used); Number and type of assessors (age and gender composition of the panel, education or employment category of the panel); Reference systems used; The grand mean score for the experiment; Original and adjusted mean scores and 95% confidence interval if one or more assessors have been eliminated.
Appendix B: Selected test results
On the following pages are some example results from the Phase 2 subjective evaluations on internet video codecs.

16 / 22

Test carried out with four scenes
All codecs Lab: NRK CIF format

Excellent

Mean score (%)

All codecs

Good 60 Fair 40 Poor

20 Bad

France Telecom R&D
Envivio (Mean + CI) QuickTime (Mean + CI) Real 9 (Mean + CI) WMV 9 (Mean + CI) 1500

Total bitrate (kbit/s)

Envivio codec Lab: NRK CIF format
100 Kayak (Mean + CI) Flower Garden (Mean + CI) Entertainment (Mean + CI) Basket (Mean + CI)
QuickTime codec Lab: NRK CIF format
Kayak (Mean + CI) Flower Garden (Mean + CI) Entertainment (Mean + CI) Basket (Mean + CI)

60 Fair

Good 60 Fair 40 Poor 20 Bad

40 Poor 20 Bad

RealNetworks 9 codec Lab: NRK CIF format
Windows Media 9 codec Lab: NRK CIF format

Excellent 100

60 Fair 40 Poor
Kayak (Mean + CI) Flower Garden (Mean + CI) Entertainment (Mean + CI) Basket (Mean + CI) 1500
Figure A1 NRK results CIF format All codecs, Envivio MPEG-4, QuickTime 6, RealNetwoks 9 and Windows Media 9

17 / 22

All codecs Lab: NRK QCIF format
Envivio (Mean + CI) QuickTime (Mean + CI) Real 9 (Mean + CI) WMV 9 (Mean + CI) 600
Envivio codec Lab: NRK QCIF format

QuickTime codec Lab: NRK QCIF format

60 Fair 40 Poor 20 Bad

RealNetworks 9 codec Lab: NRK QCIF format
Windows Media 9 codec Lab: NRK QCIF format
Kayak (Mean + CI) Flower Garden (Mean + CI) Entertainment (Mean + CI) Basket (Mean + CI) 600
Figure A2 NRK results QCIF format All codecs, Envivio MPEG-4, QuickTime 6, RealNetwoks 9 and Windows Media 9

18 / 22

All codecs Lab: FT R&D CIF format
Envivio codec Lab: FT R&D CIF format
QuickTime codec Lab: FT R&D CIF format
RealNetworks 9 codec Lab: FT R&D CIF format
Windows Media 9 codec Lab: FT R&D CIF format
Figure A3 FTRD results CIF format All codecs, Envivio MPEG-4, QuickTime 6, RealNetwoks 9 and Windows Media 9

19 / 22

All codecs Lab: FT R&D QCIF format
Envivio codec Lab: FT R&D QCIF format
QuickTime codec Lab: FT R&D QCIF format

40 Poor

RealNetworks 9 codec Lab: FT R&D QCIF format
Windows Media 9 codec Lab: FT R&D QCIF format
Figure A4 FTRD results QCIF format All codecs, Envivio MPEG-4, QuickTime 6, RealNetwoks 9 and Windows Media 9

20 / 22

NRK (Mean + CI) FT R&D (Mean + CI)
Envivio codec Inter-lab CIF format
QuickTime codec Inter-lab CIF format
RealNetworks 9 codec Inter-lab CIF format
Windows Media 9 codec Inter-lab CIF format

Good 60 Fair 40 Poor 20

60 Fair 40 Poor 20 Bad FT R&D (Mean + CI) 0

NRK (Mean + CI)

NRK (Mean + CI) FT R&D (Mean + CI) 0
Figure A5 Inter-laboratory evaluations CIF format Envivio MPEG-4, QuickTime 6, RealNetwoks 9 and Windows Media 9

21 / 22

Envivio codec Inter-lab QCIF format
QuickTime codec Inter-lab QCIF format
RealNetworks 9 codec Inter-lab QCIF format
Windows Media 9 codec Inter-lab QCIF format
Figure A6 Inter-laboratory evaluations QCIF format Envivio MPEG-4, QuickTime 6, RealNetwoks 9 and Windows Media 9

22 / 22