Advances in
computers, networking, and communications have created new distribution
channels and business opportunities for the dissemination of multimedia
content. Streaming audio and video over networks such as the Internet, local area
wireless networks, home networks, and commercial cellular phone systems has
become a reality and it is likely that streaming media will become a mainstream
means of communication.
Despite some
initial commercial success, streaming media still faces challenging technical
issues, including quality of service and cost effectiveness. For example,
deployments of multimedia services over 2.5G and 3G wireless networks have
presented significant problems for real time servers and clients in terms of
high variability of network throughput and packet loss due to network buffer
overflows and noisy channels.
Our research effort in this area has been focused on the
following aspects of streaming media:
·
Video
over wireless mesh networks
·
P2P multicasting
and streaming
· Joint source-network optimization
· Rate control for streaming media
· Content security and rights management in streaming media
· Rate shaping for networks with dynamic bandwidth
· Error resilience and error concealment
· Streaming system architecture and implementation.
Joint Source-Network Optimization
It is expected that IP-based networked
broadband multimedia system will be widely available within the next decade.
Multimedia data are delay-sensitive. IP networks can vary in bandwidth, loss,
delay, and jitter over very large time scales. In addition, topologies other
than traditional point-to-point communication, such as multicast, distributed
communication, multipath communications etc, are also
emerging. As a result, joint source-network processing/coding becomes critical
for high quality multimedia communication. This typically includes the joint
design and optimization of source coding/decoding, dynamic rate control,
error concealment, dynamic bandwidth estimation, error control, packet
scheduling and retransmission, network adaptation layer, flow control, caching,
routing, network topology, and other aspects of the system architecture.
We are particularly interested in end-to-end analysis and optimization of
streaming systems that takes
into account aspects of both media source and network communication. For
example, one of the problems we are addressing is dynamic bitstream
switching to deal with time varying wireless channels.
Papers:
1.
P. Zhu, W. Zeng, and C. Li, “Joint design of source rate
control and QoS-aware
congestion control for video streaming over the Internet,” IEEE Trans.
Multimedia, vol. 9, no. 2, pp. 366-376, Feb. 2007.
2.
B. Xie and
3.
B. Xie and W. Zeng, “Source
characteristics based fast bitstream
switching,” in IEEE International Conference on Multimedia
and Expo, 2003.
4.
W. Zeng and J. Wen, “3G wireless
multimedia: technologies and practical issues,” in Journal of Wireless
Communications and Mobile Computing,
Special
Issue on Multimedia over Mobile IP, vol. 2, issue 6, pp. 563-572,
Sept. 2002.
5.
W. Zeng and J. Wen, “3G wireless multimedia: technologies and
practical issues,” Invited paper, Proc. IEEE Inter. Conf. Image Proc., Special Session on Wireless Imaging,
Sept. 2002. (click here
for preprint in pdf format ).
Rate Control for Streaming Media
Most model based rate control solutions have
the generally questionable assumption that video sequence is stationary, and
typically suffer from the fundamental problem of model parameter mis-estimation. As a result, they usually have problems
dealing with more complex video sequences, although they might work fine for
some standard talking-head-like test sequences. For wireless streaming, an
additional constraint is the complexity imposed by the wireless devices.
To address these problems, we developed a sequence
based bit allocation framework with the capability of tracking the
non-stationary characteristics in the video source without look-ahead encoding.
A new model parameter estimation approach is also developed to solve the
problems in existing model parameter estimation schemes. Moreover, we highlight
a general concept of bit allocation guarantee to achieve the allocated bits in
a deterministic way. The proposed rate control solution can achieve constant
quality video with less quality flicker and motion jerkiness, and is
amenable to real-time implementation. Experimental results show that it
provides significantly better performance than the MPEG-4 Annex L rate control
solution.
Other on-going work includes extending the framework
to two-pass rate control solution for non-real time application, and to joint
source-network dynamic rate control.
Papers:
1.
B. Xie and
2.
Z. He, W. Zeng, and C. Chen, “Lowpass filtering of
rate-distortion functions for quality smoothing in real-time video
communication,” in IEEE Trans. on Circuits & Systems for
Video Technology, vol. 15, no. 8, pp. 973-981, Aug. 2005.
3.
J. Lan, W. Zeng, and X. Zhuang, “Operational
distortion-quantization curve based bit allocation for smooth video quality,”
in the Special Issue on Visual Communications in Ubiquitous Era, Journal
on Visual Communications and Image Representation, vol. 16, issues 4-5, pp.
527-543, August-October, 2005.
4.
B. Xie and
Digital rights management (DRM) is an
important component in a multimedia streaming system. DRM faces a number of new
challenges in streaming media application. One common method for access control
is through encryption. We are the first to recognize that, for many
applications such as streaming media application, it is essential that content
encryption (including key distribution) provides error resiliency, scalability,
network friendliness and capability of performing signal processing directly
on the encrypted bitstream, just as is required for
unencrypted content formats. In addition, for some applications with real-time
constraint, such as wireless multimedia streaming to low-power devices, a particularly
desirable feature is low processing overhead, which can usually be achieved
through selective encryption and/or some lightweight encryption techniques. A
“wholesale” encryption of the entire content bitstream
is usually not desirable or even feasible.
To that end, we have developed a framework of
performing selective encryption and spatial/frequency shuffling of compressed
digital content that maintains syntax compliance to the compression format
after content has been secured. With such a joint design of encryption and
compression, we can achieve the various aforementioned desirable
functionalities, which is extremely helpful for multimedia delivery over error
prone channels with dynamic bandwidth such as wireless networks or the
Internet. This format compliant selective encryption/shuffling framework has
been adopted into the MPEG-4 IPMP Final Proposed Draft Amendment (FPDAM).
We developed a scalable architecture for streaming of JPEG2000 images, using Hypertext Transfer Protocol (HTTP).
JPEG2000 is a new image compression standard. One of the goals of JPEG2000 is to support large images. For a large image, even the compressed image file size can be very big. Thus downloading the entire image at its full resolution can take a long time depending upon the user’s connection speed. Thus we propose to use streaming of JPEG2000 images. We use Hypertext transfer protocol (HTTP) for streaming. The use of HTTP to stream JPEG2000 images will ease its deployment, because of the widespread availability and accessibility of web servers. Thus JPEG2000 images can be hosted by web server and can be streamed to the client helper application. Our solution is scalable. The clients can view image at a variety of resolutions and quality levels. It is also possible to stream only selected region of the image at a particular resolution and decode this partial stream and display it at the client. Thus client devices with different capabilities, variety of screen resolutions, heterogeneous bandwidths can all achieve a scalable viewing of the same content stored in a single file.
Papers:
1.
S. Deshpande and W.
Zeng, “Scalable streaming of JPEG2000
images using Hypertext Transfer Protocol,” in ACM Multimedia, Oct.
2001. (full paper, 20% acceptance rate, click here for
preprint in pdf format )
( 18
citations using Google Scholar search)
Rate Shaping for Networks
with Dynamic Bandwidth
For reliable video services over networks which can not guarantee quality of services, i.e., IPv4, ATM ABR and wireless, it is advantageous to first estimate the instantaneous available bandwidth in a given network, and then adaptively scale or shape the bit rate of video in the compressed domain down to match the available bandwidth in the network. This approach is more aggressive than error-concealment-only without adaptive transmission. We proposed a novel block-dropping-based approach for rate shaping of MPEG-precompressed video. A geometric-structure-based directional interpolation scheme is incorporated at the receiver to reconstruct faithfully a large percentage of the blocks that are intelligently dropped to meet the constraint of reduced bandwidth. The blocks are dropped in a way that is optimal in the rate-distortion sense. The new approach is conceptually quite different from conventional approaches in which usually high frequency coefficients are eliminated. The gain is quite significant (up to 5 dB) due to the introduction of interpolation and pre-analysis of the video. We also showed that by jointly dropping blocks and coefficients, the bit rate can be reduced further, and additional gain is obtained. The general idea is equally applicable to uneqal error protection for joint source-channel coding.
Papers:
1.
W. Zeng and B. Liu,
"Geometric-structure-based error concealment with novel applications in
block-based low bit rate coding," IEEE Trans. on Cir. and Sys. for Video Technology, pp. 648-665, June 1999. (click here for preprint
in pdf format) ( 49
citations using Google Scholar search)
2.
Wenjun Zeng and Bede Liu, "Rate shaping by
block dropping for transmission of MPEG-precoded
video over channels of dynamic bandwidth" (full paper, 20% acceptance
rate, click here for paper in pdf format), ACM Multimedia '96, Boston, Nov. 1996. ( 24
citations using Google Scholar search)
Error Resilience and Error Concealment
A basic feature dominating general-purpose, computer-based multimedia communications is that of uncertainty. The structure of existing codecs does not fully match the above mentioned constraints. The mismatch may be even more pronounced in multipoint communications over heterogenous networks. The problem of recovering lost data in coded images due to imperfect communication channels is thus of importance. Error concealment techniques take advantage of the spatial or temporal correlation of videos without incurring much overhead and delay. We proposed a spatial directional interpolation scheme that makes use of the local geometric information extracted from the surrounding blocks to combat channel noise and cell loss due to network congestion. Unlike other approaches that usually result in a smoothed/blurred image, the proposed scheme can recover high frequency details well and is computationally very simple.
Our previous work also includes error concealment for transmission of vector quantized images over noisy channels.
Papers:
1. W. Zeng, “Adaptive Spatial-Temporal Error Concealment with Embedded Side Information,” in the Special Issue on Visual Communications in Ubiquitous Era, Journal on Visual Communications and Image Representation, vol. 16, issues 4-5, pp. 499-511, August-October, 2005.
2. W. Zeng, “Spatial-temporal error concealment with side information for standard video codecs,” in IEEE International Conference on Multimedia and Expo, 2003.
3. W. Zeng and B. Liu, "Geometric-structure-based error concealment with novel applications in block-based low bit rate coding," IEEE Trans. on Cir. and Sys. for Video Technology, pp. 648-665, June 1999. (click here for preprint in pdf format) ( 49 citations using Google Scholar search)
4. W. Zeng and Y.F. Huang, "Boundary matching detection for recovering erroneously received VQ indices over noisy channels", IEEE Trans. on Cir. and Sys. for Video Technology, vol. 6, no. 1, Feb. 1996. ( 10 citations using Google Scholar search)
5. Wenjun Zeng and Bede Liu, "Geometric-structure-based directional filtering for error concealment in image/video transmission" , SPIE Wireless Data Transmission at Information Systems/Photonics East'95 , vol. 2601, pp. 145-156, Oct. 1995. ( 22 citations using Google Scholar search)
6. W.J. Zeng and Y.F. Huang, "Boundary matching detection for progressive transmission of VQ indices over noisy channels", Proc. IEEE Int. Symp. on Cir. and Systems, vol. 3, pp. 249-252, May 1994.
Last Updated :