Please wait a minute...
Shopping Cart Email List Chinese
Advanced Search Fig/Tab
Frontiers of Computer Science

ISSN 2095-2228

ISSN 2095-2236(Online)

CN 10-1014/TP

Postal Subscription Code 80-970

2018 Impact Factor: 1.129

Featured Articles  |  Most Downloaded  |  Most Reading
Online Submission Subscribe to Our RSS Content Feed
Front. Comput. Sci.    2021, Vol. 15 Issue (3) : 153703    https://doi.org/10.1007/s11704-019-8265-3
REVIEW ARTICLE
Psycho-visual modulation based information display: introduction and survey
Ning LIU1, Zhongpai GAO2, Jia WANG3, Guangtao ZHAI3()
1. Cooperative Medianet Innovation Center, Shanghai Jiao Tong University, Shanghai 200240, China
2. MoE Key Lab of Artificial Intelligence, AI Institute, Shanghai Jiao Tong University, Shanghai 200240, China
3. Institute of Image Communication and Network Engineering, Shanghai Jiao Tong University, Shanghai 200240, China
 Download: PDF(1308 KB)  
 Export: BibTeX | EndNote | Reference Manager | ProCite | RefWorks
Abstract

Industry and academia have been making great efforts in improving refresh rates and resolutions of display devices to meet the ever increasing needs of consumers for better visual quality. As a result, many modern displays have spatial and temporal resolutions far beyond the discern capability of human visual systems. Thus, leading to the possibility of using those display-eye redundancies for innovative usages. Temporal/ spatial psycho-visual modulation (TPVM/SPVM) was proposed to exploit those redundancies to generate multiple visual percepts for different viewers or to transmit non-visual data to computing devices without affecting normal viewing. This paper reviews the STPVM technology from both conceptual and algorithmic perspectives, with exemplary applications in multiview display, display with visible light communication, etc. Some possible future research directions are also identified.
Keywords information display      human visual system      spatial frequency      temporal frequency      non-negative matrix decomposition     
Corresponding Author(s): Guangtao ZHAI   
Issue Date: 11 March 2021
 Cite this article:   
Ning LIU,Zhongpai GAO,Jia WANG, et al. Psycho-visual modulation based information display: introduction and survey[J]. Front. Comput. Sci., 2021, 15(3): 153703.
 URL:  
https://academic.hep.com.cn/fcs/EN/10.1007/s11704-019-8265-3
https://academic.hep.com.cn/fcs/EN/Y2021/V15/I3/153703
1 X Wu, G Zhai. Temporal psychovisual modulation: a new paradigm of information display [exploratory DSP]. IEEE Signal Processing Magazine, 2013, 30(1): 136–141
https://doi.org/10.1109/MSP.2012.2219678
2 D H Kelly. Spatio-temporal frequency characteristics of color-vision mechanisms. Journal of the Optical Society America, 1974, 64(7): 983–990
https://doi.org/10.1364/JOSA.64.000983
3 D Varner, D Jameson, L M Hurvich. Temporal sensitivities related to color theory. Journal of the Optical Society of America A, 1984, 1(5): 474–481
https://doi.org/10.1364/JOSAA.1.000474
4 T Instruments. DLP discovery 4100 development kit. , 2015
5 N Corporation. NVIDIA 3D Vision. , 2014
6 H Ko, J Paik, G Zalewski. Stereoscopic screen sharing method and apparatus. 2010. US Patent App. 12/503,029
7 N Corporation. Pixel density display listing. , 2018
8 A Karnik, D Martinez Plasencia, W Mayol-Cuevas, S Subramanian. PiVOT: personalized view-overlays for tabletops. In: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology. 2012, 271–280
https://doi.org/10.1145/2380116.2380151
9 G Wetzstein, D Lanman, M Hirsch, R Raskar. Tensor displays: compressive light field synthesis using multilayer displays with directional backlighting. ACM Transactions on Graphics, 2012, 31(4): 80
https://doi.org/10.1145/2185520.2185576
10 G Ye, A State, H Fuchs. A practical multi-viewer tabletop autostereoscopic display. In: Proceedings of IEEE International Symposium on Mixed and Augmented Reality. 2010, 147–156
https://doi.org/10.1109/ISMAR.2010.5643563
11 A Karnik, W Mayol-Cuevas, S Subramanian. MUSTARD: a multi user see through AR display. In: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. 2012, 2541–2550
https://doi.org/10.1145/2207676.2208641
12 A Nashel, H Fuchs. Random hole display: a non-uniform barrier autostereoscopic display. In: Proceedings of 3DTV Conference: The True Vision — Capture, Transmission and Display of 3D Video. 2009, 1–4
https://doi.org/10.1109/3DTV.2009.5069665
13 D Lanman, G Wetzstein, M Hirsch, W Heidrich, R Raskar. Polarization fields: dynamic light field display using multi-layer LCDs. In: Proceedings of the SIGGRAPH Asia Conference. 2011, 1–10
https://doi.org/10.1145/2070781.2024220
14 G Zhai, X Wu. Multiuser collaborative viewport via temporal psychovisual modulation [applications corner]. IEEE Signal Processing Magazine, 2014, 31(5): 144–149
https://doi.org/10.1109/MSP.2014.2328506
15 X Wu, G Zhai. Backward compatible stereoscopic displays via temporal psychovisual modulation. In: Proceedings of SIGGRAPH Asia Emerging Technologies. 2012
https://doi.org/10.1145/2407707.2407711
16 L Jiao, X Shu, X Wu. LED backlight adjustment for backward-compatible stereoscopic display. IEEE Signal Processing Letters, 2013, 20(12): 1203–1206
https://doi.org/10.1109/LSP.2013.2285284
17 R Ma, O C Au, P Wan, L Xu, W Sun, W Hu. Improved temporal psychovisual modulation for backward-compatible stereoscopic display. In: Proceedings of IEEE Global Conference on Signal and Information Processing. 2014, 1034–1038
https://doi.org/10.1109/GlobalSIP.2014.7032278
18 Y Chen, G Zhai, J Zhou, Z Wan, L Tang. Global quality of assessment and optimization for the backward-compatible stereoscopic display system. In: Proceedings of IEEE International Conference on Image Processing. 2017, 191–195
https://doi.org/10.1109/ICIP.2017.8296269
19 W Fujimura, Y Koide, R Songer, T Hayakawa, A Shirai, K Yanaka. 2x3D: real time shader for simultaneous 2D/3D hybrid theater. In: Proceedings of SIGGRAPH Asia Emerging Technologies. 2012, 1–2
https://doi.org/10.1145/2407707.2407708
20 S Scher, J Liu, R Vaish, P Gunawardane, J Davis. 3D+2DTV: 3D displays with no ghosting for viewers without glasses. ACM Transactions on Graphics, 2013, 32(3): 21
https://doi.org/10.1145/2487228.2487229
21 Z Gao, G Zhai, X Min. Information security display system based on temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2014, 449–452
https://doi.org/10.1109/ISCAS.2014.6865167
22 C Hu, G Zhai, Z Gao, X Min. Information security display system based on spatial psychovisual modulation. In: Proceedings of IEEE International Conference on Multimedia and Expo. 2014, 1–4
https://doi.org/10.1109/ICME.2014.6890190
23 Y Chen, N Liu, G Zhai, Z Gao, K Gu. Information security display system on android device. In: Proceedings of IEEE Region 10 Conference. 2016, 1634–1637
https://doi.org/10.1109/TENCON.2016.7848294
24 X Li, G Zhai, J Wang, K Gu. Portable information security display system via spatial psychovisual modulation. In: Proceedings of IEEE Visual Communications and Image Processing. 2017, 1–4
https://doi.org/10.1109/VCIP.2017.8305058
25 C Hu, G Zhai, Z Gao, X Min. Simultaneous dual-subtitles exhibition via spatial psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2014, 1–4
https://doi.org/10.1109/BMSB.2014.6873483
26 C Hu, G Zhai, Z Gao, X Min. Simultaneous triple subtitles exhibition via temporal psychovisual modulation. In: Proceedings of the 9th IEEE Conference on Industrial Electronics and Applications. 2014, 944–947
https://doi.org/10.1109/BMSB.2014.6873483
27 W Sun, G Zhai, Z Gao, T Chen, Y Zhu, Z Wang. Dual-view oracle bone script recognition system via temporal-spatial psychovisual modulation. In: Proceedings of IEEE Conference onMultimedia Information Processing and Retrieval. 2020, 193–198
https://doi.org/10.1109/MIPR49039.2020.00047
28 Z Gao, G Zhai, C Hu, X Min. Dual-view medical image visualization based on spatial-temporal psychovisual modulation. In: Proceedings of IEEE International Conference on Image Processing. 2014
https://doi.org/10.1109/ICIP.2014.7025436
29 W Fang, G Zhai, X Yang, J Liu, Y Chen. An eye-friendly dual-view projection system using temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2017, 1–5
https://doi.org/10.1109/BMSB.2017.7986203
30 G Zhai, X Wu. Defeating camcorder piracy by temporal psychovisual modulation. Journal of Display Technology, 2014, 10(9): 754–757
https://doi.org/10.1109/JDT.2014.2317810
31 Z Gao, G Zhai, X Wu, X Min, C Zhi. DLP based anti-piracy display system. In: Proceedings of IEEE Conference on Visual Communications and Image Processing. 2014, 145–148
https://doi.org/10.1109/VCIP.2014.7051525
32 Y Chen, G Zhai, Z Gao, K Gu, W Zhang, M Hu, J Liu. Movie piracy tracking using temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Broadband Multimedia Systems and Broadcasting. 2017, 1–4
https://doi.org/10.1109/BMSB.2017.7986217
33 Z Gao, G Zhai, C Hu. The invisible QR code. In: Proceedings of the ACM International Conference on Multimedia. 2015, 675–678
https://doi.org/10.1145/2733373.2806398
34 X Lu, B You, P Y Lin. Augmented reality via temporal psycho-visual modulation. In: Proceedings of IEEE International Conference on Multimedia Expo Workshops. 2016, 1–4
35 Q Chen, Y Chen. Polarization based invisible barcode display. In: Proceedings of International Forum on Digital TV and Wireless Multimedia Communication. 2018, 67–77
https://doi.org/10.1007/978-981-10-8108-8_7
36 S Shi, L Chen, W Hu, M Gruteser. Reading between lines: high-rate, nonintrusive visual codes within regular videos via implicitcode. In: Proceedings of the ACM International Joint Conference on Pervasive and Ubiquitous Computing. 2015, 157–168
https://doi.org/10.1145/2750858.2805824
37 X Wu, X Shu. Combining information display and visible light wireless communication. In: Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing. 2015, 1657–1661
https://doi.org/10.1109/ICASSP.2015.7178252
38 X Shu, X Wu. Frame untangling for unobtrusive display-camera visible light communication. In: Proceedings of the 24th ACM International Conference on Multimedia. 2016, 650–654
https://doi.org/10.1145/2964284.2967302
39 K Liu, X Wu, X Shu. On display-camera synchronization for visible light communication. In: Proceedings of Visual Communications and Image Processing. 2015, 1–4
https://doi.org/10.1109/VCIP.2015.7457911
40 C Hu, G Zhai, Z Gao. Visible light communication via temporal psychovisual modulation. In: Proceedings of the 23rd ACM International Conference on Multimedia. 2015, 785–788
https://doi.org/10.1145/2733373.2807404
41 W Fang, G Zhai, X Yang. A flash light system for individuals with visual impairment based on TPVM. In: Proceedings of International Conference on Cloud Computing and Big Data. 2016, 362–366
https://doi.org/10.1109/CCBD.2016.077
42 Y Zhang, G Zhai, J Liu, X Weng, Y Chen. ‘window of visibility’ inspired security lighting system. In: Proceedings of International Conference on Systems, Signals and Image Processing. 2017, 1–5
https://doi.org/10.1109/IWSSIP.2017.7965622
43 Z Gao, G Zhai, J Zhou. Factorization algorithms for temporal psychovisual modulation display. IEEE Transactions on Multimedia, 2016, 18(4): 614–626
https://doi.org/10.1109/TMM.2016.2523425
44 J Feng, X Huo, L Song, X Yang, W Zhang. Evaluation of different algorithms of nonnegative matrix factorization in temporal psychovisual modulation. IEEE Transactions on Circuits and Systems for Video Technology, 2014, 24(4): 553–565
https://doi.org/10.1109/TCSVT.2013.2280089
45 L Wang, G Zhai. Constrained nmf for multiple exhibition on a single display. In: Proceedings of Picture Coding Symposium. 2015, 292–296
https://doi.org/10.1109/PCS.2015.7170093
46 Z Gao, G Zhai, X Gu, J Zhou. Adapting hierarchical ALS algorithms for temporal psychovisual modulation. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2015, 2756–2759
https://doi.org/10.1109/ISCAS.2015.7169257
47 Z Gao, G Zhai, J Wang. Spatially-weighted nonnegative matrix factorization with application to temporal psychovisual modulation. Digital Signal Processing, 2017, 67: 123–130
https://doi.org/10.1016/j.dsp.2017.04.010
48 J Kim, H Park. Fast nonnegative matrix factorization: an active-set-like method and comparisons. SIAM Journal on Scientific Computing, 2011, 33(6): 3261–3281
https://doi.org/10.1137/110821172
49 D D Lee, H S Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 1999, 401(6755): 788–791
https://doi.org/10.1038/44565
50 E F Gonzalez, Y Zhang. Accelerating the Lee-Seung algorithm for nonnegative matrix factorization. Department of Computational and Applied Mathematics, Rice University, Houston, TX, Technical Report, TR-05-02, 2005
51 M W Berry, M Browne, A N Langville, V P Pauca, R J Plemmons. Algorithms and applications for approximate nonnegative matrix factorization. Computational Statistics & Data Analysis, 2007, 52(1): 155–173
https://doi.org/10.1016/j.csda.2006.11.006
52 J Kim, H Park. Toward faster nonnegative matrix factorization: a new algorithm and comparisons. In: Proceedings of IEEE International Conference on Data Mining. 2008, 353–362
https://doi.org/10.1109/ICDM.2008.149
53 H Kim, H Park. Nonnegative matrix factorization based on alternating nonnegativity constrained least squares and active set method. SIAM Journal on Matrix Analysis and Applications, 2008, 30(2): 713–730
https://doi.org/10.1137/07069239X
54 A Cichocki, P Anh-Huy. Fast local algorithms for large scale nonnegative matrix and tensor factorizations. IEICE Transactions on Fundamentals of Electronics, Communications and Computer Sciences, 2009, 92(3): 708–721
https://doi.org/10.1587/transfun.E92.A.708
55 N Gillis, F Glineur. Accelerated multiplicative updates and hierarchical als algorithms for nonnegative matrix factorization. Neural Computation, 2012, 24(4): 1085–1105
https://doi.org/10.1162/NECO_a_00256
56 L Itti, C Koch, E Niebur. A model of saliency-based visual attention for rapid scene analysis. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1998, 20(11): 1254–1259
https://doi.org/10.1109/34.730558
57 E Reinhard, G Ward, S Pattanaik, P Debevec. High Dynamic Range Imaging: Acquisition, Display, and Image-Based Lighting (The Morgan Kaufmann Series in Computer Graphics). San Francisco, CA, USA: Morgan Kaufmann Publishers Inc., 2005
https://doi.org/10.1016/B978-012585263-0/50010-5
58 D Li, Z Gao, X P Zhang, G Zhai, X Yang. Generative adversarial networks for non-negative matrix factorization in temporal psycho-visual modulation. Digital Signal Processing, 2020, 100: 102681
https://doi.org/10.1016/j.dsp.2020.102681
59 Z Gao, G Zhai. Dual-view display based on spatial psychovisual modulation. IEEE Access, 2018, 6: 41356–41366
https://doi.org/10.1109/ACCESS.2018.2857006
60 R E NÑsÑnen, H T Kukkonen, J M Rovamo. A window model for spatial integration in human pattern discrimination. Investigative Ophthalmology & Visual Science, 1995, 36(9): 1855–1862
61 S N J Watamaniuk, R Sekuler. Temporal and spatial integration in dynamic random-dot stimuli. Vision Research, 1992, 32(12): 2341–2347
https://doi.org/10.1016/0042-6989(92)90097-3
62 W Sun, Z Gao, G Zhai, J Zhang, Z Wang, Y Zhu. An improved algorithm for real-time dual-view display. In: Proceedings of IEEE International Symposium on Circuits and Systems. 2020, 1–5
https://doi.org/10.1109/ISCAS45731.2020.9180978
63 G Zhai, X Min. Perceptual image quality assessment: a survey. Science China Information Sciences, 2020, 63(11): 211301
https://doi.org/10.1007/s11432-019-2757-1
64 Y Chen, N Liu, G Zhai, K Gu, J Wang, Z Gao, Y Zhu. Quality assessment for dual-view display system. In: Proceedings of Visual Communications and Image Processing. 2016, 1–4
https://doi.org/10.1109/VCIP.2016.7805459
65 Y Chen, G Zhai, K Gu, X Zhang, W Lin, J Zhou. Benchmarking screen content image quality evaluation in spatial psychovisual modulation display system. In: Proceedings of Pacific Rim Conference on Multimedia. 2018, 629–640
https://doi.org/10.1007/978-3-319-77380-3_60
66 A L Scherzinger, W R Hendee. Basic principles of magnetic resonance imaging–an update. The Western Journal of Medicine, 1985, 143(6): 782–792
67 S C Bushong, G Clarke. Magnetic Resonance Imaging: Physical and Biological Principles. Elsevier Health Sciences, 2014
68 M Lambooij, W IJsselsteijn, I Heynderickx. Visual discomfort of 3d tv: assessment methods and modeling. Displays, 2011, 32(4): 209–218
https://doi.org/10.1016/j.displa.2011.05.012
69 S A Taylor. CCD and CMOS imaging array technologies: technology review. Xerox Research Centre Europe, Technical Report EPC-1998-106, 1998, 1–14
70 ISO/IEC. Information technology–automatic identification and data capture techniques–QR Code 2005 bar code symbology specification. , 2006
71 K Song, N Liu, Z Gao, J Zhang, G Zhai, X P Zhang. Deep restoration of invisible QR code from TPVM display. In: Proceedings of IEEE International Conference on Multimedia & Expo Workshops. 2020, 1–6
https://doi.org/10.1109/ICMEW46912.2020.9105961
72 S E Siwek. The true cost of copyright industry piracy to the US economy. IPI Center for Technology Freedom, 2007
73 J Dorning. Intellectual Property Theft: A Threat to U.S. Workers, Industries, and Our Economy. DPE Research Department, 2014
74 B NEWS. The fact and fiction of camcorder piracy. , 2015
75 S Byers, L F Cranor, E Cronin, D Korman, P McDaniel. An analysis of security vulnerabilities in the movie production and distribution process. Telecommunications Policy, 2004, 28(78): 619–644
https://doi.org/10.1016/j.telpol.2004.05.007
76 J Haitsma, T Kalker. A watermarking scheme for digital cinema. In: Proceedings of International Conference on Image Processing. 2001, 487–489
77 P Nguyen, R Balter, N Montfort, S Baudry. Registration methods for nonblind watermark detection in digital cinema applications. In: Proceedings of Security and Watermarking of Multimedia Contents V. 2003, 553–562
https://doi.org/10.1117/12.479735
78 J Lubin, J A Bloom, H Cheng. Robust content-dependent high-fidelity watermark for tracking in digital cinema. In: Proceedings of Security and Watermarking of Multimedia Contents V. 2003, 536–545
https://doi.org/10.1117/12.477336
79 Y Nakashima, R Tachibana, N Babaguchi. Watermarked movie soundtrack finds the position of the camcorder in a theater. IEEE Transactions on Multimedia, 2009, 11(3): 443–454
https://doi.org/10.1109/TMM.2009.2012938
80 J Davis, Y H Hsieh, H C Lee. Humans perceive flicker artifacts at 500 hz. Scientific Reports, 2015, 5: 7861
https://doi.org/10.1038/srep07861
[1] Article highlights Download
Viewed
Full text


Abstract

Cited
  Shared   
  Discussed