Topic 4: Live Video Processing

This reserch focuses visual fashion content analysis and search, by utlizing both social media iamges and UGC. We highlight two sub-projects on “fashion-clothing suggestion and pairing” and “fashion-makeup recommendation and synthesis”.

Sub-Project 4.1: Fashion-Clothing Suggestion and Pairing

For the clothing part, we mainly target at two tasks: cross-scenario clothing retrieval and clothing recommendation.  As shown in Figure 4.1, the goal of cross-scenario clothing retrieval is: given a daily human photo captured in general environment (e.g., the street view), find similar clothing in online shops, where the photos are captured more professional. We first propose to alleviate the human pose discrepancy by locating 30 human parts detected by a well-trained human detector. Then, we propose a two-step calculation to obtain more reliable similarities between the query daily photo and online photos. The extensive experimental evaluations demonstrate the effectiveness of the proposed framework. For occasion-oriented clothing recommendation task, we develop at a practical system, named magic closet, as shown in Figure 4.2. Given a user-input occasion (e.g., wedding, shopping or dating), magic closet intelligently suggests the most suitable clothing from the user’s own clothing photo album, or automatically pairs the user-specified reference clothing (upper-body or lower-body) with the most suitable one from online shops. We adopt middle-level clothing attributes (e.g., clothing category, color, pattern) as a bridge. The clothing attributes are treated as latent variables in our proposed latent SVM based recommendation model. Extensive experiments demonstrate the effectiveness of the magic closet system. 

Figure 4.1: Cross-scenario clothing retrieval

Figure 4.2: Occasion-oriented clothing recommendation

Sub-Project 4.2: Fashion-Makeup Recommendation and Synthesis

For facial makeup part, we mainly focus on building a practical system for automatic and personalized facial makeup recommendation and synthesis. To build such a system, firstly, various local facial makeup effects including shape of eye shadow, colors of eye shadow, lip and foundation are decomposed based on face alignment and segmentation techniques, and their compatibility with makeup-invariant facial attributes is mined from our collected Beauty Makeup Database (BMD) with 500 celebrities/models of various makeup styles. Next, a latent SVM model describing the relations among makeup-invariant facial features, attributes and makeup effects is learned as the makeup recommendation model for suggesting the most suitable makeup effects for a face without makeup. Finally, the recommended makeup effects are seamlessly synthesized onto the given facial image. Extensive experiments on 100 testing images well demonstrate the effectiveness of the proposed makeup recommendation and synthesis system.

Figure 4.3: The flowchart of facial makeup recommendation and synthesis. 

Sub-Project 4.3: Social Recommendation

Exponential growth of information generated by online social networks demands effective recommender systems to provide useful results. Traditional techniques are inappropriate as they ignore social relation information. On the other hand, existing social recommendation approaches consider social network structure, but the social context has not been fully considered. It is significant and challenging to fuse social contextual factors which are derived from users motivation of social behaviors into social recommendation. In this research, we investigate social recommendation on the basis of psychology and sociology studies, which exhibit two important factors: individual preference and interpersonal influence. We first present the particular importance of these two factors in online item adoption and recommendation. Then we propose a novel probabilistic matrix factorization method to fuse them in latent spaces. We conduct experiments on both Facebook style bidirectional and Twitter style unidirectional social network datasets in China. The empirical result and analysis on these two large datasets demonstrate that our method significantly outperform the existing approaches.

Social networks enable users to create different types of personalized items. In dealing with serious information overload, the major problems of social recommendation are sparsity and cold start. In existing approaches, relational and heterogeneous domains cannot be effectively utilized for social recommendation, which brings a challenge to model users and multiple types of items together on social networks. In this research, we further consider how to represent social networks with multiple relational domains and alleviate the major problems in an individual domain by transferring knowledge from other domains. We propose a novel Hybrid Random Walk (HRW), which can integrate multiple heterogeneous domains including directed/undirected links, signed/unsigned links and within-domain/cross-domain links into a star-structured hybrid graph with user graph at the center. We perform random walk until convergence and use the steady state distribution for recommendation. We conduct experiments on a real social network dataset and show that our method can significantly outperform existing social recommendation approaches.


Representative Publication

[1] Liu, S., Song, Z, Liu, G., Xu, C., Lu, H., Yan, S.: Street-to-shop: Cross-scenario clothing retrieval via parts alignment and auxiliary set. CVPR 2012.

[2] Liu, S., Feng, J., Song, Z., Zhang, T., Lu, H., Xu, C., Yan, S.: Hi, magic closet, tell me what to wear!. ACM Multimedia, 2012