Urban-rural differences have long been an important research topic but most of the existing literature has focused on the economic aspects, failing to explore the differences in the quality of life between urban and rural residents. Based on annual data from the China General Social Survey and using a latent class modeling approach to build a mixed regression model, this paper explores the differences in and influencing factors of the leisure lifestyles of Chinese urban and rural residents. The findings show that (1) the leisure lifestyles of urban and rural residents are remarkably different, with urban residents participating in leisure activities much more frequently than rural residents as a whole; (2) the differences in the types of leisure participation between urban and rural residents are obvious, with urban residents participating more frequently in leisure activities that help promote personal growth and rural residents being less likely to be involved in high-quality leisure activities; (3) public factors such as public education, health care, and social security and individual factors such as age, health, and education level are the main factors leading to the differences in leisure lifestyles between urban and rural residents. The findings of this study are conducive to narrowing down urban-rural differences, balancing the sense of happiness and gain of urban and rural residents, and providing empirical evidence for research on urban-rural differences in the leisure lifestyles of Chinese residents.