With the rapid spread of smart devices, such as smartphones and tablet PCs, user authentication is becoming increasingly important because various kinds of data concerning user privacy are processed within them. At present, in the case of smart devices, password-based authentication is frequently used; however, biometric authentication has attracted more attention as a user authentication technology. A smart device is equipped with various sensors, such as cameras, microphones, and touch panels, many of which enable biometric information to be obtained. While the function of biometric authentication is available in many smart devices, there remain some problems to be addressed for more secure and convenient user authentication. In this paper, we summarize the current problems with user authentication on smart devices and propose a novel user authentication system based on the concept of context awareness to resolve these problems. We also present our evaluation of the performance of the system by using biometric information that was acquired from smart devices. The evaluation demonstrates the effectiveness of our system.