You are allowed to sign as many recruits as you want, regardless of open scholarship spots. But yes, that would result in someone losing their scholarship. It's called "oversigning." Indiana is a pretty good example -- Tom Crean has done it the past two years, you can read about it here:
http://espn.go.com/blog/collegebasketballnation/post/_/id/83643/since-when-is-oversigning-ok