In the context of wireless mobile ad hoc networks, node clustering is a well-known solution for handling the scalability issue. While existing work focused on unstructured (i.e., flat) networks, this paper investigates a clustering algorithm to handle stable size-restricted clusters for structured (i.e., group-based) networks. In addition, we have identified that the ad hoc network clustering literature lacks a theoretical framework. This paper fills this gap by proposing to use coalition game theory, identifying coalitions to clusters and players to nodes. This theoretical framework allows us to derive a novel generic distributed node clustering algorithm. The algorithm is proved to converge to Nash-stable partitions. It is based on the concept of switch operations, where nodes take decision whether to leave or not their current coalition based on the coalition values. These decisions are made independently on any node individual payoff, meaning that the coalition formation game has a transferable utility. This generic algorithm is then tailored to both structured and unstructured networks, by defining judiciously the value functions and the heuristics dedicated to selecting suitable switch operations. Based on extensive simulations, we show that our proposed solutions outperform the existing ones especially in terms of cluster size and stability.