IR theorists should adopt the concept of bounded rationality. Roughly speaking, this models human behaviour on the assumptions that people (a) don’t always act rationally, (b) have limited information, and (c) have a limited capacity to process the information they do have. It applies to states because “the state” doesn’t actually decide anything by itself. All the policy decisions are taken by human beings who have bounded rationality.
They could also adopt the concept of utility from economics. Economists do not assume that people act to maximise their wealth. Rather, they assume that people act to maximise the value they obtain from their economic transactions, and that the value each individual assigns to a particular good (e.g. money) will vary. Some people will act to maximise their wealth, others will prioritise other goods such as leisure time or job satisfaction. Utility provides a concept with which to model this.
It is applicable to states because policy decisions are not always taken to maximise tangible benefits such as control of territory or natural resources. States may pursue ideological goals that appear entirely irrational to others, i.e. they may derive considerable utility from goods that have little or no value to others. Of course, these goals are set by the ruling elite who may pursue objectives that only provide utility to themselves.
So we should say that states act with bounded rationality to maximise their utility, as defined by the current ruling elite.
