Behavioral finance is a field of combined psychological and economics study that accounts for the fact that people do not think rationally, nor do they support efficient markets. Up to the time of behavioral finance study, assumptions were that people think rationally and that they will only deal with efficient markets. Very lovely. It was all supposed to work out on it's own under these assumptions, teaching the foolish to be better thinkers and decimating markets that were a mess of inefficiency.
Behavioral Economics is a field that is closely related to Behavioral Finance. Both fields of study examine the cognition and emotion of consumers, investors and others who make financial and economic decisions, either with aggregate or single source major financial consequences.
Economic self interest is believed to be at the heart of the idea of individual action. But the cognitive errors, such as relying on recent history and over confidence, for example get in the way of the classic optimization of economic self interest and arbitrage (efficiency in markets). This creates distortions in effect that make the old, set way of thinking about behavior of humans in relation to markets less than useful.
Modeling is important to behavioral finance study. The new modeling considers that people are not necessarily rational thinkers or that they think correctly. One model is the Efficient Market Hypothesis (EMH), which assumes that when people are expecting great profit, they drive market prices toward their so-called "correct" value. It is not the investor who is rational enough to make unbiased forecasts of the future, but the market is rational and capable of making unbiased forecasts of the future, according to the assumption in the EMH.
In other words, the EMH assumes that the markets provide efficient information.
The major cognitive biases that are studied include: heuristics, or the "rule of thumb" style of thinking that may lead to biases in thinking. For example the "1/N" rule, where 1/3 or 1/4 of investments are allocated to particular areas, is an example of heuristic decision making.
There is overconfidence, where too little diversification is an example. Overconfidence in one or few areas will lead to owning too much stock in the employer's firm, too much money invested in the home and so on. Men have been found to have more of this problem than women.
Mental accounting is another source of cognitive error. The example given by Jay R. Ritter in the citation is excellent: we will never eat gourmet food items at home, where it would be much cheaper to get the ingredients and to prepare, but will pay more for it to have it in a restaurant. If we ate the simpler and less expensive fare in the restaurant, and saved the lobster and steaks for home, it would save money, overall. This is called "separate decision making" when the decisions should go together: ie, budgeting for "home food" versus "restaurant food" should be combined for economic purposes.
As a side note, it would be interesting to study the current situation, where consumers are being forced into exploring far more frugal alternatives than before, and where consumers have far more resources, such as cooking lessons on television or on line and local markets offer special ingredients and ethnic/imported foods at lower prices. Anyone who owns a Casio electronic piano, for example, can find and order parts at the Casio site, as opposed to buying a new piano. Many are gravitating toward thrift shops that will have brand new items that are overstock liquidations as another example. Finally, those bulging closets and stuffed garages are actually creating new markets through on line auctions, garage sales and private sales. Mental accounting functions at micro and at macro levels.
Framing is a concept of how the issue is presented to an individual. If they think that there is a special deal, discounted price, more palatable way of receiving bad news, or other perk, even if there is no difference in price or overall outcome, they will be more inclined to accept the offer or choice.
Representativeness deals with recency of past results. The "Law of small numbers", is the name for this cognition. If high returns have been going on for a while, high returns are regarded as normal. Lesser returns are viewed as losses, rather than the opportunity to have a return of any type. Looking at long term averages is supposed to be the more rational approach.
There are other cognitive errors, such as conservatism or anchoring, where people get set in their decisions and under react to change. This is in conflict with representativeness.
There is disposition or salience effect, where small losses are weighed against small gains. Going for the small gains somehow manages to increase the tax burden.
There are also other models, such as those which examine anomalies of economic behavior and those which examine anomalies in markets.
In terms of criticism, the most prominent one involves "model dredging", where bias toward a single cognitive error leads to faulty predictions of under or over reaction based on that one type of cognitive error. Another way to look at model dredging is in that it looks at the causes after the effect has happened, leading to explanations that can be manipulated to fit the theory.
The behavioral models are said to combine some psychological understandings with neo classical economic theory. Classical economic theory was closely bound to psychology, with the writings of Adam Smith and Jeremy Bentham. Then, economics went technical, with psychology set aside. In the 20th century, psychology was reintroduced to economic study as neo classical thought came into popularity.
Jay R. Ritter, "Behavioral Finance", U of Florida, September 2003
Wikipedia: Behavorial Finance and Behavioral Economics