Is Zillow “Cursed?” A Behavioral Economics Perspective

https://ift.tt/3HQaEmW Opinion What the iBuying business tells us about uses and misuses of predictive models (This article is based, amo...

https://ift.tt/3HQaEmW

Opinion

What the iBuying business tells us about uses and misuses of predictive models

(This article is based, among other things, on my book Behavioral Data Analysis with R and Python: Customer-Driven Data for Real Business Results [1])

Introduction

The real estate company Zillow has recently announced that it is shutting down its home-buying division, Offers, and laying off 25% of its employees after having lost more than $300m in the past few months.

Zillow Offers was the company’s foray into iBuying: in Zillow’s words, “Instant buyers, commonly known as iBuyers, are home buying and selling services that typically buy off-market homes directly from sellers at market rate and quickly list them for sale on the open market after making light repairs and updates”[6]. Zillow’s core business is to list properties for sale (i.e., a real estate marketplace), an activity supported by their (in)famous tool to estimate the value of houses, the Zestimate.

As a behavioral scientist, I’m well aware of the hindsight and narrative biases. When focusing on a single past event, it’s easy to fall for the illusion of a simple explanation that was predictable all along. It’s also all too easy to take an “inside view” and focus on the details and anecdotes of the case at hand rather than take a broader “outside view”. For example, I have seen explanations along the lines of “data science is not precise enough for this use case” or “the market was too volatile” [3,4,5]; these do not account for the fact that over the same time period other iBuyers, such as Opendoor and Offerpad, did much better than Zillow Offers. However, there are valid reasons to believe that predictive analytics is not the right tool for this business. Before describing these reasons, I’ll first present the different types of analytics.

The different types of analytics

“Data science” has come to encompass all business analytics as long as they’re fancy enough, but a key distinction in my mind is between descriptive, predictive and causal analytics:

• Descriptive analytics provides a description of data. In simple terms, I think of it as “what is” or “what we’ve measured” analytics. How many customers canceled their subscriptions last month? How much profit did we make last year? Descriptive analytics is the simplest form of analytics, but it is also underappreciated. Many organizations actually struggle to get a clear and unified view of their operations.
• Predictive analytics provides a prediction. I think of it as “what will be, assuming current conditions persist” or “what we haven’t yet measured” analytics. Most machine learning methods belong to this type of analytics and help us answer questions like “How many customers will cancel their subscription next month?” and “Is that order fraudulent?”
• Finally, causal analytics provides the causes of data. I think of it as “what if?” or “what will be, under different conditions” analytics. It answers questions such as “How many customers will cancel their subscription next month unless we send them a coupon?”.

Over the past 25 years or so, predictive analytics has taken the world by storm, and today legions of data scientists are busy building predictive models for businesses. These models are not perfect (as Yogi Berra said, “it’s tough to make predictions, especially about the future”), but they routinely outperform human experts with decades of experience. Why then should we expect predictive analytics to fare less well in the iBuying business in particular? For two reasons:

• House buying and selling behaviors are influenced by a variety of volatile factors (aka, human beings are complicated);
• The iBuying market has a strong strategic dimension.

In the following two sections, I’ll examine these two aspects and point at their solutions.

House buying and selling behaviors are influenced by a variety of volatile factors

Predictive analytics works best when the phenomenon under study obeys few, stable and well-known factors. Let’s take the example of predictive maintenance for wind turbines:

• There is a limited number of relevant factors, and they have limited interactions with each other. Once you’ve accounted for the force of the wind, the friction between moving parts, and the corrosion of materials, you have a pretty good grasp of what is going to happen.
• The laws of physics are constant (gravity doesn’t take any day off!).
• Physics is well understood. For simplicity’s sake, a model may neglect secondary or rare phenomena, such as a 1-in-100-year storm, but even when such outliers occur, the outcome remains somewhat predictable: the blade will break and fall, not go on strike or quit.

On the other hand, in the real estate market:

• There are many, many different factors at play — demographic, economic, but also cultural and psychological — and they have complex interactions. This means that the predictive power of even the 3 or 4 most important aspects will be disappointingly low.
• These factors are not constant; preferences change over time, sometimes brutally. This is amplified by three characteristics of the real estate market. First, behaviors are “lumpy”; one buys or sell a whole house at once, consumers cannot easily reduce their monthly consumption of housing by 5%. Second, the timing of transactions is flexible; one can rush or delay buying a new house based on market conditions, much more than one can delay buying food or household products. Finally, housing can be a big component of people’s wealth, which amplifies the effect of price changes (conversely, much fewer people get richer and go on a spending spree when coffee gets more expensive). This can lead to roller-coaster changes in transaction volumes that you just don’t see in other consumer products.
• The conjunction of the previous two aspects means that the number of possible “states” of the market is huge (an issue known in statistics and other quantitative fields as “the curse of dimensionality”). This has important implications from a mathematical modeling perspective. Predictive analytics fares best when predicting outcomes for a combination of predictor values that is “close enough” to the training data. Admittedly, in any non-trivial dataset, it is pretty common to have unique data points, with a combination of values for the predictors that has never occurred in the training dataset (pretty much any continuous variable such as height or income will do that). However, most of the time, even unique data points can be approximated by a linear combination of points in the training dataset. Conversely, in the case of the real estate market, it is easily conceivable that entire local markets (say, the Phoenix area) are literally in uncharted waters at once. Predictive analytics must then rely on extrapolations, which get less and less robust as we move away from previously observed values (aka, overfitting).

“In the case of the real estate market, it is easily conceivable that entire local markets (say, the Phoenix area) are literally in uncharted waters at once. Predictive analytics must then rely on extrapolations, which get less and less robust as we move away from previously observed values (aka, overfitting).”

As such, the real estate market offers a tough nut to crack for analytics. This is not a new problem. Already in the 1970s, the macroeconomist Robert Lucas articulated that point (the “Lucas critique” of Keynesian models). He recommended instead identifying the deeper parameters of human behaviors such as consumer preferences, and I couldn’t agree more.

Using causal analytics in conjunction with the analysis of underlying behavioral drivers can mitigate these issues. In my book, I developed for that purpose what I called the “causal-behavioral framework for data analysis”.

The causal-behavioral framework for data analysis

Whereas predictive analytics is content to build black boxes with high accuracy, causal analytics strives to unearth the fundamental laws and regularities at play in a situation. This difference in goals means that causal analytics will often have lower predictive power in “regular” situations than its predictive sibling, but will shine in more complex and volatile situations.

The iBuying market has a strong strategic dimension

A crucial characteristic of the iBuying business is that you bid on individual properties. There’s actually a lot to unpack in that sentence, so let’s take it one piece at a time:

• Each property is unique. Even in the most uniform markets, houses are slightly different from one another (e.g. different levels of wear of tear). In particular, two almost identical houses in different locations are not interchangeable the way that cans of beans on a supermarket shelf are; buyers have varying preferences for different locations: someone’s “far from downtown” is someone else’s “close to my parents”.
• Each transaction is unique. A supermarket sells many identical or similar items, whereas real estate transactions are mostly one-on-one. Even if someone is selling several houses, the price discussed for one house doesn’t automatically apply to another one. Houses are also long-lasting products that change hands infrequently.
• There is no fixed price. Sellers start with a relatively high price (compared to their expectations) that they progressively decrease until they get a buyer. In turn, that buyer might try to negotiate down that price. Conversely, people who have not officially posted a sale price may still entertain spontaneous offers from potential buyers. This is especially relevant here, given that 85% of iBuyers’ purchases concern houses that were not previously in the market (SOURCE). Finally, when multiple buyers are interested in the same property, they may outbid each other as there is no “first come first serve” rule.

All of these aspects combine to mean that the so-called “real estate market” is really a loose aggregation of individual, temporary, bargaining markets of one house. As such, Zillow’s stated aspiration to become a “market maker” for real estate — similar to those in the stock market — had shaky foundations at best. We talk of the “dating market”, but what would it mean to be its market maker???

In particular, this exposed Zillow to the Winner’s Curse. This sinister-sounding concept from auction theory refers to situations which, by nature, lead to buyers overpaying their purchases. It is best understood through a simplified example, so let’s imagine that we have three iBuyers in a market. They each build a machine-learning model to predict the price at which they can profitably buy and flip a house. Because they have access to the same resources (contractors for renovation, etc.), that price is the same for all three of them. A house is a profitable flip at say $300k, regardless of which iBuyer does it. The predictive models built by the three companies are also similar: each one of them correctly predicts the price one third of the time (so $300k in the previous example), undershoots by 10% one third of the time ($270k), and overshoots by 10% one third of the time ($330k). This implies that in and of itself, each model is reasonably sound and profitable. Finally, let’s assume for the sake of simplicity that the errors of the three models are independent. One company’s model overshooting in its prediction doesn’t affect the probability that another company’s model will be accurate, or the direction of its error.

If the three companies compete with each other in a given market, they will end up each getting one third of the houses on average, but most of them bought at the inflated price. The average $300k house receives one 270k bid, one 300k bid, and one 330k bid which wins the house. In other words, even though the average bid for each company is profitable, the average winning bid is not. The winner is cursed with a self-harming success.

“even though the average bid for each company is profitable, the average winning bid is not. The winner is cursed with a self-harming success”.

You can see how this ties back to the limitations of predictive analytics. Even a model that predicts very accurately the potential reselling price of a property will lead to losses if used naively. On top of that, any volume or market share goal will make your profitability depends on the accuracy of your competitors’ models: if your competitors have bad models and overpay for properties, you can only outbid them by overpaying even more! Fortunately, game theorists and auction theorists have developed models and tools that can help mitigate these issues by modeling the strategic behaviors of other actors.

Conclusion

To recap: I don’t believe that the iBuying business is inherently flawed, and I don’t find compelling the argument that algorithms do a worse job than humans at pricing for it. However, I do believe that this line of business offers unique challenges to predictive analytics, which yields some broader insights:

• Like the proverbial hammer, predictive analytics is not always the right tool for a job. Sometimes a problem calls for descriptive or causal analytics.
• If you want your model to perform well through economic cycles and turbulence, you need to understand and factor in the fundamental, lasting drivers of human behavior.
• Some markets substantially diverges from the theoretical ideal of many buyers and many sellers exchanging identical products at a single price. When that’s the case, you need to use the relevant economic concepts, such as game theory and auction theory. Otherwise, you too will be cursed!

But wait, there’s more!

Here comes the final shameless plug. If you want to learn more about the Winner’s Curse, ̶y̶o̶u̶ ̶s̶h̶o̶u̶l̶d̶ ̶r̶e̶a̶d̶ ̶m̶y̶ ̶b̶o̶o̶k̶ you should read Richard Thaler’s book The Winner’s Curse[2] (duh!), and also follow on LinkedIn the economist John List for good measure. However, if you want to learn more about causal analytics and how to use it in business to understand human behavior, my book will show you:

• How to build causal diagrams and understand causal effects;
• How to think clearly about deeper psychological and behavioral factors;
• How to build models to profitably inform business decisions;
• And a lot of other cool things about analyzing customer data in business.

References

• [1] F. Buisson, Behavioral Data Analysis with R and Python: Customer-Driven Data for Real Business Results, O’Reilly Media, 2021.
• [2] R. Thaler, The Winner’s Curse: Paradoxes and Anomalies of Economic Life, 1st ed. 1994, kindle ed. 2012.
• [3] The Guardian, The $300m flip flop: how real-estate site Zillow’s side hustle went badly wrong, 4 Nov. 2021.
• [4] The Wall Street Journal, What Went Wrong With Zillow? A Real-Estate Algorithm Derailed Its Big Bet (paywall), 17 Nov. 2021.
• [5] Wired, Why Zillow Couldn’t Make Algorithmic House Pricing Work, 11 Nov. 2021.
• [6] Zillow Research, iBuyers Are Helping People Move in Record Numbers, 7 Sep. 2021.

---

Is Zillow “Cursed?” A Behavioral Economics Perspective was originally published in Towards Data Science on Medium, where people are continuing the conversation by highlighting and responding to this story.

from Towards Data Science - Medium https://ift.tt/2ZhhbG5
via RiYo Analytics