We do this by running A/B tests on new features and ideas to see how they influence user behavior.
Fig. 1
The original card didn't have much structure, leading our users to remark that it was difficult to use. On cards without pricing discounts, it's difficult to understand where the price is without scanning deeply.
Research
While poring through notes of ongoing user research tests (Booking runs full-funnel user research tests every quarter for this product), I identified some user issues with the current search results card implementation.
Users were telling us that it was difficult to scan our search results, because we weren’t placing things logically and individual cards were “messy.”
Brainstorming
The team sat down together and decided to tackle this problem in our next sprint. I volunteered to work on this story, and came up with some wireframes that showed how we could improve the hierarchy and information display within cards. The initial thought, which we ended up testing, was that a separation between a property's metadata (facilities, room information, location) and the property's review score, price, and CTA (the main decision-making information on each card) would make it easier for users to understand how to differentiate between a long list of properties.
Fig. 2
The new card had to support all of the cases that the current card did, including having discrete areas for the property photo, name, star rating, reviews, facilities, and price.
Based on this wireframe, I got buy-in from all the stakeholders involved in this area. We added this story to our backlog.
Implementation
Then, as part of that week's sprint, I implemented the design changes in HTML and CSS, taking into account all of the other running A/B tests in the area and RTL languages (Booking currently supports 43 languages, including Arabic and Hebrew).
Fig. 3
My new implementation — it separates decision making content from the property's characteristics into different columns, making it easier to compare information between different cards.
Validation
We ran this through a quick, informal user testing session, by piggybacking onto another team's session. We could only show the changes to two users, who both identified the newer version as better — the reasons they mentioned echoed the same sentiments as the previous user test insights we had gathered.
Designers use Booking's A/B testing tool to validate the impact of their changes. After running this as an A/B for a predetermined amount of time (based on traffic and expected impact), data showed us that users saw more properties (a 3% increase, an amazing change for such a highly optimized site), bounced less, and ran more searches with date information (a 2.6% increase) — all positive things, but the change didn't increase this test's main metric: overall net conversion (essentially, the number of bookers, minus the number of cancellers).
Post-mortem
While this wasn't the first time that quantitative and qualitative data showed us different results, we realized that there are flaws with relying only on qualitative research when making informed decisions — what the two user test participants mentioned they preferred wasn't what was preferred by our general audience. We also realized that making the CTA more visible was one of the reasons this redesign didn't help us increase net conversion — users weren't ready to continue onto the next step, and this change pushed them further, prematurely. The lessons learned here have informed future A/B testing — another team took this reorganization and tested it on our desktop experience, where it was successful.