newpad.ca, a formerly fantastic apartment search engine
It all started out as a mess of back-end hacks, a quick-fix to a real problem: apartment hunting via the classifieds is a miserable experience. The major sites could filter by price and size of apartments, but failed at the most important criterion: location.
When Newpad's map search first came online in 2010, it grew a solid user base fairly quickly despite a virtual lack of promotion. People were telling others how great it was and how much easier it was to search for apartments when you could use a map. Innovative for its time, Newpad won an award from CIRA in 2011, and even got its thirty seconds of fame on CBC TV's "The National" in 2012.
Housing maps mashups are a dime a dozen now, even Craigslist has ceased to sue them into oblivion and responded productively by launching its own version. Times have changed, and Newpad isn't the cool shiny new thing it used to be. Using it over the course of the past two years has revealed a lot of potential for improvement. When the data feeds broke again in early 2013, a decision was made to take the service down.
The design was driven by intent to minimize necessary UI interactions - users would return frequently to check for new listings, so tasks would get repetitive quickly. Eg. the price range was set to include the 90% median prices in a given area by default, then the user's last set price range was used for subsequent visits. Search area would be initially centered on the default city coordinates, and on the user's last search (with search performed automatically) after the first visit.
Search criteria included the price range (set via slider) and location. The geographic scope of the search was set by clicking on the map - search was conducted within a set radius; a broader area could be searched by zooming out the map, a more focused search by zooming in.
The method of displaying the search results aimed to pack information into a visually tight design without cluttering the space. Each hit was shown as a dot on the map. The size of the dot corresponded to how recent the ad was, with newest being biggest. The colour of the dot reflected the price, with low end of the range mapping to pale blue and high to dark blue. This allowed the user to spot most attractive results at a glance.
A mouse-over any of the results triggered a tooltip summary: exact price, posting age, and the original title. Clicking a result brought up a more verbose summary, with actions to see the original ad, dismiss as not interesting (result removed from the map for that particular user), and flag as expired (the system would prune old postings automatically, but not in realtime)
Some of the more interesting parts of the system were invisible to the users. For instance, a common pain-point with classified listings is that people re-post their ads often, adding content without value. Newpad screened new ads and found duplicates (using a technique similar to automated plagiarism detection), so that reposts would not show up for those users who already dismissed their previous incarnations. This also allowed to track how many times a given rental was reposted - a bit of information usually missing from the classifieds.
A useful side-effect of selecting a search location was that, collectively, the users would indicate which areas were most sought after. Similarly with individual postings: the ones that were clicked-through the most were usually the most attractive. This was used to source an "Amazing Deals" feed on Newpad's FB page. The image below shows most popular rental areas in Montreal.
Newpad scraped data from Craigslist and Kijiji (part of the reason it is no longer active) - soon enough there was enough data to construct a rough heatmap of rent prices. Below is an example from Montreal, with green indicating lower rents and red higher ones. High rents roughly correspond to most sought-after areas (as per map above), with South Shore having the questionable distinction of nobody wanting to rent there despite lots of cheap places available.
A notable feature of the scraping mechanism was that it was self-scheduling. Newpad covered a dozen major Canadian cities - the challenge was to be as up-to-date as possible while minimizing the number of times source pages were scraped. In some cities, there would be tens of new postings every hour at peak times; in smaller cities there would be a handful of postings per day. The scraper was tuned to never be more than a few postings behind in each city, and it scheduled data retrieval based on historical posting volume.
The graph below shows an example record of data retrieval for Toronto over a sample 24h period. Note how retrieval is more frequent during busy posting times (late morning and mid-evening) and quite sparse in slow times (at night).
Did you know that the proportion of usage of Kijiji and Craigslist differs from province to province? The western provinces are heavily using Craigslist, with much less activity on Kijiji. Ontario and the Maritimes use both comparably often. Quebec clearly favours Kijiji over Craigslist.
The chart below shows normalized (across all cities) average posting frequency for Toronto over a 24 period.