BeautifulSoup not returning full html script from airbnb search page

I don't use selenium too often but recommend the requests lib. Try this from requests import get from bs4 import BeautifulSoup headers = {'User-agent':'Mozilla/5.0 (X11; Linux i686; rv:100.0) Gecko/20100101 Firefox/100.0.'} res = get('https://www.airbnb.com/s/Kyoto-Prefecture--Japan/homes?tab_id=home_tab&flexible_trip_lengths%5B%5D=one_week&refinement_paths%5B%5D=%2Fhomes&place_id=ChIJYRsf-SB0_18ROJWxOMJ7Clk&query=Kyoto%20Prefecture%2C%20Japan&date_picker_type=flexible_dates&search_type=unknown', headers=headers) soup = BeautifulSoup(res.text, features="html.parser") url_list = soup.find_all("meta", attrs={"itemprop":"url"}) In my case, it returned 20 results, which is as many that can be displayed on one page. If you want more results to be returned then you need to scrape further pages. The use of the Firefox user agent is very important. It provides an old scrape case usage, that a lot of webpages don't block when this agent is used.