StackOverflow - StackOverflow
BeautifulSoup not returning full html script from airbnb search page
I don't use selenium too often but recommend the requests
lib.
Try this
from requests import get
from bs4 import BeautifulSoup
headers = {'User-agent':'Mozilla/5.0 (X11; Linux i686; rv:100.0) Gecko/20100101 Firefox/100.0.'}
res = get('https://www.airbnb.com/s/Kyoto-Prefecture--Japan/homes?tab_id=home_tab&flexible_trip_lengths%5B%5D=one_week&refinement_paths%5B%5D=%2Fhomes&place_id=ChIJYRsf-SB0_18ROJWxOMJ7Clk&query=Kyoto%20Prefecture%2C%20Japan&date_picker_type=flexible_dates&search_type=unknown', headers=headers)
soup = BeautifulSoup(res.text, features="html.parser")
url_list = soup.find_all("meta", attrs={"itemprop":"url"})
In my case, it returned 20 results, which is as many that can be displayed on one page. If you want more results to be returned then you need to scrape further pages.
The use of the Firefox user agent is very important. It provides an old scrape case usage, that a lot of webpages don't block when this agent is used.
Was this helpful?
Related Articles
Have a different question?
Can't find the answer you're looking for? Submit your own question to our community.
🛎️ Get Weekly OTA Fixes
New answers, vendor issues, and updates — straight to your inbox.