Web Scraper API

Parameter description

1. Basic parameters

The parameters of different crawlers are different in context and need to be included in it

parameter name	description	type	Is it required?
locale	Language/region setting (such as "en-US")	string	no
source	data source type,	string	yes
geo	area	string	no
format	Required return format	[]string	no
js_render	Whether to enable js rendering false: disable true: enable Default is (false)	boll	no
context	Query context container	object	yes

2.Youtube crawling parameters

Function name	Parameter name	description	type	Is it required?
YouTube-Video details	url_list ├─ url	Video URL list, array, containing multiple video objects	array<object>	yes
	url_list ├─ url	Single YouTube video link	string	yes
Youtube-Video Search	keyword	Search keywords	string/null	yes
	continuation	Search results paging cursor (for getting more results)	string/null	no
Youtube-Video Subtitles	url	YouTube video full link	string/null	yes
	output_format	Subtitle output format, currently supports "srt"	string	no
Youtube-User Information Acquisition	url_list ├─ url	List of user homepage URLs, including multiple user objects	array<object>	yes
	url_list ├─ url	Single YouTube user homepage link	string	yes

The following example uses YouTube to grab subtitles:

"request_example": {

"locale": "en",

"context": {

"video_url": "https://www.youtube.com/watch?v=N0p05Tq7oq0",

"output_format": "srt"

}

3.Facebook crawl parameters

Function name	Parameter name	description	type	Is it required?
Facebook-User Profile	url_list	List of Facebook user or page URLs to extract profile information from	list	yes
	list	A valid Facebook profile or page URL	string	yes

Here we take the example of grabbing FB user information:

"request_example": {

"locale": "en-US",

"source": "facebook_user_profile",

"context": {

"url_list": [

{

"url": "https://www.facebook.com/zuck/"

}

]

}

4. Amazon crawling parameters

Function name	Parameter name	description	type	Is it required?
Amazon-Product Search	keyword_list ├─ keyword	Search keyword list, support multiple keyword crawling	list (object[])	yes
	keyword_list ├─ keyword	A single search keyword to crawl	string	yes
	start_pages	Starting page number, from which page to start crawling (such as the first page is 1)	number	no
	pages	Number of pages to crawl, how many pages of data to crawl in total	number	no
Amazon-Best Seller	category_list ├─ category_id	Category ID list, array, containing multiple category objects	array<object>	yes
	category_list ├─ category_id	Category unique identifier	string	yes
	start_page	Starting page number, from which page to start crawling	integer	no
	pages	Total number of crawled pages, how many pages are crawled	integer	no
Amazon Product Reviews	product_url_list ├─ product_url	Product URL list, including multiple product objects	array<object>	yes
	product_url_list ├─ product_url	URL for a single product	string	yes
Amazon product details	product_url_list ├─ product_url	Product URL list, including multiple product objects	array<object>	yes
	product_url_list ├─ product_url	URL for a single product	string	yes

Here we take the example of crawling Amazon product keywords:

"request_example": {

"geo": "US",

"format": false,

"locale": "en-US",

"source": "amazon_search",

"context": {

"pages": 2,

"start_page": 1,

"keyword_list": [

{

"keyword": "1077068"

}

]

"js_render": false

}

5.TikTok crawling parameters

Function name	Parameter name	description	type	Is it required?
TK Video Search	keyword	Search for keywords and the system will search for videos based on the keywords	string	yes
TK comments	video_id	Tik Tok video ID, used to locate the video	string	yes
	cursor	Cursor, used to paginate to get more responses	string	no
	count	Limit the number of responses returned per page	integer	no
TK User Information Search	url_list ├─ user_url	User URL list, array, containing multiple user objects	array<object>	yes
	url_list ├─ user_url	Single TikTok user homepage URL	string	yes

Here we take grabbing TK video information as an example:

"request_example": {

"source": "tiktok_video",

"context": {

"sec_uid": "911proxy"

}

6. X crawl parameters

Function name	Parameter name	description	type	Is it required?
X tweet information	user_id	The ID of a Twitter user, used to get the tweets of that user	string	yes

Here we take the example of grabbing X tweet information:

"request_example": {

"locale": "en-US",

"source": "twitter_tweets",

"context": {

"url_list": [

{

"tweet_id": "44196397"

}

]

}

7. Google crawl parameters

Function name	Parameter name	description	type	Is it required?	Optional
Google job search	job_name	Job title keywords	string	yes	-
	job_location_list ├job_location	Place list	array<object>	no	-
	experiences	Work experience screening value	array<string>	no	INTERN_AND_APPRENTICE EARLY MID ADVANCED DIRECTOR_PLUS
	has_remote	Is the position remote?	boolean	no	-
	skills	Skill keywords (comma separated)	string	no	-
	degree	Education filter value	array<string>	no	PURSUING_DEGREE ASSOCIATE BACHELORS MASTERS DOCTORATE
	employment_type	Position type	array<string>	no	FULL_TIME PART_TIME TEMPORARY INTERN
	sort_by	Sort Field (Relevance/Time)	string	no	relevance date
	page	Pagination page number (starting from 1)	integer (建议) / string	no	-
Google Events	keywords_list	Keyword list (array, containing keyword field)	array<object>	yes	-
	start_page	Starting page number (from which page to start crawling)	integer	no	-
	end_page	End page number (the page to be captured)	integer	no	-
Google Image	keywords_list ├keyword	An array of keyword objects, used to specify search keywords	array<object>	yes	-
	keywords_list ├keyword	Search keywords	string	yes	-
	start_page	Starting page number (which page does the crawl start from)	integer / null	no	-
	end_page	End page number (the page to be captured)	integer / null	no	-
Google Autocomplete	keywords_list ├keyword	Keyword object array	array<object>	yes	-
	keywords_list ├keyword	Search keywords to get auto-complete suggestions	string	yes	-
Google Finance	keywords_list ├keyword	Keyword object array	array<object>	yes	-
	keywords_list ├keyword	Search keywords (such as company name, stock code, etc.)	string	yes	-
Google Search	keywords_list ├keyword	Search keyword object array	array<object>	yes	-
	keywords_list ├keyword	Search keyword strings (such as product names, brand names, etc.)	string	yes	-
	start_page	Starting search results page number, from which page to start searching (such as 1)	int/null	no	-
	end_page	End search results page number, the page to end the search on (such as 3)	int/null	yes	-

Here we take crawling Google job as an example:

"request_example": {

"locale": "en-US",

"source": "google_job_search",

"context": {

"page": "1",

"degree": "BACHELOR",

"skills": "python, sql",

"sort_by": "date",

"job_name": "software engineer",

"locations": [

{

"location": "London"

}

"has_remote": true,

"experience_list": [

{

"experience": "MID"

}

"employment_type_list": [

{

"employment_type": "FULL_TIME"

}

]

}

"response_example": {

"result": [

{

"job_href": "jobs/results/108913291534705350-senior-software-engineer-generative-ai-google-workspace?q=Software&has_remote=true&page=1",

"job_title": "Senior Software Engineer, Generative AI, Google Workspace",

"job_detail": "Minimum qualifications....",

"job_remote": "Remote eligible",

"job_company": "Google",

"job_location": "New York, NY, USA; Boulder, CO, USA; +4 more; +3 more"

}

]

}

Web Scraper API

Parameter description

1. Basic parameters

2.Youtube crawling parameters

3.Facebook crawl parameters

4. Amazon crawling parameters

Here we take the example of crawling Amazon product keywords:

5.TikTok crawling parameters

Here we take grabbing TK video information as an example:

6. X crawl parameters

Here we take the example of grabbing X tweet information:

7. Google crawl parameters

Here we take crawling Google job as an example:

Contact us with email