Not every site has an API to access data from it. Most don’t, in fact. If you need to pull that data, one approach is to “scrape” it. That is, load the page in web browser (that you automate), find what you are looking for in the DOM, and take it.
You can do this yourself if you want to deal with the cost, maintenance, and technical debt. For example, this is one of the big use-cases for “headless” browsers, like how Puppeteer can spin up and control headless Chrome.
Or, you can use a tool like scrapestack that is a ready-to-use API that not only does the scraping for you, but likely does it better, faster, and with more options than trying to do it yourself.
Say my goal is to pull the latest completed meetup from a Meetup.com page. Meetup.com has an API, but it’s pricy and requires OAuth and stuff. All we need is the name and link of a past meetup here, so let’s just yank it off the page.
We can see what we need in the DOM:
To have a play, let’s scrape it with the scrapestack API client-side with jQuery:
$.get('https://api.scrapestack.com/scrape',
{
access_key: 'MY_API_KEY',
url: 'https://www.meetup.com/BendJS/'
},
function(websiteContent) {
// we have the entire sites HTML here!
}
);
Within that callback, I can now also use jQuery to traverse the DOM, snagging the pieces I want, and constructing what I need on our site:
// Get what we want
let event = $(websiteContent)
.find(".groupHome-eventsList-pastEvents .eventCard")
.first();
let eventTitle = event
.find(".eventCard--link")[0].innerText;
let eventLink =
`https://www.meetup.com/`
event.find(".eventCard--link").attr("href");
// Use it on page
$("#event").append(`
${eventTitle}
`);
In real usage, if we were doing it client-side like this, we’d make use of some rudimentary storage so we wouldn’t have to hit the API on every page load, like sticking the result in localStorage and invalidating after a few days or something.
It works!
It’s actually much more likely that we do our scraping server-side. For one thing, that’s the way to protect your API keys, which is your responsibility, and not really possible on a public-facing site if you’re using the API directly client-side.
Myself, I’d probably make a cloud function to do it, so I can stay in JavaScript (Node.js), and have the opportunity to tuck the data in storage somewhere.
I’d say go check out the documentation and see if this isn’t the right answer next time you need to do some scraping. You get 10,000 requests on the free plan to try it out anyway, and it jumps up a ton on any of the paid plans with more features.
Direct Link →
The above is the detailed content of scrapestack: An API for Scraping Sites. For more information, please follow other related articles on the PHP Chinese website!
Hot AI Tools
Undress AI Tool
Undress images for free
Undresser.AI Undress
AI-powered app for creating realistic nude photos
AI Clothes Remover
Online AI tool for removing clothes from photos.
Clothoff.io
AI clothes remover
Video Face Swap
Swap faces in any video effortlessly with our completely free AI face swap tool!
Hot Article
Hot Tools
Notepad++7.3.1
Easy-to-use and free code editor
SublimeText3 Chinese version
Chinese version, very easy to use
Zend Studio 13.0.1
Powerful PHP integrated development environment
Dreamweaver CS6
Visual web development tools
SublimeText3 Mac version
God-level code editing software (SublimeText3)
Hot Topics
CSS tutorial for creating loading spinners and animations
Jul 07, 2025 am 12:07 AM
There are three ways to create a CSS loading rotator: 1. Use the basic rotator of borders to achieve simple animation through HTML and CSS; 2. Use a custom rotator of multiple points to achieve the jump effect through different delay times; 3. Add a rotator in the button and switch classes through JavaScript to display the loading status. Each approach emphasizes the importance of design details such as color, size, accessibility and performance optimization to enhance the user experience.
Addressing CSS Browser Compatibility issues and prefixes
Jul 07, 2025 am 01:44 AM
To deal with CSS browser compatibility and prefix issues, you need to understand the differences in browser support and use vendor prefixes reasonably. 1. Understand common problems such as Flexbox and Grid support, position:sticky invalid, and animation performance is different; 2. Check CanIuse confirmation feature support status; 3. Correctly use -webkit-, -moz-, -ms-, -o- and other manufacturer prefixes; 4. It is recommended to use Autoprefixer to automatically add prefixes; 5. Install PostCSS and configure browserslist to specify the target browser; 6. Automatically handle compatibility during construction; 7. Modernizr detection features can be used for old projects; 8. No need to pursue consistency of all browsers,
Styling visited links differently with CSS
Jul 11, 2025 am 03:26 AM
Setting the style of links you have visited can improve the user experience, especially in content-intensive websites to help users navigate better. 1. Use CSS's: visited pseudo-class to define the style of the visited link, such as color changes; 2. Note that the browser only allows modification of some attributes due to privacy restrictions; 3. The color selection should be coordinated with the overall style to avoid abruptness; 4. The mobile terminal may not display this effect, and it is recommended to combine it with other visual prompts such as icon auxiliary logos.
Creating custom shapes with css clip-path
Jul 09, 2025 am 01:29 AM
Use the clip-path attribute of CSS to crop elements into custom shapes, such as triangles, circular notches, polygons, etc., without relying on pictures or SVGs. Its advantages include: 1. Supports a variety of basic shapes such as circle, ellipse, polygon, etc.; 2. Responsive adjustment and adaptable to mobile terminals; 3. Easy to animation, and can be combined with hover or JavaScript to achieve dynamic effects; 4. It does not affect the layout flow, and only crops the display area. Common usages are such as circular clip-path:circle (50pxatcenter) and triangle clip-path:polygon (50%0%, 100 0%, 0 0%). Notice
What is the difference between display: inline, display: block, and display: inline-block?
Jul 11, 2025 am 03:25 AM
Themaindifferencesbetweendisplay:inline,block,andinline-blockinHTML/CSSarelayoutbehavior,spaceusage,andstylingcontrol.1.Inlineelementsflowwithtext,don’tstartonnewlines,ignorewidth/height,andonlyapplyhorizontalpadding/margins—idealforinlinetextstyling
What is the CSS Painting API?
Jul 04, 2025 am 02:16 AM
TheCSSPaintingAPIenablesdynamicimagegenerationinCSSusingJavaScript.1.DeveloperscreateaPaintWorkletclasswithapaint()method.2.TheyregisteritviaregisterPaint().3.ThecustompaintfunctionisthenusedinCSSpropertieslikebackground-image.Thisallowsfordynamicvis
How to create responsive images using CSS?
Jul 15, 2025 am 01:10 AM
To create responsive images using CSS, it can be mainly achieved through the following methods: 1. Use max-width:100% and height:auto to allow the image to adapt to the container width while maintaining the proportion; 2. Use HTML's srcset and sizes attributes to intelligently load the image sources adapted to different screens; 3. Use object-fit and object-position to control image cropping and focus display. Together, these methods ensure that the images are presented clearly and beautifully on different devices.
What are common CSS browser inconsistencies?
Jul 26, 2025 am 07:04 AM
Different browsers have differences in CSS parsing, resulting in inconsistent display effects, mainly including the default style difference, box model calculation method, Flexbox and Grid layout support level, and inconsistent behavior of certain CSS attributes. 1. The default style processing is inconsistent. The solution is to use CSSReset or Normalize.css to unify the initial style; 2. The box model calculation method of the old version of IE is different. It is recommended to use box-sizing:border-box in a unified manner; 3. Flexbox and Grid perform differently in edge cases or in old versions. More tests and use Autoprefixer; 4. Some CSS attribute behaviors are inconsistent. CanIuse must be consulted and downgraded.
