Codeblock Tutorial on how to get Content from URL

[Momo]

I found a few posts around this topic like this one:

Get the contents from a URL - Which settings should I use?

So, I was just made aware of the “Send HTTP request” piece, and it can read a URL and get the content using the GET function. Very well But I struggle. It gets all the content in the , but I only want the content without the CSS, javascript, head etc. How can I tell this piece to clean up the content so I only have the text content? Or use another piece to do just that? I want to use this in OpenA for further work, but for now I can’t because the whole code from the URL is far too big.…

And this one:

How to extract a part of the content of a Web page

I would like to know how to extract part of the content of a web page. I have a workflow to get the Feed and summarize it in ChatGPT. I am able to get the full text of the body with “Send HTTP reqest”, but I want to extract only the elements in which the div is main.

But I and I feel like a lot of others would get a ton of value for a short (ideally video)-step by step tutorial on how to filter the usable content out from a http request.

This will allow anyone to make all kinds of research focused automations. Automatically rewriting content and posting it on their social media becomes possible. Be it from an rss feed or from a list of links where each one gets summarized and then there comes out a blogpost with 5 sources.
All becomes possible, if we just get that one piece sorted.

[Bram]

Hi Momo,

Have you tried www.apify.com? it has a very good free tier which allows you to scrape a lof of content from many sites, It is also very easy to integrate into AP.

Also you are able to scrape leads from www.apollo.io which works quiet well and there is a video on YT about it. And they have scrapers that work with ChatGPT too.

I am also in touch with freelancers for custom builds, on APIfy. it’s not expensive

KR

Bram

[Momo]

Hi Bram,
Thank you. And for Apify I only need a webhook to get the content from a blog for instance.
That is great. I will aim to figure this out somehow. If you got the link to the Video I would be grateful.

All the best,
M

[Preben]

Do you have any tutorial on how to use apify with AP? @Bram , BTW do you build flows for people?

[Bram]

Hi Momo,

Yes that is correct and with a HTTP Request you can start the Actor. Let me try to create a static example on a custom Actor, but they all work the same way.

Receiving Data via a webhook and how to get the datasets

Go to your actor and go to Integrations:

Hit this blue button

Select HTTPS Webhook

Press “Configure”

Now create a new flow in Active Pieces starting with a webhook and Copy the webhook URL

Set the event type to Run suceeded and Paste the webhook into the URL

Now go back to active pieces and you webhook and press “send data” with me it is “Gegevens verzenden”

Head back to APIfy and scroll down your actor and press save an test, Your webhook should receive something like this

Next add a new HTTP step to your flow and make it GET

Triggering your Actor via Active Pieces

Select: API → API Clients

The click on cURL ans SCROLL DOWN!!!

Use the URL, POST type and -h (header) and set them into Active Pieces.

The set-up in this case should look like this in Active Pieces

Please remove the “$API_TOKEN” from the URL and replace it with your token, found here, when you SCROLL UP in APIfy API Client:

Then you need to copy the Body from your API Client, in this case the green text starting from { and ending with }

{
“searchStringsArray”: [
“restaurant”
],
“locationQuery”: “New York, USA”,
“maxCrawledPlacesPerSearch”: 50,
“language”: “en”,
“onlyDataFromSearchPage”: false,
“includeWebResults”: false,
“scrapeDirectories”: false,
“deeperCityScrape”: false,
“reviewsSort”: “newest”,
“reviewsFilterString”: “”,
“scrapeReviewerName”: true,
“scrapeReviewerId”: true,
“scrapeReviewerUrl”: true,
“scrapeReviewId”: true,
“scrapeReviewUrl”: true,
“scrapeResponseFromOwnerText”: true,
“searchMatching”: “all”,
“placeMinimumStars”: “”,
“skipClosedPlaces”: false,
“allPlacesNoSearchAction”: “”
}

Should look like this in AP:

Then you can try it by testing the step.

This should be the result:

Then in APIfy you can see the Actor running:

To collect the scraped data in a webhook there are a few options available, I assume you will run this actor more often, and if not it doesn’t matter but this is the way to save more results.

[Bram]

Hi @Momo ,

Sorry for the late reply I had the above post as a draft ready to be posted…

Hope this helps, @Preben

[Bram]

I currently don’t build flows for people, I also think my hourly rate will be a bit to expensive. You can try the marketplace option here. Or dm me if you are interested

[Momo]

Wow - thank you so much!!! I just saw it - wasn’t logged into the community as often anymore. Super helpful.