What's your scraping use case?

Hi everyone,

I can notice great interest in scraping / monitoring websites and combining that with LLMs.

If I have enough input on how everyone would like to do the scraping in an automation tool and why they’d like to do it, I could possible run an internal initiative to make it possible.

So, what’s your website scraping use case that you’d like to automate?

This would be an awesome function, a few options that personally come to mind:

  • Customer Reviews and Feedback: Automate the collection of reviews and feedback from a range of platforms. Some of these platforms don’t even actively notify about new reviews.
  • Competitor Analysis: Track competitor websites to stay updated on their product offerings, pricing strategies, and marketing tactics.
  • Trend Spotting: Monitor industry news, online forums, and social media platforms to stay informed about the latest trends, innovations, and market shifts.
  • Regulatory Compliance: Keep updated on legal and regulatory changes to my business sector.
6 Likes

This would be great, I tried some application like visualping and wachete. Some of my use case:

  1. Gain a competitive edge: Automate the monitoring of competitors’ prices to stay informed about any changes and adjust your own pricing strategy accordingly.
  2. Track YouTube channel growth: Automate the process of monitoring subscriber counts to keep a close eye on the progress of your channel and make data-driven decisions to optimize your content.
  3. Stay updated with exchange rates: Automate the monitoring of exchange rates and receive notifications when there are significant changes, allowing you to make timely decisions in response to market fluctuations.
2 Likes

That will be an awesome function.

  • Track text change on specific fields

I am looking forward to seeing this on activepieces.

2 Likes

Hi guys,

Recently I started using Apify, you can try it for free Apify: Full-stack web scraping and data extraction platform works really well and had some great results like searching for companies on Google maps based on location and based on their website scrap their e-mailadres and phone number. I do believe AP should keep their focus on creating a better integration platform to become real competitors of Zapier, Make, Pabbly and so on. So it would might be great to team up and create some great pieces which can scrap “anything”

Not hear to change minds but it think it’s good to not create new software if there is something good available.

Kr Bram

3 Likes

Ohh that’s awesome!! :heart_eyes:

PLS tell me if you do develop scraping options!! I’d love it for outreach.

It would be a dream come true to scrape biz emails/leads to people that might be interested in your services! It take so much manual work to find those people and email addresses.

For example, scrape biz emails from google maps, filtered by area, type of business etc.

The extra Bonus would be for all scraped info to be imported into a Excel sheet or CRM platform.

So you automatically scrape and save each new email with details like:

Company role/website/type of business/mo company earning/area.

It would be any marketers dream!!

  • C
2 Likes

Read weather data from the BOM (ie the Australian government weather which is very reliable but doesn’t have a traditional API) send a notification when weather changes.

Create simple LLM “agents” who can research using google, go to a web page, read the web page, summarise the web page, draft blog posts based on research

Google a local band or artist, see if they have an instagram, facebook, spotify, or bandcamp, verify the links are actually for the same artist, add those verified links to a database or a promotional social media post.

2 Likes

Am looking on how to perform automated or scheduled web scraping.

For instance, every week new product appears on the AppSumo website, and I want to scrape that data automatically every week and pull it into a Google Sheet, but only the unique data that arrives each week.

2 Likes

I’d like to scrape flight prices, travel advisories, hotel reviews, and new blog posts about living overseas to create a “feed” of fresh information on traveling to specific destinations!

2 Likes

Will be the best addition for me. I like to automate list creation:

  1. Collating best tools - scrape tool info from websites
  2. Collate trends - scrape news from websites
  3. list of use cases - scrape use case listings
2 Likes

I’d like to scrape wedding vendors from google or yelp and add onto my brilliant directory website as a member(user) which ‘‘activepieces supported Brilliant Directories’’
I really need that feature in order to use my activepieces account. Thnx

1 Like

This will be great for eg. lookiing to automate sale/deal or offer posting website where user submitted sales are posted and once the post goes live, will like activepieces to scrape the webpage and use that image on that post, share on social media and include it as a media file.

This could be great for SEO purposes, especially if we can retain the headline structures and so on.

Hi Guys,

For some of the readers scraping might be new, also the scraping via API, I currently have this set up for client with Frontly, Apify and Active Pieces. If some of you are interested I can create a video how I’ve done the set-up so you can use it too. @here

Let me know.

KR Bram

5 Likes

This would be wonderful!

2 Likes

This would be great. Thanks.

1 Like

Perfect, will see what I can do, busy year end but maybe I find some time in between.

2 Likes

I would love to simply get a news link from an RSS feed - and get just the main content out of it.

Hi @here,

So I’ve actually found some time this morning to create a video, sorry for the poor quality of sound also this is one of the first times I am recording myself. But I hope you can take that for granted. I hope this will help you understand the use case and how I have set things up to help my client to extend their network with ease by using three different tools:

Here is a link to the video:
Scraping the web with Frontly, Apify and Activepieces.mp4

In case of any questions about this use case or other cases, please let me know.

I’ve also set-up the feature request for APIFY, APIFY: Webscraping so please vote this one up if you think this piece should be there.

KR Bram

6 Likes