How to extract a part of the content of a Web page

[Takeru]

I would like to know how to extract part of the content of a web page.
I have a workflow to get the Feed and summarize it in ChatGPT.
I am able to get the full text of the body with “Send HTTP reqest”, but I want to extract only the elements in which the div is main.

[Dennis]

What's your scraping use case?

Hi everyone, I can notice great interest in scraping / monitoring websites and combining that with LLMs. If I have enough input on how everyone would like to do the scraping in an automation tool and why they’d like to do it, I could possibly run an internal initiative to make it possible. So, what’s your website scraping use case that you’d like to automate?

[Takeru]

I did not quite understand the reference.
I consulted with AI and they output the code below, do I just paste this somewhere?

import requests
from lxml import html

response = requests.get("URL")
tree = html.fromstring(response.content)

main_div = tree.xpath('//div[@id="main"]')[0]
print(main_div.text_content())

[Takeru]

Can someone tell me if anyone can figure this out?

[Abdul]

Hello @Takeru

sorry for the late response, but could you please explain further what you mean by
but I want to extract only the elements in which the div is main