How to write an http request to get the final URL of a redirected one

[Elgregy]

Hello activepiecers,

I am new to writing HTTP request and not a coder from education so apologize if my question seems a little noobie.

I want to write an HTTP request that would take the final URL of redirected URL from google news but I am not sure how to write it.

Here is a screenshot of what I have tried but the code seems to be in python and not working on the Json body.
Looking a bit on th internet th code in python that could extract the final URL should be : response = requests.get(url, allow_redirects=True) but I am not sure how to write it in the Http request module.

If anyone has more knowledge on me on the topic I would gladly learn from him.

[menacestudio]

@Elgregy if you’re just doing a HTTP GET on that URL, you don’t need to pass anything in the body. Making a HTTP call should return the redirected URL, if not, you will have do it via code to get the new URL. Hope that made sense.

[abuaboud]

Hi @Elgregy,

Another option is to add a JavaScript code snippet, then press “ASK AI” and ask it to perform the following task. It works great for such tasks. Let me know if it works.

Thank you,
Mo.

[Elgregy]

Hello Menacestudio,

thank you very much for your support, I really appreciate it.
So, following your instruction, I have tried a simple get HTTP request on the redirected link and the body doesn’t seem to have a direct link toward the article that the google news link redirect to. I have put a screenshot of the answer below.

I guess I have to try the Code module from Activepiece to extract the final URL. I will test what Abuaboud have told me but right now it is grayed out I will set it up shortly and be back to you guys.

Thank you again a lot for the help

[Elgregy]

Hello Abuaboud,

thank you for the suggestion. I have tried to modify the .env file on the root of activepiece to add my OpenAI API but my " ASK AI " button seems still greyed out.

I am still checking how I can make it work and then I will try out your suggestion asap.

If you have any help on how to make the ASK AI work let me know.

Have an excellent day.

[abuaboud]

Hi,

Could you verify that it’s AP_OPENAI_API_KEY, and please restart afterwards? It should work. If not, you can also use it on the cloud and then copy and paste the results to the self-hosted platform in the meantime.

Also, please open a GitHub issue, I will have to check it soon

[Elgregy]

Hi Abuaboud,

hope you are doing good, thank you again for your help.

I have entered a reboot command "sudo reboot " to restart and I ran into a weird error where, the page to access my self hosted activepieces was loading indefinitely without showing the page. Thank god I did create a snapshot and ran back to it, but still ASK AI is greyed out since it went back before the reboot was made. May I ask you what command you use to restart ?

As for the .env modified file, I can attest it does have the OPEN AI API with this exact syntax. You can check it below.

So I have followed your second option which is to deploy a cloud active piece to ask AI, it, unfortunately, didn’t work and ran into an “Invalid or unexpected token”. Here is the console log :
---------error-------
/root/codes/step_2/index.js:2
throw new Error('Compilation Error:
^^^^^^^^^^^^^^^^^^^

SyntaxError: Invalid or unexpected token
at internalCompileFunction (node:internal/vm:73:18)
at wrapSafe (node:internal/modules/cjs/loader:1274:20)
at Module._compile (node:internal/modules/cjs/loader:1320:27)
at Module._extensions…js (node:internal/modules/cjs/loader:1414:10)
at Module.load (node:internal/modules/cjs/loader:1197:32)
at Module._load (node:internal/modules/cjs/loader:1013:12)
at Module.require (node:internal/modules/cjs/loader:1225:19)
at require (node:internal/modules/helpers:177:18)
at (webpack://activepieces/engine/src/lib/handler/code-executor.ts:41:65)

Here is what the generated code looks like :

export const code = async (inputs) => {
const response = await axios.head(inputs.redirectedURL, { validateStatus: status => status >= 200 && status < 400 });
return response.request.res.responseUrl;
};

You can also check the screenshot of the code module below

Let me know if you need any other information

Have a wonderful day.

[abuaboud]

Hi @Elgregy,

That’s a weird issue. I used docker-compose down and docker-compose up to restart.

I think OpenAI missed the import. Ouch!

Try this code and input the “url”:

export const code = async (inputs) => {
  const response = await fetch(inputs.url, { method: 'HEAD', redirect: 'follow' });
  const finalUrl = response.url;
  return finalUrl;
};

Or you can use this prompt:

Use fetch to follow redirects and grab the final URL.

[Elgregy]

Hi Abu,

thank you for the code and help, I have tried both solutions and it seems to be giving more or less the same URL, it just added a language query in the URL.

Here is the result of both the AI code and your code :

Google News

I will paste below the URL that is fetched as retrieved by the RSS flow just so you can see its structure and how it states when opened "redirecting to [name of the article] ":

https://news.google.com/rss/articles/CBMilQFodHRwczovL3d3dy5hZ3JvLW1lZGlhLmZyL2FjdHVhbGl0ZS9jb25kaXRpb25uZW1lbnQtc2lkZWwtbGFuY2UtdW5lLW5vdXZlbGxlLWJvdXRlaWxsZS0xMDAtcnBldC1wb3VyLWxlcy1ib2lzc29ucy1nYXpldXNlcy1ub24tYWxjb29saXNlZXMtNTg1MjYuaHRtbNIBmQFodHRwczovL3d3dy5hZ3JvLW1lZGlhLmZyL2FjdHVhbGl0ZS9jb25kaXRpb25uZW1lbnQtc2lkZWwtbGFuY2UtdW5lLW5vdXZlbGxlLWJvdXRlaWxsZS0xMDAtcnBldC1wb3VyLWxlcy1ib2lzc29ucy1nYXpldXNlcy1ub24tYWxjb29saXNlZXMtNTg1MjYuaHRtbC9hbXA?oc=5"

if you copy and paste the link above, it redirects to the direct article webpage and this is the URL that I would need as an output.

In case this helps here is a screenshot of the result code :

many thanks again for your valuable help, i hop there is a way to extract the final Url.

Hav a wonderful Sunday

[moo]

Trying the same but can’t find the .env file - do you remember the path to that? Thanks!

[Elgregy]

Hello moo,

yes I do please try following these steps :
-connect to your server through putty or other similar option.
-go to activepieces file | cd activepieces
-once here use command | ls -a

normally it should show a file called .env.

-use edit command | nano .env
-add the line of open ai as stated in the screenshot with your APIKEY.

Let me know if this worked.

[abuaboud]

Hi @Elgregy,

If I am not mistaken, the reason it doesn’t really reply with a redirect is that it opens the page and after a couple of seconds, it redirects. That’s why this snippet doesn’t work. However, if all URLs are from Google, I tried to make the snippet read the page and read the first URL after opening. With Ask AI, I had to iterate a couple of times.

export const code = async (inputs) => {
  const response = await fetch(inputs.url, { method: 'GET', redirect: 'follow' });
  const text = await response.text();
  const regex = /Opening\s*<a\s*href="([^"]+)"/i;
  const match = regex.exec(text);

  if (match && match[1]) {
    const extractedUrl = match[1];
    return extractedUrl;
  } else {
    throw new Error("No URL found after 'Opening'");
  }
}

[Elgregy]

Hello Abuaboud,

I hope you are doing well.

I have tried your code and it worked perfectly ! It does fetch the final URL of the article. Thank you SO MUCH for the support, this has been greatly helpful and I couldn’t have done it without you. I’ll mark your answer as solution since it solved the issue. Good job buddy

Have an excellent day ahead, I can continue my automation thanks to you .