How to write an http request to get the final URL of a redirected one

Hello activepiecers,

I am new to writing HTTP request and not a coder from education so apologize if my question seems a little noobie.

I want to write an HTTP request that would take the final URL of redirected URL from google news but I am not sure how to write it.

Here is a screenshot of what I have tried but the code seems to be in python and not working on the Json body.
Looking a bit on th internet th code in python that could extract the final URL should be : response = requests.get(url, allow_redirects=True) but I am not sure how to write it in the Http request module.

If anyone has more knowledge on me on the topic I would gladly learn from him.

@Elgregy if you’re just doing a HTTP GET on that URL, you don’t need to pass anything in the body. Making a HTTP call should return the redirected URL, if not, you will have do it via code to get the new URL. Hope that made sense.

1 Like

Hi @Elgregy,

Another option is to add a JavaScript code snippet, then press “ASK AI” and ask it to perform the following task. It works great for such tasks. Let me know if it works.

Thank you,
Mo.

1 Like

Hello Menacestudio,

thank you very much for your support, I really appreciate it.
So, following your instruction, I have tried a simple get HTTP request on the redirected link and the body doesn’t seem to have a direct link toward the article that the google news link redirect to. I have put a screenshot of the answer below.

I guess I have to try the Code module from Activepiece to extract the final URL. I will test what Abuaboud have told me but right now it is grayed out I will set it up shortly and be back to you guys.

Thank you again a lot for the help :innocent:

Hello Abuaboud,

thank you for the suggestion. I have tried to modify the .env file on the root of activepiece to add my OpenAI API but my " ASK AI " button seems still greyed out.

I am still checking how I can make it work and then I will try out your suggestion asap.

If you have any help on how to make the ASK AI work let me know.

Have an excellent day.

Hi,

Could you verify that it’s AP_OPENAI_API_KEY, and please restart afterwards? It should work. If not, you can also use it on the cloud and then copy and paste the results to the self-hosted platform in the meantime.

Also, please open a GitHub issue, I will have to check it soon

Hi Abuaboud,

hope you are doing good, thank you again for your help.

I have entered a reboot command "sudo reboot " to restart and I ran into a weird error where, the page to access my self hosted activepieces was loading indefinitely without showing the page. Thank god I did create a snapshot and ran back to it, but still ASK AI is greyed out since it went back before the reboot was made. May I ask you what command you use to restart ?

As for the .env modified file, I can attest it does have the OPEN AI API with this exact syntax. You can check it below.
OPNAIAPIKEYACTIVEPIECES2

So I have followed your second option which is to deploy a cloud active piece to ask AI, it, unfortunately, didn’t work and ran into an “Invalid or unexpected token”. Here is the console log :
---------error-------
/root/codes/step_2/index.js:2
throw new Error('Compilation Error:
^^^^^^^^^^^^^^^^^^^

SyntaxError: Invalid or unexpected token
at internalCompileFunction (node:internal/vm:73:18)
at wrapSafe (node:internal/modules/cjs/loader:1274:20)
at Module._compile (node:internal/modules/cjs/loader:1320:27)
at Module._extensions…js (node:internal/modules/cjs/loader:1414:10)
at Module.load (node:internal/modules/cjs/loader:1197:32)
at Module._load (node:internal/modules/cjs/loader:1013:12)
at Module.require (node:internal/modules/cjs/loader:1225:19)
at require (node:internal/modules/helpers:177:18)
at (webpack://activepieces/engine/src/lib/handler/code-executor.ts:41:65)

Here is what the generated code looks like :

export const code = async (inputs) => {
const response = await axios.head(inputs.redirectedURL, { validateStatus: status => status >= 200 && status < 400 });
return response.request.res.responseUrl;
};

You can also check the screenshot of the code module below

Let me know if you need any other information :tophat:

Have a wonderful day.

Hi @Elgregy,

That’s a weird issue. I used docker-compose down and docker-compose up to restart.

I think OpenAI missed the import. Ouch! :sweat_smile:

Try this code and input the “url”:

export const code = async (inputs) => {
  const response = await fetch(inputs.url, { method: 'HEAD', redirect: 'follow' });
  const finalUrl = response.url;
  return finalUrl;
};

Or you can use this prompt:

Use fetch to follow redirects and grab the final URL.

Hi Abu,

thank you for the code and help, I have tried both solutions and it seems to be giving more or less the same URL, it just added a language query in the URL.

Here is the result of both the AI code and your code :

I will paste below the URL that is fetched as retrieved by the RSS flow just so you can see its structure and how it states when opened "redirecting to [name of the article] ":

https://news.google.com/rss/articles/CBMilQFodHRwczovL3d3dy5hZ3JvLW1lZGlhLmZyL2FjdHVhbGl0ZS9jb25kaXRpb25uZW1lbnQtc2lkZWwtbGFuY2UtdW5lLW5vdXZlbGxlLWJvdXRlaWxsZS0xMDAtcnBldC1wb3VyLWxlcy1ib2lzc29ucy1nYXpldXNlcy1ub24tYWxjb29saXNlZXMtNTg1MjYuaHRtbNIBmQFodHRwczovL3d3dy5hZ3JvLW1lZGlhLmZyL2FjdHVhbGl0ZS9jb25kaXRpb25uZW1lbnQtc2lkZWwtbGFuY2UtdW5lLW5vdXZlbGxlLWJvdXRlaWxsZS0xMDAtcnBldC1wb3VyLWxlcy1ib2lzc29ucy1nYXpldXNlcy1ub24tYWxjb29saXNlZXMtNTg1MjYuaHRtbC9hbXA?oc=5"

if you copy and paste the link above, it redirects to the direct article webpage and this is the URL that I would need as an output.

In case this helps here is a screenshot of the result code :

many thanks again for your valuable help, i hop there is a way to extract the final Url.

Hav a wonderful Sunday :innocent:

Trying the same but can’t find the .env file - do you remember the path to that? Thanks!

Hello moo,

yes I do please try following these steps :
-connect to your server through putty or other similar option.
-go to activepieces file | cd activepieces
-once here use command | ls -a

normally it should show a file called .env.

-use edit command | nano .env
-add the line of open ai as stated in the screenshot with your APIKEY.

Let me know if this worked.

Hi @Elgregy,

If I am not mistaken, the reason it doesn’t really reply with a redirect is that it opens the page and after a couple of seconds, it redirects. That’s why this snippet doesn’t work. However, if all URLs are from Google, I tried to make the snippet read the page and read the first URL after opening. With Ask AI, I had to iterate a couple of times.


export const code = async (inputs) => {
  const response = await fetch(inputs.url, { method: 'GET', redirect: 'follow' });
  const text = await response.text();
  const regex = /Opening\s*<a\s*href="([^"]+)"/i;
  const match = regex.exec(text);

  if (match && match[1]) {
    const extractedUrl = match[1];
    return extractedUrl;
  } else {
    throw new Error("No URL found after 'Opening'");
  }
}

Hello Abuaboud,

I hope you are doing well.

I have tried your code and it worked perfectly ! It does fetch the final URL of the article. Thank you SO MUCH for the support, this has been greatly helpful and I couldn’t have done it without you. I’ll mark your answer as solution since it solved the issue. Good job buddy :handshake:

Have an excellent day ahead, I can continue my automation thanks to you :innocent:.

1 Like

This topic was automatically closed 24 hours after the last reply. New replies are no longer allowed.