Hi everyone!
We ran into a sudden issue on Aug 28, ~11:30 AM GMT where several of our workflows started failing on Cloud, specifically any workflow using the Ask ChatGPT piece or the HTTP piece began failing immediately. We eventually discovered that newer versions of both pieces were available; upgrading to them resolved the issue.
Errors we saw:
- OpenAI/GPT piece:
Missing tiktoken_bg.wasm at .../piece-openai/src/index.js:7:533218
- HTTP piece:
Request with GET/HEAD method cannot have body
Questions:
-
Why can a newer piece version break older versions that were previously working? Is there a way to get notified in advance when a piece version we’re using is being deprecated, so we can upgrade proactively instead of discovering it via failures in production?
-
Is there a way to enable automatic upgrades for pieces (globally or per-piece), instead of having to manually upgrade each one individually?
-
Is there a way to set up better alerting or export run telemetry (e.g., to Grafana/Prometheus) so we can build our own dashboards and alerts? This would help us catch issues like this one much earlier next time.
Thanks in advance for any guidance!
Hello @hdninenine,
I’m assuming your account is on our cloud not self hosted right ?
We are going to debug this issue, Can you please share a full flow url that has seen this failure.
Thanks for your feedback and help
Hey @hdninenine,
For the openai issue . that was on us . we had a global migration to resolve memory issue applied to all pieces . openai piece seems to have been upgraded to a broken version 0.10.0
You don’t need to do anything . if we ever notice broken piece versions we auto upgrade them
For the Http piece . can you share which flow/run failed with that message .
For alerting . currently we have Alert email feature . that you can use in the project settings in top right nav bar
Thanks @ChakerAtallah,
for the HTTP piece I lost the run to be honest, if I get it again will send it to you, but once I upgraded the piece everything back to normal.
For alerting, maybe I’m looking for something more advanced ways such as Prometheus so we can build our analysis and alerting on runs. Is this available now on paid plans or self-hosted option? if not any future plans to build such a functionality?
For self-hosted we provide an endpoint that export queue metrics to Prometheus . but it only sends jobs count so I don’t think it helps much in your case
Otherwise . the more advanced feature we have is event streaming . it sends webhook to an external api that you can configure
Hello @ChakerAtallah, thanks for the info 
The issue happened again with another piece (HTML to Markdown), the error below:
When I upgraded the piece the workflow worked again, the main problem here is that we don’t get notified that a the version we are using is getting deprecated so we have to upgrade
