PayPal Instant Payment Notification warning

brick_it_uk

Has anybody else received an email like this?

Myself and another UK-based store owner received one in the last 48 hours.

The links all check out, so it is legitimate.

Is there any action I need to take?

Lawrence

We have been seeing the errors in the logs. Ironically, it seems to be PayPal itself which has the issue. When our system asks PayPal to validate the IPN, it is occasionally responding with a server error, and so we also then need to respond with a server error. Fortunately, PayPal usually resend the IPN, and then their system starts working again.

This isn't any action users would need to take

Stellar

There was an email from PayPal a couple months ago that said something about this:

Hola,

PayPal está ampliando la infraestructura de notificaciones de pago instantáneas (IPN) que se utiliza para informar a los vendedores sobre eventos relacionados con el estado de las transacciones de PayPal. Este cambio añadirá dos direcciones IP adicionales desde las que se enviarán las IPN. A continuación se muestran todas las direcciones IP que se utilizarán para IPN.

66.211.170.66
173.0.81.1
173.0.81.0/24
173.0.81.33
173.0.81.65 (Nueva)
173.0.81.140 (Nueva)

Empezarás a recibir IPN con las direcciones IP anteriores el 18 de enero de 2021 a las 8:00 (Hora del Pacífico, PST).

Call to Action

Si actualmente utilizas alguna lista de control de acceso (ACL) o filtros para las direcciones IP de las IPN recibidas de PayPal, te pedimos que añadas las direcciones IP mencionadas anteriormente. De este modo, no te perderás ninguna IPN de PayPal.

Gracias,

PayPal

Vosblokjes

I got the same mail (in Dutch) this morning about the IPN server errors, seeing it was about BrickOwl I assumed that if fixing was needed Lawrence would take care of that

XS Bricks

English version in case it's useful.


As communicated via email in the first week of December 2020, PayPal is expanding the Instant Payments Notification (IPN) infrastructure used to notify merchants about events related to the status of PayPal transactions. This change has added seven new IP addresses from which IPNs are being sent since January 18, 2021. Below are all of the IP addresses that are being used for IPN.

66.211.170.66
173.0.81.1
173.0.81.0/24
173.0.81.33
173.0.81.65 (New)
173.0.81.140 (New)
64.4.240.0/21 (New)
64.4.248.0/22 (New)
66.211.168.0/22 (New)
173.0.80.0/20 (New)
91.243.72.0/23 (New)

Call to Action

If you are implementing any Access Control List (ACL) or filters on IP addresses for the IPNs received from PayPal, we request that you add all the aforementioned IP addresses before May 1, 2021. Doing so will help avoid missing IPNs from PayPal.

If you have any further questions about this migration activity, please feel free to reach out to Merchant Technical Support (www.paypal-techsupport.com).

Vosblokjes

This morning at 6:49 a fresh warning mail from PP about the BO IPN server.
At 7:13 a new order came in (mail from BO), 7:14 mail from PP, payment received.
But status in BO order:
23 Mar 2021 07:37 Status Change - Payment Received
23 Mar 2021 07:12 Order Placed
So a 25 min lag between ordering and the Payment confirmation.
looking at previous orders (22 Mar 13:26) there was only a minute lag, after that is becoming more and more.

Lawrence

Thank you for the update, I can see that we only received one IPN for that order and it processed successfully. I can also see a lot of sporadic 500 (Server error) responses to various different parts of the paypal process in our logs, so this does seem to be a Paypal issue.

Brick arena

FYI - I also received this message from Paypal.

Mrs Swoop

I've had a similar thing on Ebay. Several times recently I've gone through the checkout process and paid with paypal but the transaction has remained as unpaid for a while afterwards. So that also suggests that the issue is with paypal.

Emporiosa

Also received the same email at 1:45AM EST

Bricks-Shop

It seems that we receive the paypal alert email now daily. Yesterday and today at the same time.

Vosblokjes

The mail is getting in a minute earlier than the day before, 6:48 today.

@Lawrence is it possible for you to contact PP about it?
I know it is on their side to fix, but they 'threatening' to disable IPN on 'my' account and that would not be very funny has we would have to check every order and payment by hand and set paid order to payment received. At the moment not a very big issue would it be necessary because it is a slow time, but if sales would pick up not very funny to do.

vegitt

I'm now getting emails from PP about failed IPNs too.

Lawrence

I have checked our logs again, we successfully receive thousands of IPNs a day. Occasionally, when we attempt to validate these, the PayPal service respond with a server error, and so therefore we respond with a server error, to inform PayPal they need to retry the request.

Instead, it looks like PayPal interprets the brick owl server error as our endpoint permanently failing, and then sends an e-mail to every single store with a PayPal account to let them know.

As a result, we have now changed this so our system will hide the error, so PayPal think it has been successful, when it fact it failed. The downside is, when their system errors, we will not get the IPN and it will be lost. The upside is, they will stop emailing the thousands of stores every day.

Mrs Swoop

@Lawrence when you say "sends an email to every single store with a Paypal account" I haven't had any of these warning emails, I may've misunderstood something here but just thought I'd mention it.

paulvdb

I haven't had one of those emails either. I think it only goes to the store where the IPN failed.

marcuslp

I got one of these messages about an hour ago so still an issue somewhere - probably Paypal though!

justbrix

I also received this message an hour ago.

SimplyBricks

I’ve just had an email from PayPal as well!

Brick-Cat

Just got one myself,I,m honoured 😉

TechnicMadMike

I'm also still getting them

paulvdb

I got my first one today.

Graham

Received this am

Does this mean an order was lost?

Lawrence

No, it has no affect on orders

Graham

Thanks!

vonveederbrick

Also just received an email threatening to disable IPNs for my account because of brickowl use... seems concerning

VividBricks

@Lawrence we just received an email from PayPal as well. In which PayPal urged us to contact the service provider (https://www.brickowl.com/paypal/ipn) for failing IPNs. And is threatening to disable IPN for our account all together, if the issue persists.

Lawrence

Okay, it looks like there is a history page here https://www.paypal.com/ie/cgi-bin/webscr?cmd=_display-ipns-history&nav=0.3.2

Would someone be able to send me a screenshot of what that looks like on their account?

Vosblokjes

Also still getting the mails daily.
The URL gives me:
Geschiedenis IPN (Directe betaalberichten)
Functie niet beschikbaar voor privé-rekening

So can't help you there, but maybe that is the problem? I don't have a business account with PayPal.

VividBricks

@Lawrence

Lawrence

@VividBricks Thank you, if you click on the top message ID (8ML30...), could you let me know what the "HTTP response code" is?

VividBricks

@Lawrence HTTP response code 200

VividBricks

@Lawrence

beaverbrick

I'm getting them pretty much daily too now... HTTP response code 500

Lawrence

Thank you for the extra information. Looking at the IPNs we get per day, we received a steady amount every day for many years, but last week it looks like we started getting a massive amount of them. It looks like something changed on the Paypal side that's interpreting something on our side as 500, and then they send lots of retry requests.

It may be that the speed of our response is an issue, so we've made a change, and will monitor this to see if the amount of retries goes down.

Mrs Swoop

I'm really confused now. I tried Lawrence's link and it says "The IPN feature is turned off. You must turn on IPN in order to create history data.". I do get notifications of instant payments and I haven't had any warning emails so I don't want to mess with it.

Mrs Swoop

And I just realised that it's showing my payments anyway

Mrs Swoop

FYI, Paypal confirmed that my IPN is turned on and they don't why it says it isn't. They suggested I could click on the "turn it on" link anyway but I'm leaving well alone.