Looking for help resolving PayPal IPN failures

[bugman]

This question is related to another one of mine, but a little different so I am posting it anew & referencing the old topic (which was (viewtopic.php?f=65&t=18754).

THE ISSUE - we cannot seem to eliminate PayPal Instant Payment Notification failures.

All is good (and by that I mean only an occasional IPN failure) for 2-3 weeks maybe, then we'll get a flurry of IPN failures. My hosting provider & developer (Rich Guarino at A3IT Solutions) is great, and has been doing his best to find a resolution, but we're not there yet.

There seem to be two reasons for the IPN failures:

1) Apparent time-outs while waiting for IPN's
2) Changing IP addresses that the IPN's come from

For the time-outs issue, how can we (Rich!) change the time that Able listens for the IPN response? That would at least help until we can get the bigger issue (#2) resolved.

And where #2 is concerned... Rich has what I believe is termed a "PCI CERTIFIED" environment?, and only allows inbound traffic from known IP addresses through his firewall. Apparently PayPal uses a 3P provider (AKAMAI) for some of the IPN communications, and they (Akamai) sometimes randomly change the IP addresses they send the IPN's back from. When we find that it happened Rich adds the new IP's to his firewall Allow list & all (or mostly all) is good - for a while - then new IP's pop up again and we have to add them, etc, etc, etc. Meanwhile we have to watch the site for failed PayPal IPN's which really puts handcuffs on automated transaction flow!!

Rich is good & I like him, but he willingly states that he is not a PayPal integration expert, so he agrees that we would both be well served by finding someone that IS more proficient in PayPal integration to assist with this issue, so... ANYBODY WANT SOME WORK???

Both Rich and I have discussed this with Paypal support multiple times, they admit that "the IPN system isn't 100% reliable", and they suggest using their API system instead.

OK, that's their PayPal PayFlow Pro Gateway, right? I DO have that setup for credit card processing (and it works fine), but in the PayFlow Pro gateway configuration in the backoffice, "PayPal" (regular PayPal account balance payments) is not an option - only credit cards.

Is there a way to enable the PayFlow Pro gateway to process regular PayPal account balance payments as well as credit card payments?

Any help is greatly appreciated!

[mazhar]

Few quick internet searches suggest that other people do have this issue with AKAMAI and paypal IPNs. Have you tried asking PayPal and maybe they could point out some fire configurations like possible IP ranges?

[bugman]

Yes, I have discussed with Paypal numerous times. They did indeed supply a document with IP ranges, and to my knowledge Rich does have them all on ALLOW list in his firewall (I will triple-check with him to be SURE he has the entire IP RANGE in his firewall), but apparently they still throw in some new undocumented ones periodically. The Paypal tech I discussed this with said "yeah, they (the IP's) can change"! (if there was a good emoticon of fire shooting out of my eyes I would insert it here!)

It boggles my mind that a company as big (and presumably profitable) as Paypal would allow such hiccups to remain unresolved.

Am I correct in the belief that the PayPal PayFlow Pro" gateway is what Paypal considers its "API"?

If so, is there a way to process Paypal account payments (not credit cards) via that gateway? Able doesn't list "Paypal" as an option for that gateway in the back end... is there a way to add Paypal as a payment method for PayFlow Pro?

The Paypal techs have suggested that it is possible, but then abruptly state "we have no control over how individual shopping cart platforms integrate our products - talk to Ablecommerce". (HEY - I found that emticon!)

Thanks for your answer and for any and all other thoughts!!!!!

[Katie]

I am truly sorry for your frustration. Sometimes I think it's the big companies that provide the worst support.

Am I correct in the belief that the PayPal PayFlow Pro" gateway is what Paypal considers its "API"?

To better understand the (PayPal) PayFlow Pro gateway, you'll need a bit of history. It was originally implemented in AbleCommerce years ago, and was not part of PayPal. It was a Verisign company and was kept in the software for backwards compatibility, plus it was renamed to reflect Paypal's acquisition.

If you want to accept credit cards via Paypal, then I would suggest using the Paypal Website Payments Pro integration. It is just a few more configuration items on the std. Paypal payments page. That is what we use.

I honestly don't know what Paypal considers PayFlow Pro and how it differs from Website Payments Pro. I think that the latter is their preferred gateway for processing credit cards, but the downside is that they want you to use Paypal Express in conjunction with Website Payments Pro. However, I'm sure you can manually disable the option in AbleCommerce if that is what you wanted.

If so, is there a way to process Paypal account payments (not credit cards) via that gateway? Able doesn't list "Paypal" as an option for that gateway in the back end... is there a way to add Paypal as a payment method for PayFlow Pro?

If you want to use std Paypal, then you have to configure it via the Paypal configuration page and create/assign the "Paypal" payment method. It will appear for the admin users and the retail users. However, if you are an admin, you can select std. Paypal as a payment method, but it will not allow you to make that payment using someone else's Paypal account.

I hope that I was able to help answer some of your questions. Sadly, I don't know that there is any solution for missing IPN's though.

Katie

[bugman]

Some interesting stuff here (at least to me!).

I spent some time on the phone with a good Paypal Merchant Support tech named Chris yesterday. He helped me get my head around a couple things a little better, but the experience also produced some confusion as well.

Firstly, he said that Paypal PayFlow Pro and Paypal Website Payments Pro are not exactly the same thing, but they are the same TYPE of thing - they are both API's to process credit cards. He said "one is a gateway that uses 3 credentials for access, the other uses FOUR credentials" (I forget now which was which), "but NEITHER is used to process payments from Paypal ACCOUNT BALANCES - just credit cards".

He said the tools that provide Paypal ACCOUNT BALANCE payments are:

1)PayPal EXPRESS CHECKOUT - (the YELLOW button that Ablecommerce displays in the mini-cart control), and...

2)PayPal STANDARD - (the blue-ish "PayPal Checkout" button Able displays on final page of the Checkout process??).

(Of course when both of those swoop the user out to login to their Paypal account to pay, the user can THEN choose to pay with a credit card from within their PP account, but neither of those is designed to process credit card payments without logging in to one's Paypal account.)

Now... (and this is where it really got interesting - and confusing)... Chris noticed that in my Paypal merchant account, the URL to return the IPN's to was listed as "/ProcessPaypal.asHx". He said it needed to be .asPx and not asHx. I told him I had copied that straight from Ablecommerce's back end, he said it was wrong.

To confirm it, he ran a tool from his end to test my IPN connection (with my setting at asHx), and he said it failed - could not connect. He had me change the setting to asPx, he ran the test again, and said the result was "connection established, handshake successful" (or something to that effect - a good result). We changed it BACK to asHx, he ran the test again - it failed. We changed it back to asPx, he ran his test again - said it was successful.

I went back to my live Ablecommerce site, processed several real orders, making payment with my personal Paypal account, and received the IPN for all three of them immediately!

So... I left it set the way he said "asPx".

I then browsed the local backup copy I keep of my Able site files looking for a file named ProcessPayPal.asPx, and (as I suspect you may already chomping at the bit to say), I did not find one - apparently such a file doesn't exist (at least not in MY Able site files) - only ProcessPayPal.asHx which does exist in the root folder of my site files.

BUT... since his tests appeared to make a connection (which I cannot understand), I decided to leave it set to asPx.

I left it this way for about 6 hours, during which time I received several more real orders from customers, and got the IPN's just fine.

But then, I had another live IPN failure - an order was indeed paid for with a PP account balance, but in the back end Able showed "waiting for IPN". So, since that didn't really "completely" fix the problem - and since I cannot help but think he is WRONG about that since no such file exists in my site - I changed the setting back to asHx. I have received another dozen or so orders since then, and so far all have gotten the IPN's just fine.

But based on past experience, I fully expect the wheels to come off again (more IPN failures) at some point.

ANOTHER interesting point - while doing all this, I also was trying to determine what (in Able) controls each of the two non-credit card Paypal payment methods (the yellow Express Checkout button & the blue Paypal Checkout button (which seems to equal "PayPal Standard"??). So, under the "PayPal" gateway (NOT "PayFlow Pro" gateway - just "PayPal" gateway) in the Able backoffice, under "Payment Methods" in that gateway, I UNchecked the only one box that I had checked which is "Paypal" (I am using the PayFlow Pro gateway for credit cards so none of the cc boxes were checked in this "PayPal" gateway). I expected this might remove one or both those Paypal buttons & disable my ability to facilitate "regular" Paypal account payments, but it did not - the yellow Express Checkout button still appears in my mini-cart control, and the blue Paypal Checkout button still appears on the final page of my Checkout process - and both work just fine (both take user to Paypal to login to their PP account).

This is still how I have it set now, and all has seemed fine today - numerous orders have been paid for with Paypal, and no issues that I am aware of (no customer complaints).

So... although my understanding is improving (I think!), I still have some questions:

1) Do I need to have both gateways setup in Able? The "PayPal" gateway has checkboxes for credit cards as well as a checkbox for "Paypal"... if I check the cc boxes to enable this gateway for credit cards, why would I still need the PayFlow Pro gateway?

2) Since I now have "Paypal" UNchecked in the Paypal gateway and customers are still abel to login to their Paypal accounts to make payment, what is the Paypal checkbox there for? What does it do? How am I able to get PP payments with it UNchecked?

3) What controls the two Paypal payment buttons (yellow Express Checkout & blue Paypal Checkout on final Checkout page) in my Able site? I *THINK* (from testing back when I was building this site) that when I completely REMOVE the Paypal Gateway (not PayFlow Pro one), the Paypal buttons disappear, & then there was no option to pay with Paypal - is that corect?.

4) Am I the only one that is confused by all this?! :-O

Any further thoughts or comments are MOST welcomed!!

[Katie]

1) Do I need to have both gateways setup in Able? The "PayPal" gateway has checkboxes for credit cards as well as a checkbox for "Paypal"... if I check the cc boxes to enable this gateway for credit cards, why would I still need the PayFlow Pro gateway?

You don't need to implement the PayFlow Pro gateway as long as you can receive credit card payments by enabling PayPal's Website Payment Pro.

2) Since I now have "Paypal" UNchecked in the Paypal gateway and customers are still abel to login to their Paypal accounts to make payment, what is the Paypal checkbox there for? What does it do? How am I able to get PP payments with it UNchecked?

That's odd. I just tested this and you are right. The Paypal payment option should disappear if it's not assigned to the Paypal gateway. I'll report the issue. As a work-around, you would simply need to delete the payment method to remove it from the checkout.

3) What controls the two Paypal payment buttons (yellow Express Checkout & blue Paypal Checkout on final Checkout page) in my Able site? I *THINK* (from testing back when I was building this site) that when I completely REMOVE the Paypal Gateway (not PayFlow Pro one), the Paypal buttons disappear, & then there was no option to pay with Paypal - is that corect?.

The blue Paypal button should only show up if the Paypal payment is created and assigned to the gateway. But as mentioned, there is a bug and to remove it, you'll need to delete the Paypal payment method.

The yellow Paypal express button automatically appears when you are using Paypal Website Payments Pro. This is Paypal's requirement - not us. As I mentioned earlier in another post, it would probably be easy to hide the Express Checkout button by making a manual code change.

Keep in mind that Express checkout is more reliable than std Paypal payments because the customer is effectively going back and forth between our checkout and Paypal's system. Once the payment is made, then the customer is brought back to Able and completes checkout. I don't think it uses IPN.

Now....about that IPN and what the Paypal tech was saying in regards to using the .aspx instead of .ashx. That doesn't make any sense to me. As you've already found, the .aspx page doesn't even exist so I am wondering how he can get any successful tests. This IPN reference has been this way for about 7-8 years (maybe longer), and if it was wrong, then I'm sure we would have found out a long time ago.

Very interesting though...

Have you thought about asking your network administrator to open the firewall to inbound requests, at least temporarily until to see if that solves the issue?

Have you ever tested your SSL to see how it ranks? I use this website because it gives you lots of good information -
https://www.ssllabs.com/ssltest/index.html

4) Am I the only one that is confused by all this?! :-O

This is crazy stuff for sure. It took me a long time to get my head around the different Paypal configurations. I hope you're starting to have a better understanding as well.

Thanks,
Katie

[bugman]

Thanks Katie.

Although we've gotten dozens of orders that have successfully been paid with Paypal since I unchecked that Paypal payment method box in the Paypal gateway, today we've had three orders that DID actually get paid properly (money is in my PP account), but in Able backoffice there was NOTHING under the Payments tab for these orders - no transactions at all - not even "Waiting for IPN".

To be safe I re-checked that checkbox, so... I'm back like I have been for the last couple months (my site just went live in early January).

So... the yellow Paypal Express Checkout button is produced when Paypal Website Payments Pro is active, right? And if I understand you correctly, if I delete the Paypal PAYMENT METHOD, it will eliminate the blue PP checkout button, but as long as I have the Paypal GATEWAY active (which is Paypal Website Payments Pro, right?) I will still have the yellow Express Checkout button active, right? Will it ONLY appear in the mini cart control, or would it appear on the final checkout page as well if I eliminate the "regular" blue PP button by deleting the PP payment method?

Since the yellow Express button (because of the payment process it uses) doesn't use IPN at all may be more reliable, I may want to try that. At first run through, it seems a bit more cumbersome to me (my site/PP site/back to my site to REALLY complete the order), but... if it's more reliable overall, may be worth trying to get customers comfortable with it & ditch the blue button.

As always, any thoughts appreciated (you have been consistently responsive to my many questions & I do appreciate it!!!).

My network admin (hosting provider) isn't keen to quickly loosen his firewall policies, so... I continue to battle the IPN issue.

(Is there an email address I could use for you if I had something I'd prefer to not post publicly on the forum? Thanks...)

[jmestep]

bugman:
This is a little off topic, but do your Paypal payment method orders show up OK in Google Analytics?

[bugman]

Judy,

Thanks for chiming in.

I don't have GA set up for this website yet. I have just added it to my current GA account, and asked my developer to put the tracking code in place, but... I have his plate full with lots of other needs that are probably going to delay implementation for a bit.

Also, I'm no GA expert, so... if you know of a good website or tech document that will help me get up to speed on GA, please share a link!

[Shopping Cart Admin]

And where #2 is concerned... Rich has what I believe is termed a "PCI CERTIFIED" environment?, and only allows inbound traffic from known IP addresses through his firewall. Apparently PayPal uses a 3P provider (AKAMAI) for some of the IPN communications, and they (Akamai) sometimes randomly change the IP addresses they send the IPN's back from. When we find that it happened Rich adds the new IP's to his firewall Allow list & all (or mostly all) is good - for a while - then new IP's pop up again and we have to add them, etc, etc, etc. Meanwhile we have to watch the site for failed PayPal IPN's which really puts handcuffs on automated transaction flow!!

The paypal IPN notification comes in on a standard URL and port (443) just like the people buying crickets on your website, the firewall is not the source of the issue unless your site is an intranet site (it isn't).

[jmestep]

bugman,
Here is a link to Able help that shows how to enable Google Analytics.
http://help.ablecommerce.com/mergedProj ... lytics.htm

I had asked about the tracking because we have a site where Paypal orders rarely show up, but this site has custom code which we did not do. I was just wanting to compare it with another site which is using the Able Paypal code to try to help troubleshoot the problem.
Thanks

[jguengerich]

The paypal IPN notification comes in on a standard URL and port (443)...

The destination url / port and the source IP address are two completely different things. If the firewall is configured to only allow inbound connections from specific IP addresses, and the PayPal IPN comes in from an IP address that is not on that list, it will get blocked.

[jguengerich]

I'm wondering if he could change his firewall configuration to allow any IP address to create an inbound connection to the ProcessPaypal page. I'm not a PCI compliance expert, so maybe he can't. It just seems like the firewall would have to already be allowing any IP address to create an inbound connection to your site's other pages though, otherwise how could anyone access your site?

[Katie]

So... the yellow Paypal Express Checkout button is produced when Paypal Website Payments Pro is active, right? And if I understand you correctly, if I delete the Paypal PAYMENT METHOD, it will eliminate the blue PP checkout button, but as long as I have the Paypal GATEWAY active (which is Paypal Website Payments Pro, right?) I will still have the yellow Express Checkout button active, right? Will it ONLY appear in the mini cart control, or would it appear on the final checkout page as well if I eliminate the "regular" blue PP button by deleting the PP payment method?

Answers are yes, yes, and yes. The express checkout (yellow) button only appears in the mini-cart and basket page because it has to be initiated before the checkout process starts.

Katie

[bugman]

Thanks to all who have weighed in on this!

JGUENGERICH - it's my understanding that indeed my hosting provider's firewall is configured to only allow inbound connections from specific IP addresses, and if the PayPal IPN comes in from an IP address that is not on that list, it will get blocked. But you raise an interesting question:

It just seems like the firewall would have to already be allowing any IP address to create an inbound connection to your site's other pages though, otherwise how could anyone access your site?

Not sure how this is gotten around...

Rich (my developer/hosting provider) has asked me to post a formal support request with Able which I am about to do.

Again, many thanks to all who have given thoughts/answers.

Unfortunately the issue is still alive & well... yesterday, out of 22 orders on my site that were paid with Paypal, NINE had IPN failures. I HAVE to get this resolved somehow...

Any further comments/questions/responses/opinions are much appreciated!!!