One of our clients notified us that they were facing CAPTCHA challenges/images on a more and more frequent basis during regular web surfing.
An ever-increasing share of their outgoing emails was also being rejected as spam.
A quick look-up at http://cbl.abuseat.org showed that their public IP had been blacklisted for partaking in a bot net sending out spam mails. This lookup tool is very useful for troubleshooting and it updates every 10-15 minutes.
All user PC’s and servers were using Malware Bytes Premium and we initiated full scans on all computers. All of them came back clean. We also ensured that all clients and servers were fully patched.
The client’s Internet Service Provider (ISP) had provided a basic router with an elementary firewall function. Using the router’s firewall, we blocked all traffic on TCP ports 25, 465 and 587, but their public IP kept being blacklisted for sending out spam emails.
Our next action was to push out Windows firewall rules via Windows Group Policy (GPO). The Windows Firewall on all clients and servers was set to block all traffic on destination ports 25, 465 and 587. The GPO also enabled the Windows Firewall logging function. Alas, the blacklisting remained and the log files showed no SMTP traffic being blocked.
The next day, all staffers were asked to make sure they shut down their computers before going home. This had no effect on the blacklist status.
The following night, we took this strategy a step further and shut down all servers, printers and Wifi access points, to no avail – The CBL blacklist kept insisting that something on our client’s network was sending out spam.
We keep a supply of computer spare parts and network equipment in our own office and the following afternoon, we installed a proper firewall appliance behind the client’s ISP router. The firewall was set to allow only HTTP and HTTPS traffic from clients devices and DNS traffic from the AD Domain Controller. We set the firewall to trigger alerts whenever it encountered SMTP traffic, but not a single alert was fired and the blacklist status remained unchanged.
At this point, we were running out of options and as a final attempt at solving this mystery, we looked into the particulars of the ISP router – the brand was Huawei. A bit of googling showed multiple results for compromised Huawei routers partaking in some of the world’s largest bot nets.
We reported our findings to the ISP, who of course claimed their router was fine and denied that it had been compromised in any way. Eventually, we convinced the ISP to provide a new router of a different brand. Shortly thereafter, the client’s IP address disappeared from the CBL block list and things returned to normal.
Our conclusion is that the ISP’s Huawei router contained vulnerabilities that not only had allowed hackers to break into the router, but also to bypass the router’s firewall function.