---- Pixeldrain: Hi, Your network abuse system incorrectly identified an attack from my servers. There is not actually an attack happening, there is simply a lot of traffic on my site today, there are a lot of users connected! Because of this all my IP addresses have been locked. I have over a hundred servers hosted by you! All my database servers, storage servers, even my payment gateway is down! Please unblock all my IP addresses immediately! I have millions of users who are currently not able to access their data, on one of the busiest days so far! This is really bad! ---- Hetzner: Dear Client We can't unblock your server until you have solved the cause of the problem. we have merged all the tickets together. This is not an flase postive. All your server caused an attack about 25.5 Gbps, 2.1 Mpps. why all your server are flooding this IP 2a01:4f9:2b:3201::2 with high pps? ---- Pixeldrain: That IP address is of my own server. I was installing a new caching servers and had to transfer a lot of files to fill up the cache. My Hetzner servers don't even talk to outside systems, it's all internal communication. ---- Hetzner: Dear Client We can't unblock your server until you have solved the cause of the problem. This looks not as normal traffic and ratio pps is very high. What kind of services do you use? This looks an attack and peak 25.5 Gbps, 2.1 Mpps. You may use a KVM console or the white list option to gain access to your server. For more details, please see: https://docs.hetzner.com/robot/dedicated-server/troubleshooting/guideline-in-case-of-server-locking/ ---- Pixeldrain: I am running a self-developed object storage system. The cluster stores about 10 PB of user data for 10000 paying customers (who are currently unable to access their files!). I have caching servers in front with a different hosting provider because Hetzner does not provide fast enough internet. Nearly all traffic is to and from the caching nodes as they serve the public API as well. ---- Hetzner: Dear CLient, If this server is your IP then you should tune the traffic since this is seen as an attack. The traffic pps is very high and this will trigger our abuse system. Could you please provide us an example file of your upload? Your server have only outgoing traffic and minimal zero incoming. This looks not as normal traffic and looks like mining. ---- Pixeldrain: Here is a directory of files I use to test the site: https://pixeldrain.com/d/umC3AsDy. Mind you it's pretty slow because you took down half of my database servers! There are all kinds of files hosted on this system. The traffic is high because it's just a high traffic site. My website is in the top #386 websites globally. There are hundreds of millions of monthly visitors. The site serves an average of 250 Gigabit per second. Most of that is served by the caching servers outside of Hetzner, the servers hosted by Hetzner are only accessed when a file has not been cached yet. This is the second time your abuse system has incorrectly blocked all my servers. This is a huge problem! I will have to start moving data to a different provider. Please fix this AS SOON AS POSSIBLE. ---- Hetzner: Dear Client, Unfortunately this lead to this abuse and we see this as an attack. Could you please limit that rate traffic of your servers? We want to avoid such attack on the future. ---- Pixeldrain: I'm trying to limit the traffic by improving my caching system. I'm really doing my best. But my site is growing very fast, that's why I need a hundred servers. The servers are not just there for fun, I'm actually using them. All servers should have 1 Gbps guaranteed bandwidth, that's what you're selling me and what I'm paying for. If the servers get blocked for using a quarter of that then you should tell me, then I'll pack my things and go to a different provider. I REALLY need the gigabit per server. I can't run my website without it. Selling unmetered gigabit and not providing it is false advertising! If your abuse system detects high usage as an attack then your abuse detection system is flawed. This does not work for large scale clients. You're about to lose a very big customer with a rapidly growing business. This relationship should be mutually beneficial, but you're making it very hard on me. Besides, it's been over an hour and my servers are still down. PLEASE UNBLOCK THE ADDRESSES ALREADY. ---- (Half an hour later) Please act quickly! My whole business has been offline for nearly two hours now. This is a huge reputation loss! I have no time to argue, I just need servers to be back online so I can move my files to a different place. ---- Hetzner: Dear Client, Please tune your server and if this happen again please provide an reply on abuse ticket. we have unlocked the servers now. ---- Pixeldrain: Thanks.. There were two problems with the abuse tickets. First is that the original abuse notification arrived at 1 AM, and when I started checking my mail for the day all the servers were already locked. And second is that your system sent me like 200 mails at once. How do I even reply to that? Responding to all of that would take more time than I have. The notifications also said that the case would be manually checked. I had hoped that the person checking the tickets would realize that the system had made a mistake and discard the tickets. I find it troubling that such impactful descisions are made by an automated system. This does not put me at ease at all. I have been a very content customer for over five years, I have paid nearly as much for my Hetzner servers as the house I live in. Most of that trust is gone now. I'm keeping my eyes open for alternatives. This has caused so many problems. My database has gotten corrupted because half of the servers suddenly disappeared. I need to repair that. I have a dozen support mails from angry customers to answer. ---- Hetzner: Dear Client, please reduce or control the traffic on your server to avoid such abuse.