SOMETHING SO IMPORTANT ABOUT THE DOWNLOAD SERVER IT NEEDS A THREAD

[Kirby]

Ok, so apparently my webserver's address finally got into some lists it didn't need to be in, and it is now getting hammered with vulnerability scanners and the likes, quite often.

I'm taking measures against this because as it's running on a VDS, it's set up to run in a mode that isn't very heavy on RAM but it does spawn a lot of child processes, which makes it so some of the vulnerability scanners can effectively DoS the server by forcing Apache to spawn 300 new instances of itself every second or so. The solution is the ubiquitous request rate limiting that many websites use, and have the warning on them "Please disable any download accelerators or you might get blocked"

Two of you have already been blocked by Apache, someone using RoadRunner in Ohio and someone attending Christopher Newport University. I've removed both blocks, but you both need to turn off your stupid download accelerators, to download 4 maps you made over 30 connections each to the webserver, pointlessly.

The server sits on a gigabit connection and can use every last bit of it, it could packet flood you offline by itself. You're not getting files any faster by using an accelerator, it's uploading just as fast as you can download.

I'm not going to check Apache every hour and cross-reference the access logs with who's been blocked to see if you're a brbu'er or some automated scanner hitting the server, so this is going to serve as the only heads up to everyone.




tl;dr

Nope, read it.


-edit-

And just to make it clear, it is 100% impossible to trigger this block if you're downloading replays or maps in-game. Download accelerators, scanning Apache with an exploit/vulnerability scanner or mashing on F5 on a single page are the only possible ways to trigger it.

[Something Swain]

I use Roadrunner and live in Ohio, but I have no idea what a "download accelerator" is or how to turn it off.

[CaffeinePowered]

I use Roadrunner and live in Ohio, but I have no idea what a "download accelerator" is or how to turn it off.

Did you download from the direct links under the TFT thread?

What browser do you use? (And any plugins other than the default?)

[Something Swain]

[quote author=Something Swain link=topic=6298.msg241931#msg241931 date=1332885693]
I use Roadrunner and live in Ohio, but I have no idea what a "download accelerator" is or how to turn it off.

Did you download from the direct links under the TFT thread?

What browser do you use? (And any plugins other than the default?)
[/quote]

I try and download the links in the threads, but sometimes it downloads them in-game anyway.

As far as my browser goes I use Google Chrome. I don't think I'm running any plugins though.

[Kirby]

[quote author=Caffeine link=topic=6298.msg241932#msg241932 date=1332886681]
[quote author=Something Swain link=topic=6298.msg241931#msg241931 date=1332885693]
I use Roadrunner and live in Ohio, but I have no idea what a "download accelerator" is or how to turn it off.

Did you download from the direct links under the TFT thread?

What browser do you use? (And any plugins other than the default?)
[/quote]

I try and download the links in the threads, but sometimes it downloads them in-game anyway.

As far as my browser goes I use Google Chrome. I don't think I'm running any plugins though.
[/quote]



Then you're probably not putting them in the right place.

The person in Ohio in question is using Firefox and also uses a referrer spoofing plugin as well, so it's not you.

[at0m]

Crossreference IP with forum access logs, email the guy?

Sent from my Desire HD

[Luca Shoal]

[quote author=Something Swain link=topic=6298.msg241935#msg241935 date=1332890383]
[quote author=Caffeine link=topic=6298.msg241932#msg241932 date=1332886681]
[quote author=Something Swain link=topic=6298.msg241931#msg241931 date=1332885693]
I use Roadrunner and live in Ohio, but I have no idea what a "download accelerator" is or how to turn it off.

Did you download from the direct links under the TFT thread?

What browser do you use? (And any plugins other than the default?)
[/quote]

I try and download the links in the threads, but sometimes it downloads them in-game anyway.

As far as my browser goes I use Google Chrome. I don't think I'm running any plugins though.
[/quote]



Then you're probably not putting them in the right place.

The person in Ohio in question is using Firefox and also uses a referrer spoofing plugin as well, so it's not you.
[/quote]I use Firefox, and download directly like Swain does. But I have no idea what a referrer whatsit plugin is, nor what it would do. Unless this addon is the problem. That's my only guess.

[StolenToast]

someone attending Christopher Newport University.

:-X My bad.  I use Downthemall in firefox just to generically handle downloads because it lets me easily select where to download each file to (like /tf2/maps).  I set it to limit the connections to "http://216.52.148.214/" to 1 connection, so it will no longer trip the ban.

[Luca Shoal]

And I just did the same thing.

[Kirby]

Yeah that would be the plugin.

And just to clarify, the multitude of connections isn't really the problem, it's the time window that the plugin opens them all in that's the issue.

It requests smaller chunks of the same file in the same manner that torrents split up files into little chunks, but does so all at once so it looks like a scan or DoS attack against the server from the Apache security module's standpoint.

[StolenToast]

But having only one connection means it can only download the file as one chunk right?  No splitting, which is the problem.

[Kirby]

But having only one connection means it can only download the file as one chunk right?  No splitting, which is the problem.

If you have problems downloading a file in the same way as the rest of the world, in the manner that the web was designed to transfer a file from it's very inception... then you need to fix your computer.

[HeK]

I'm going to see how many weird IP ranges that I can get banned from Kirby's server...

[Duck, Duck, Goose]

tl;dr

Nope, read it.

tl;dr

[Luca Shoal]

[quote author=StolenToast link=topic=6298.msg241974#msg241974 date=1332974449]
But having only one connection means it can only download the file as one chunk right?  No splitting, which is the problem.

If you have problems downloading a file in the same way as the rest of the world, in the manner that the web was designed to transfer a file from it's very inception... then you need to fix your computer.
[/quote]I don't think that's what he meant. He was asking for clarification that "hey, if I set it to just one connection, it shouldn't trip the killswitch on me, right? Then I'm good for the future?"

[Kirby]

[quote author=«(''«) link=topic=6298.msg241998#msg241998 date=1332993702]
[quote author=StolenToast link=topic=6298.msg241974#msg241974 date=1332974449]
But having only one connection means it can only download the file as one chunk right?  No splitting, which is the problem.

If you have problems downloading a file in the same way as the rest of the world, in the manner that the web was designed to transfer a file from it's very inception... then you need to fix your computer.
[/quote]I don't think that's what he meant. He was asking for clarification that "hey, if I set it to just one connection, it shouldn't trip the killswitch on me, right? Then I'm good for the future?"
[/quote]


Herp, you're probably right.


The splitting isn't the problem, it's the (small) time frame in which the splits are requested.

Download accelerators ask for the file's size and split it up into even chunks and ask for these smaller chunks all at once, which spawns 6 - 12 requests to Apache for the same file. Apache then goes and accesses the file 6 - 12 times in different locations because it got requests starting at a specific amount of bytes into the file, which causes 6 - 12 threads to be spawned by Apache to serve one single file.

I have Apache set to map a file into RAM so that if/when it receives any further requests for that file for a little while, it won't need to reload the file from the hard drive and the request can be served as fast as the internet allows, but the byte specific requests by download accelerators bypass the memory mapping and load the file from disk per request.

As I mentioned in the OP, the server is a virtual server, so I share the PC with at most 3 other people. Processing power isn't the issue, the RAM I've got available and the hard drive latency if someone else on the machine is accessing it as well are the issues. Memory mapping in Apache bypasses the hard drive latency, but each successive byte specific request made by download accelerators makes Apache spawn a new thread and use more RAM, 6 - 12x more than was needed in the first place, depending on the number of chunks that are requested.

Download accelerators still perform byte specific requests if you set the connection limit to 1, but they won't all be spawned at the same time, more like over 10-15 seconds which reduces the impact to little to none.



The rules I set in Apache's security to stave off scanners are rather strict because I want it to be very fast to react to scans and/or attacks to reduce the spike load on the server, so a download accelerator left alone can trigger it too, hence the post.

[Luca Shoal]

It's cool broheim. Most of us aren't *that* technically savvy I figure, so it's good to give edumacation and all that.