What's with that server AGAIN?
 

Tonight the Bots United server has been unreachable from 5:30AM to 8:15AM or so. I was pinging it continuously to check when it would be back online. What's wrong with it again ??? >:( Is it the hosting provider or is it something else ?? I noticed that there was a new eMule service and also a new overnet service installed on our server, wtf ? questions :

1. It's not the first time the server goes down for a duration exceeding one hour. It already went down one night a few days ago. Whose fault ? Providers ? If so, what is their excuse ?
2. Is there a NEED for an eMule server and/or an Overnet server on the Bots United machine ?
3. Would these servers run, what is the advantage of putting up such services on the Bots United machine over using one of the thousands existing servers in the world for our peer to peer needs ?
3. Are eMule and Overnet servers secure ? In order to be efficient servers (worth the consideration) how many client hits do they need ? What bandwidth does this represent ?

So gentlemen, we want a professionnal quality of service or what ? >:( >:(

Re: What's with that server AGAIN?
 

Notice, the php page to reply don't work sometimes. It was the case, yesterday and the day before for example. I had to wait for 1 hour.

---Edit---
seems to be the case here... 5 min...

Re: What's with that server AGAIN?
 

Answer to 1. I asked the Hoster, 11 days ago there was a downtime at night of 3 hours due to a lightning strike that destroyed the whole UPS Powere system. Theye were restoring these systems last night when a fuse blew and ALL servers lost power.

Answer to 2,3,4
Can be deleted, is not running anyway. i used it to get some rare files that just download faster on the appropriate bandwidth. just delete the whole /overnet folder

Re: What's with that server AGAIN?
 

Well Nova, I am somehow suspecting that the guy lied to you. Why so, I don't have a clue. Like I told you, I was pinging the server continuously and I was also running traceroutes from time to time to ensure all was OK by my side. And I can tell you that at least the routers of greatnet.de were ALWAYS up and working, and the results of the last hop were alternatively: "ping timeout" for 1 or 2 minutes or so, and "no route to host" for 30 seconds. This is typical : the guys were shutting down the machine, unplugging wires, then plugging them again and checking whether it worked, then unplugging/shutting down again, then testing another thing, etc. They did this little comedy for more than one hour before it managed to work. I don't have a clue what they were up to, but it was clearly not just a bad fuse to replace.

Anyhow, what sort of manners are these ? If I were a hosting provider, I would AT LEAST send an email to my customers telling them that from time t1 to t2 I would bring their servers down for maintenance, so the customers could AT LEAST put a notification on their web servers to their users before the shutdown, politesse oblige, what do you think ?

Re: What's with that server AGAIN?
 

True.

Well, luckily this wasnt at daytime ... :)

Re: What's with that server AGAIN?
 

I agree. totally, but
after hosting dedicated servers 24/7 on the internet since 1994, with 4 different providers, I have found it doesn't happen. They just go down now and then. You can call and scream at them, you can take the price breaks written in your contract for the service going down but it will still happen, unfortunately.

The best thing to do is have your DNS setup with a register that has dynamic DNS (I use dotyou.com) and have a back up server somewhere that has one web page that says the server is down for maintenance check back later. This server could even be at someone home sitting on DSL since it is just serving a small web page, performance will still be ok even for a lot of hits. Then when you see the site is down kick in your dynamic DNS to point the domain to the back up server.

This process can be automated using the IE web control to create a custom browser to check the web site every 30 seconds to make sure it gets a response. If it doesn't it can browse to the domain name registrar service and set DNS to the back up ip address. It can them keep monitoring the main server to see when it comes back on line and set the IP back to the main server’s IP. This way everyone will either get the main web site or a page telling them to check back.

Dynamic DNS works very well. It is amazing how fast the world knows about the IP change at their local DNS server.

If you want a sample VC++ project that shows how to do this let me know I will send it. I have done a number of projects for companies that use custom web browsers. But you may want to take a look at doing this in Java.

Nova " Well, luckily this wasnt at daytime ..."
Daytime for who? :-)

Re: What's with that server AGAIN?
 

*BUMP*

what the F happened this morning again ??? BU has been down for HOURS !!! I could ping it but no services could respond at all !! Router problem I bet !! They're doing it on purpose or what ??? Now what's their excuse again ? Someone gotta tell them there's a limit and they're getting pretty damn close !!!
>:( >:( >:(

Re: What's with that server AGAIN?
 

well you can say they don't have a status page and do not post maintanace on a website or via email, but i have been working for an isp for 2 years and can tell you as long as humans do some work, it will fail from time to time.

depending on how expensive the hoster is the more backup's it has, i.e. more human maintainable. but even then the craziest thing you can imagine can happen:
someone switches _the_ master switch ..... blob .... all dark .... 1,5 mil domains offline .... 24 GBIT internet up/downlink dead .... 200 000 phones dead .....

and like on the NASA mars mission which missed mars some years ago (metric system ;) not usen in calcs.), is was a human error, someone wanted to work on high voltage and used the main switch, like he used to do it allways, no one told him that is was _the_ master switch.
in your case a fuse busted (if that is true), you can hardly send emails before a fuse explodes when you fix the power backup. and lightnings (like bullzosers) can hardly be totaly excluded from the danger list.

i could tell you examples of real life, but just think yourself when was the last local power outage, when was the last flooding somewhere you wantet to walk along, when was the last train/plain canceld you needed ?

s**t happens :D
* use backups *

Cheers MeMeD


[b.t.w. it happend 2002 in duesseldorf/germany at the level3 site, just for the suspicious. they have all you can dream of, double diesel genrators, 4x power backup, 3x aircondition, two physical differnt conncetion routes on 8+ telco carriers, handscanner access, security guards,..., and until 2002 they had a single big red main power switch for this, AFAIK they now have 2 with loud warning signals]

Re: What's with that server AGAIN?
 

According to hosting, this time it wasnt their fault. Look at this, its from /var/log/warn:

Jan 29 18:48:45 server-one kernel: VM: killing process httpd
Jan 29 18:48:45 server-one kernel: swap_free: Bad swap offset entry 80098500

Thats the last message in any log before the server crashed. PM, memed, is that of any use to you ?

Re: What's with that server AGAIN?
 

FYI the server was ONCE MORE unable to answer HTTP/FTP/SSH requests (and basically nothing but ICMP) this morning at 7AM, 8AM and 9AM. I did not check earlier (was sleeping) but I'd bet my ass it has been unreachable half the night.

*edit*

just seen your post Nova... YIKES, this looks like a REALLY nasty thing. The server's HDD may be corrupted... but I find it weird, though, because when the server was unreachable, it was just like all the userland code was frozen, only the kernel was still alive and able to reply to ICMP requests... I'm not sure a missing swap entry could crash the whole userland code, memed what do you think ?

Re: What's with that server AGAIN?
 

I know PM, it was unreachable since 7 last evening. I have been on their backs since then. No answer to anything but ICMP. But PM, its not the router. They did nothing but press the reset button now ...

Re: What's with that server AGAIN?
 

hmm looks like the VM tried to access the swapfile beyond it's acctual size.
i will check swap settings ASAP.
also i noticed the server beeing dead since 19:42 yesterday evening (planned to fragg on the server at 20:00) and i checked the CS server randomly until 22:00, also IRC was dead at 19:45.

Cheers memed

Re: What's with that server AGAIN?
 

Everything has been dead since 19:42 yesterday evening ... and i start to think its actually NOT the hosters fault (for once at least)

Re: What's with that server AGAIN?
 

yes, sorry, we've been crossposting. I've edited my last post. I agree with memed, this is a nasty problem. Either the HDD is starting to die, or something else, but I'm not sure what would cause the swap space to be reported incorrectly...

Re: What's with that server AGAIN?
 

when it's full, it will get a access denied, though im not shure it will yield that error...

Re: What's with that server AGAIN?
 

Hi,
/edit

stupid me, shoud not post such stuff here

edit/

ok the server should go better now, but i will debug things a little more,
any council member or mod intressted in more details, i'll be on IRC most of the time.

cheers memed

Re: What's with that server AGAIN?
 

Hi,
ok i had to restart the server, but now it should go better, yet anyone having trouble conntact me :)

Cheers memed

Re: What's with that server AGAIN?
 

what was the reason for the long downtime yesterday ? a problem with the server or a problem with the hosting company again ?

Re: What's with that server AGAIN?
 

it was the server spiraling down for no obvious reason, and the hoster for not reacting from friday night until this noon.
i wrote 5 emails (saturday early moning {but not very specific}, saturday noon, sunday noon, one early this morning and one this noon). 3 times they send an acknowlegment, one time they realy did restet the server...
for the server it is not an option to stay on that hardware/software for much longer. there has allready been one (shorter) outage last week and Nova is in search of a new hoster (problem: setup fee's).
I will _try_ to track the real cause, something can't malloc memory, there is one GB of RAM and supposed Stefan fixed the memory leak in relabot there should be no software eating it up, but i'm no exert for irc servers, so if anyone knows of dos attacks against the IRC or other servers we use, please tell me (or all of us) your suspicions, it is better to hunt shadows that waiting for the next dead time.

Memed

Re: What's with that server AGAIN?
 

there are no malloc problems that i am aware off. I do have fixed a few memory 'access' problems. Afaik the new rb (which does not want to compile, check the other forums) is 100% memory leakage free :D

Re: What's with that server AGAIN?
 

yeah i know, it was just the only non standard software in use which once had a known problem...
i'm not amused, to say it P.C. :-(