Webserver SUMMER TIME was: Web-Server Spring-Break
 

Hi,

as you may have noticed, there is something on the move with the physical server hosting. Atm the server is somewhat sluggish I must admit, but we makde a big backup session and had to disable DMA for safety reasons.

Okay most might guess, we did not do this for fun, but the new disks are already placed in the server (yes, there where something like 20 reboots in 48 hours), and ATM the sync to disk two is running.

LOAD is at 6+ and we put all resources into the syncing process, so the webserver (mysql in particular) server stays sluggish.

But the forecast is sunny and bright, the RAM had already been upgraded recently for server-one, server-two is already running two gameservers and will get a ram upgrade soon, so it can serve more.

The server move last autumn enabled us to add more services and have a unified interface. Although I must admit the password issues where more troublesome than expected :)

Besides the BU forum and websites we also provide IRC, Email, SPAM filtering and Mailinglists, as well as preinstallable webaplications (no one of BU uses them ATM, but if anyone wants a blog or osCommerce shop at BU :-)).

In beta stage are: the gameservers (managed by THE STORM) and development environments which enable developers to make the most out of it -> full root access with comfortable debian 4.0 software management (THE STOM is betatesting one).

Besides the BU concentric stuff we provide webhosting to 8 more web sites (family and friends) and DNS hosting for 30 domains overall, split to 24 master and slave and 6 slave only hostings (most are aliases for the hosted sites). Apart from server-one.de and server-two.de there are more 3 DNS slave servers, not involved with other services than DNS (server-three,...).


So much news after such a long time, sorry for that. I will try to keep you more updated this summer.

Cheers Onno

Re: Web-Server Spring-Break
 

Hi,

update so far: two tested disks are installed, we started installing a second instance for the webserver and will split the services apart into several seperate enviroments.
For the time being the server should run perfectly fine 75% of the day. During the CEST evening hours I still have to perform I/O sesitive operations. So the server might be sluggish during 18 - 24h CET/GMT.

Cheers Onno

Re: Web-Server Spring-Break
 

I've been trying today (about 14:00 Polish time) to login to the filebase (with "admin login"). While logging I saw this strange message:

"paFileDB was unable to successfully run a MySQL query.
MySQL Returned this error: SELECT command denied to user 'bu_filebase'@'localhost' for table 'user' Error number: 1142
The query that caused this error was: SELECT userid,username,password,salt FROM bu_forums.user WHERE username = 'KWo' AND usergroupid >5"

Is the filebase working then or not?

Re: Web-Server Spring-Break
 

Hi,

I will check this now, it looks lookse a collateral damage, I'm, not sure. For the time beeing the big work is donw, yesterday night the the servers had been synced, so we are now redundant in terms of having a synced cold stand by system.

--OK

Re: Web-Server Spring-Break
 

Hi,

okay that's trivial, the msql security settings had been restored to stock defaults, as you can see I read a bu_forums table for the username/pass stuff with the user of the bu_filebase ressoruces. Sorry my fault, will be fixed ASAP.

--OK

Re: Web-Server Spring-Break
 

gnarf, why is phpmyadmin sooo painfull?
wyh is there no admin tool for mysql (one you can use/learn within 3 minutes to add users, acls and revoke them if needed)?
why is postgresql so slow?

anyhow fixed :-)

Re: Web-Server Spring-Break
 

Thanks, Onno again (which time I had to say this?) :)

Webserver SUMMER TIME
 

np, this time I boke it :)
I fix it ;)

Okay I must now confess:

The whole Webserver/Mail/DB and Plesk-Admin-CP have been moved into a virtual enviroment, phisicaly this is thill the same server, but the whole OS (debian 3.1), Webserver and Sites, Plesk 8.1.1 (that is qmail, spamassasin, admin tools) and the db (now mysql 4.1 instead of 4.0)
are running in a Open-Virtuozzo VE (see http://openvz.org).

Before I will try answer why, I have to mention the easy of installation of OpenVZ, Debian-VE, Plesk inside of that and the move of all plesk managed stuff from server-one.de to the new VE (www100.server-one.de).
All was done the debian way, all packages are "ment for production use", the system ways backed up about 9 times during the process, the total "full" downtime was about 4,5 hours. During the prepareations (after the HW issues had been resolved) the server had been sluggish for 4 days, the apache server was reduced to less allowed clients resulting in reduced availability to users.
During the reinstallation of debian/plesk 8.1.1 mysql version was changed to 4.1 w/o further notice, this lead to the only problems so far. Since mysql 4-1 preferes innodb two databases had lost some tables/files.

** in case you have websites missing db tables/files/files please contact me **
** I can fix it in an instant, but I can not see the problem my self **
** files are there, but expected data type is wrong **

Why:
One mayor reason for doing this was the fact that spamassassin ate roughly 1.2 gb of ram, making the other services suffer more and more.
We learned, that spamassasin will grow in mem usage over the months, because the patterns and learned rules get fragm,ented and bloated. This is usually cleaned up by SA updates, but a plesk controlled SA does get only updates, and no cleanup commands. This wont happen again :=)

On the other hand the HW issues reminded us, that having a real good hot standby would be cool, and since I work with the commercial Virtuozzo for 3,5 years, I know it is made for linux webservers (and plesk is by the same company and integrates very well) and it is the only virtuallisation software giving you countable gain in cpu power, w/o wasting ram or slowing down I/O.
I know it is "only a better jail/(chroot)", but that is exactly what it is, it's better that jails, more powerfull than virtual-pc emulators and as easy to maintain as a normal application.
4 commands to generate, preconfigure and start a virtual enviroment, 0 (zero/nada) steps for network-set up , if routable IP's are used.
Firewall and private IP's are as easy to setup as iptables is. With a (included) guide you need 4 rules to have a service running on a private IP inside a VE offered to a public port by DNAT :)

I also"tuned" mysql, apache (allows now 30% more users than previously) and spamassasin, the system now gives you a 40% reduced answer time for the most complex website (according to mysql-slow queries log).
SA processing time went down from avrg. 21 seconds to 1.2 seconds.

I case people are interessted, I can open a wiki page for the server stuff or simply explain mire here :)

TODO:
IRC stuff will be avail soon
Kode, can you please contact me, since I had problems reconstructing your need from the services running on server-two previously. (IRC was not moved, no ICR port should be mapped to the new system)

Problems:

contact me, ICQ is fine, mail too:
onno@kreuzinger.com, please have a meaningfull subject, because I use strict spam filtering on that address, its imposed by my boss :)
And when I walk though the positive matches, I look over "Hi!" subjects very fast ;)

Cheers Onno

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

It will be good one wiki page with all that stuff that you had setuped and good explaintion how we can setup our own. :)

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

nice work and impressingly long progress reports :)

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

Hi,

small update, no problems so far, apart the monitoring system being not yet 100% (a false alert, missing stats for some hours). The performace is fine, spamfiltering is not yet perfect, but that will improve soon.

Update2:
The stats had been wrong due to a OOM condition cause by multiple ssh password scans. I changed the ssh config accordingly (on all servers), the VE stuff (memory barrier and hard limit) prevented the running process from starving and apache + mysql (and mail ...) kept running.
We lost some viewers due to apache not being able to spawn more preforker processes for about 30 minutes, after all it was a good lesson :)

Update3:
Important for all BU Mailusers:
we now use greylisting (again), see wikipedia for detailed info on "what is that" ;)
We use a softer variant, that is sending 421 temp failure SMTP codes on the first connection of a IP, after some minutes the next connect will succeed. The database stores any known ip/target-address for weeks, so the initial delay is payed of later on.
We also use DNS RBL blacklists, they generate a proper 5xx failure code, since those listed hosts will be rejected as long as they are listed.
Please complain to me, if you are having problems!

Both antispam systems send human readable error messages pointing the affected user to the cause of the delay or permanent reject.

For SENDING email nothing changes, the system recognizes, that you authenticate to the mailserver, thus you need no greylisting and by god no RBL list. Assuming you are on a dial-up line, your home ip should be listed in the blocklist. You are no mailserver :)

You might notice also, that any smtp connection is throttled heavily during handshake, this is intentionally and cost about 15 seconds. This way mailwaves cant make spamassasin go wild :) But on the other hand websites testing your emailaddress have a delay and might even fail. (I only know one website rejecting a emailadress after a 421 error code, the error code is clear: you are fine (sender, recipent), but I cant accept mail from your: temp. local failure).

Cheers Onno

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

Hi,
just a tale from "the net", the DNS Realtime Black List works very strict, please be sensitive to people complaining to you, always have the error message forwarded and read it carefully.

The goot hing is, a email adress check from network-tools.som works like a charm: (http://network-tools.com/default.asp...ots-united.com)
While the same kind of tool from a "email marketing company" gets permanent rejected:
http://www.email-unlimited.com/tools/verify-email.aspx


And I have some stats from the webserver VE:
(see: stats-graph.png)
The system was rebootet almost 23h ago, so that is a good starting point, lets see what longtime stats show.

The apache stas are now processed using awstats, I think of publishing them somehow.

Cheers Onno

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

It looks like the problem with big files uploading (for example 3.6 MB) to the filebase appeared again. It didn't give me the link after uploading the file. The link should look so:
http://filebase.bots-united.com/uplo...ll_V3_B18c.zip
"The file podbot_full_V3_B18c.zip has been uploaded! The URL to the file is
Click Here to place this URL in the Download URL field."

Even if I place the link manually, the file cannot be downloaded from that location. You still miss some settings they should be remaining during forum updates :( It hapnes after about 60seconds of uploading. Maybe the file cannot be fully uploaded (because of some time limit setting)?

Please check the log from today's morning (about 8:00 - 8:30 my (Polish) timezone).

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

Ok,

sorry for beeing late on this one :)
I will fix it ASAP.

* Updated: *
please retry, it should work now for files around 30 - 40 mbytes, anyone having larger files, please contact me directly.
* -- *

for the rest here a status update:
Server runs fine, we have some greylisting (no spamassasin/content filtering )spam stats (see two attached SVG Images in ZIP, hope you can open them).
As well as performance stats from the whole server (see PNGs).

Also we now have web-analytics, but I have not figured out, how to make cool stats for every subdomain automaticaly, so that is a future thing.

The server gets more and more popular now, we are at ~40% peak-avg load for a 15 min timeframe, so still room for more :)

Cheers Onno

Re: Webserver SUMMER TIME was: Web-Server Spring-Break
 

Thank You very much. It works now. :)