Webserver SUMMER TIME was: Web-Server Spring-Break

[Onno Kreuzinger]

Hi,

as you may have noticed, there is something on the move with the physical server hosting. Atm the server is somewhat sluggish I must admit, but we makde a big backup session and had to disable DMA for safety reasons.

Okay most might guess, we did not do this for fun, but the new disks are already placed in the server (yes, there where something like 20 reboots in 48 hours), and ATM the sync to disk two is running.

LOAD is at 6+ and we put all resources into the syncing process, so the webserver (mysql in particular) server stays sluggish.

But the forecast is sunny and bright, the RAM had already been upgraded recently for server-one, server-two is already running two gameservers and will get a ram upgrade soon, so it can serve more.

The server move last autumn enabled us to add more services and have a unified interface. Although I must admit the password issues where more troublesome than expected

Besides the BU forum and websites we also provide IRC, Email, SPAM filtering and Mailinglists, as well as preinstallable webaplications (no one of BU uses them ATM, but if anyone wants a blog or osCommerce shop at BU :-)).

In beta stage are: the gameservers (managed by THE STORM) and development environments which enable developers to make the most out of it -> full root access with comfortable debian 4.0 software management (THE STOM is betatesting one).

Besides the BU concentric stuff we provide webhosting to 8 more web sites (family and friends) and DNS hosting for 30 domains overall, split to 24 master and slave and 6 slave only hostings (most are aliases for the hosted sites). Apart from server-one.de and server-two.de there are more 3 DNS slave servers, not involved with other services than DNS (server-three,...).


So much news after such a long time, sorry for that. I will try to keep you more updated this summer.

Cheers Onno

[Onno Kreuzinger]

Hi,

update so far: two tested disks are installed, we started installing a second instance for the webserver and will split the services apart into several seperate enviroments.
For the time being the server should run perfectly fine 75% of the day. During the CEST evening hours I still have to perform I/O sesitive operations. So the server might be sluggish during 18 - 24h CET/GMT.

Cheers Onno

[KWo]

I've been trying today (about 14:00 Polish time) to login to the filebase (with "admin login"). While logging I saw this strange message:

"paFileDB was unable to successfully run a MySQL query.
MySQL Returned this error: SELECT command denied to user 'bu_filebase'@'localhost' for table 'user' Error number: 1142
The query that caused this error was: SELECT userid,username,password,salt FROM bu_forums.user WHERE username = 'KWo' AND usergroupid >5"

Is the filebase working then or not?

[Onno Kreuzinger]

Hi,

I will check this now, it looks lookse a collateral damage, I'm, not sure. For the time beeing the big work is donw, yesterday night the the servers had been synced, so we are now redundant in terms of having a synced cold stand by system.

--OK

[Onno Kreuzinger]

Hi,

okay that's trivial, the msql security settings had been restored to stock defaults, as you can see I read a bu_forums table for the username/pass stuff with the user of the bu_filebase ressoruces. Sorry my fault, will be fixed ASAP.

--OK

[Onno Kreuzinger]

gnarf, why is phpmyadmin sooo painfull?
wyh is there no admin tool for mysql (one you can use/learn within 3 minutes to add users, acls and revoke them if needed)?
why is postgresql so slow?

anyhow fixed :-)

[KWo]

Thanks, Onno again (which time I had to say this?)

[Onno Kreuzinger]

np, this time I boke it
I fix it

Okay I must now confess:

The whole Webserver/Mail/DB and Plesk-Admin-CP have been moved into a virtual enviroment, phisicaly this is thill the same server, but the whole OS (debian 3.1), Webserver and Sites, Plesk 8.1.1 (that is qmail, spamassasin, admin tools) and the db (now mysql 4.1 instead of 4.0)
are running in a Open-Virtuozzo VE (see http://openvz.org).

Before I will try answer why, I have to mention the easy of installation of OpenVZ, Debian-VE, Plesk inside of that and the move of all plesk managed stuff from server-one.de to the new VE (www100.server-one.de).
All was done the debian way, all packages are "ment for production use", the system ways backed up about 9 times during the process, the total "full" downtime was about 4,5 hours. During the prepareations (after the HW issues had been resolved) the server had been sluggish for 4 days, the apache server was reduced to less allowed clients resulting in reduced availability to users.
During the reinstallation of debian/plesk 8.1.1 mysql version was changed to 4.1 w/o further notice, this lead to the only problems so far. Since mysql 4-1 preferes innodb two databases had lost some tables/files.

** in case you have websites missing db tables/files/files please contact me **
** I can fix it in an instant, but I can not see the problem my self **
** files are there, but expected data type is wrong **

Why:
One mayor reason for doing this was the fact that spamassassin ate roughly 1.2 gb of ram, making the other services suffer more and more.
We learned, that spamassasin will grow in mem usage over the months, because the patterns and learned rules get fragm,ented and bloated. This is usually cleaned up by SA updates, but a plesk controlled SA does get only updates, and no cleanup commands. This wont happen again :

On the other hand the HW issues reminded us, that having a real good hot standby would be cool, and since I work with the commercial Virtuozzo for 3,5 years, I know it is made for linux webservers (and plesk is by the same company and integrates very well) and it is the only virtuallisation software giving you countable gain in cpu power, w/o wasting ram or slowing down I/O.
I know it is "only a better jail/(chroot)", but that is exactly what it is, it's better that jails, more powerfull than virtual-pc emulators and as easy to maintain as a normal application.
4 commands to generate, preconfigure and start a virtual enviroment, 0 (zero/nada) steps for network-set up , if routable IP's are used.
Firewall and private IP's are as easy to setup as iptables is. With a (included) guide you need 4 rules to have a service running on a private IP inside a VE offered to a public port by DNAT

I also"tuned" mysql, apache (allows now 30% more users than previously) and spamassasin, the system now gives you a 40% reduced answer time for the most complex website (according to mysql-slow queries log).
SA processing time went down from avrg. 21 seconds to 1.2 seconds.

I case people are interessted, I can open a wiki page for the server stuff or simply explain mire here

TODO:
IRC stuff will be avail soon
Kode, can you please contact me, since I had problems reconstructing your need from the services running on server-two previously. (IRC was not moved, no ICR port should be mapped to the new system)

Problems:

contact me, ICQ is fine, mail too:
onno@kreuzinger.com, please have a meaningfull subject, because I use strict spam filtering on that address, its imposed by my boss
And when I walk though the positive matches, I look over "Hi!" subjects very fast

Cheers Onno

[The Storm]

It will be good one wiki page with all that stuff that you had setuped and good explaintion how we can setup our own.

[@$3.1415rin]

nice work and impressingly long progress reports