![]() |
Server DOWNTIME 9./10th August
Hi,
about 30 hours ago the server went into normal background manintanace, unfortunately some one also did start a cache cleanup on his "Gallery" DB [table g2_CacheMap]. Short info:since Debian 4.0 converts all DB's to innodb it thinks that are fne with that (in our case the larger one of two installed gallery instances...), the DB was unexpectetly innodb type. The "delete from" transaction was canceled by the backup tool after 4600 seconds. The backup failed due to mysqlhotcopy reporting (correctly) a improper shut down DB. This happend around 20h GMT. This was a deadlock case, since db was unusable and backups got stuck having no realy current copy to restore from. Unfortunately I could not get hold of a mysql crack (~23:30 local time) so it took me some time to get the DB back up and running. At aprox 0:30 GMT (thats 2h off local time) the db was accessible again and with in 15 minutes the webservices had been back online. Do not ask me what I did wrong, I assume my only brave deed was to have dumped the data before messing aroung with ibd and log files... I am not used to a DB rolling back a 65K rows transaction totaly silent (45 minutes), the percent display gets updated after the 100% has been finished, and having no to chance to stop it. At the moment the DB is running without innodb engine, all innodb databases have been converted to MYISAM (which at least has myisamchk and one file per table, not one file for everything but the log). The backupcycle was continued, a current backup exists now, BUT we are facing minor problems in some web-applications. So expect another two or three short outages for some/all websites in the next days. As the problem is not yet investigated in depth, we (Me and Nova) can not tell if and how long maintance will be. In case anyone knows how to reinitialize the innodb storage engine, please send me a PM ith good hints. And no, just deleting the log and db file and restarting mysql does not work at all. Thanks for staying tuned. Cheers Onno |
Re: Server DOWNTIME 9./10th August
I have no clue to help you, but good job with this!
I was wondering though, if all trouble is solved, if there was any chance to get SVN back up? |
Re: Server DOWNTIME 9./10th August
Hi,
after having slept and done some other non pc stuff, I was able to reinitialise innodb as descibed above: delete (move) old innodb files, restart mysql (only difference: this time I started in on the console), stop mysql, restart whole server: done Now we can convert back any required db to innodb format. According to the error logs only minor db updates for session and cache handling did got lost (none at BU sites). Phew, desaster prevented, hours lost, never again I swear, I will touch a DB engine I dont know in depth :-) And Mysql/MyISAM has its right for existence, the server became 35% faster after droping and disabling innodb. (it still is, the engine innodb is not yet used, 10MB db file so far), and ist restarts in seconds. Cheers Onno |
Re: Server DOWNTIME 9./10th August
Good job man, btw can I get my BU email back ? :)
|
Re: Server DOWNTIME 9./10th August
Yeah- it looks like a good job, but as always something is stopping working :D
Can You give me back my ftp access to www.bots-united.com with podbotmm user? [EDIT] Should I check if I can upload long files (7MB) into the filebase or You do that check before I'll start to complain again about that (the most frequently problem with the filebase after all server's updates)? :D [/EDIT] |
All times are GMT +2. The time now is 00:37. |
Powered by vBulletin® Version 3.8.2
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.