.:: Bots United ::.  
filebase forums discord server github wiki web
cubebot epodbot fritzbot gravebot grogbot hpbbot ivpbot jkbotti joebot
meanmod podbotmm racc rcbot realbot sandbot shrikebot soulfathermaps yapb

Go Back   .:: Bots United ::. > YappA > The Agora
The Agora This is the place to go if you have suggestions, or if you want to participate in Council discussions. Everyone is welcome!

Reply
 
Thread Tools
Server DOWNTIME 9./10th August
Old
  (#1)
Onno Kreuzinger
aka: memed / Server Admin
 
Onno Kreuzinger's Avatar
 
Status: Offline
Posts: 705
Join Date: Jan 2004
Location: germany
Default Server DOWNTIME 9./10th August - 10-08-2007

Hi,

about 30 hours ago the server went into normal background manintanace, unfortunately some one also did start a cache cleanup on his "Gallery" DB [table g2_CacheMap].

Short info:since Debian 4.0 converts all DB's to innodb it thinks that are fne with that (in our case the larger one of two installed gallery instances...), the DB was unexpectetly innodb type. The "delete from" transaction was canceled by the backup tool after 4600 seconds. The backup failed due to mysqlhotcopy reporting (correctly) a improper shut down DB. This happend around 20h GMT.

This was a deadlock case, since db was unusable and backups got stuck having no realy current copy to restore from.

Unfortunately I could not get hold of a mysql crack (~23:30 local time)
so it took me some time to get the DB back up and running. At aprox 0:30 GMT (thats 2h off local time) the db was accessible again and with in 15 minutes the webservices had been back online. Do not ask me what I did wrong, I assume my only brave deed was to have dumped the data before messing aroung with ibd and log files... I am not used to a DB rolling back a 65K rows transaction totaly silent (45 minutes), the percent display gets updated after the 100% has been finished, and having no to chance to stop it.

At the moment the DB is running without innodb engine, all innodb databases have been converted to MYISAM (which at least has myisamchk and one file per table, not one file for everything but the log).

The backupcycle was continued, a current backup exists now, BUT we are facing minor problems in some web-applications. So expect another two or three short outages for some/all websites in the next days. As the problem is not yet investigated in depth, we (Me and Nova) can not tell if and how long maintance will be.

In case anyone knows how to reinitialize the innodb storage engine, please send me a PM ith good hints. And no, just deleting the log and db file and restarting mysql does not work at all.

Thanks for staying tuned.

Cheers Onno


sunny morning view from my balcony:

see our WIKI!
see our filebase!
  
Reply With Quote
Re: Server DOWNTIME 9./10th August
Old
  (#2)
stefanhendriks
RealBot Author
 
stefanhendriks's Avatar
 
Status: Offline
Posts: 3,088
Join Date: Nov 2003
Location: Netherlands
Default Re: Server DOWNTIME 9./10th August - 10-08-2007

I have no clue to help you, but good job with this!

I was wondering though, if all trouble is solved, if there was any chance to get SVN back up?


Author of RealBot, "Arrakis" and "Dune 2 - The Maker" | co-Founder of Bots-United | Fundynamic | Blog | E-Mail me
  
Reply With Quote
Re: Server DOWNTIME 9./10th August
Old
  (#3)
Onno Kreuzinger
aka: memed / Server Admin
 
Onno Kreuzinger's Avatar
 
Status: Offline
Posts: 705
Join Date: Jan 2004
Location: germany
Default Re: Server DOWNTIME 9./10th August - 10-08-2007

Hi,

after having slept and done some other non pc stuff, I was able to reinitialise innodb as descibed above:
delete (move) old innodb files, restart mysql (only difference: this time I started in on the console), stop mysql, restart whole server: done
Now we can convert back any required db to innodb format. According to the error logs only minor db updates for session and cache handling did got lost (none at BU sites).

Phew, desaster prevented, hours lost, never again I swear, I will touch a DB engine I dont know in depth :-)
And Mysql/MyISAM has its right for existence, the server became 35% faster after droping and disabling innodb. (it still is, the engine innodb is not yet used, 10MB db file so far), and ist restarts in seconds.


Cheers Onno


sunny morning view from my balcony:

see our WIKI!
see our filebase!
  
Reply With Quote
Re: Server DOWNTIME 9./10th August
Old
  (#4)
The Storm
Council Member / E[POD]bot developer
 
The Storm's Avatar
 
Status: Offline
Posts: 1,618
Join Date: Jul 2004
Location: Bulgaria
Default Re: Server DOWNTIME 9./10th August - 10-08-2007

Good job man, btw can I get my BU email back ?
  
Reply With Quote
Re: Server DOWNTIME 9./10th August
Old
  (#5)
KWo
Developer of PODBot mm
 
KWo's Avatar
 
Status: Offline
Posts: 3,425
Join Date: Apr 2004
Default Re: Server DOWNTIME 9./10th August - 12-08-2007

Yeah- it looks like a good job, but as always something is stopping working
Can You give me back my ftp access to www.bots-united.com with podbotmm user?

[EDIT]
Should I check if I can upload long files (7MB) into the filebase or You do that check before I'll start to complain again about that (the most frequently problem with the filebase after all server's updates)?
[/EDIT]
  
Reply With Quote
Reply


Currently Active Users Viewing This Thread: 1 (0 members and 1 guests)
 
Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is On

Forum Jump



Powered by vBulletin® Version 3.8.2
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
vBulletin Skin developed by: vBStyles.com