server was down all the weekend again...
 

I bet the swap partition bug struck again.

We really must get this fixed at all costs, admins. I'll check the logs tonight. What if I put back the default kernel ? I don't see why mine would cause the problem, though, but better make sure... No time for that right now, but in the meanwhile if memed or Nova want to take a look, please guys, do !

And to all members... we are sorry :) The server crashed just at the beginning of the weekend and the hosting company was closed and nobody could reboot it until this morning :( We'll be investigating the issue ASAP. Hopefully this won't happen ever again.

Re: server was down all the weekend again...
 

That's really sad as I considered this hoster for some of our servers but now I won't :-)

Re: server was down all the weekend again...
 

A bit off topic perhaps, but I am wondering about the modified kernel. Where did it come from, and why is it better than the default?

Re: server was down all the weekend again...
 

it's a kernel I compiled myself to get rid of the useless hardware drivers and all the bloatware there is in a generic Linux kernel. Recompiling a custom kernel can save up to 80% kernel space, saves resources (IRQ and memory) and leads to a non neglectable speed improvement.

Re: server was down all the weekend again...
 

Hmm, but your kernel was around 15% bigger on disk and you included everything static, allthough this may be more "secure" it prevents all kinds of nice features (ppp_mppe for el cheapo vpn, modules for loopback, network modules in general, e.g. ip_tables relies on modules [no firewall!])
But this are all not the reasosn, the only bug appearing which is a real indicator for anything is the swap_free call returning ugly errors, this is usually one of the las messages, then it takes some time until it's dead.

Oh b.t.w. the Hoster was not closed, they just did refuse to reboot, because i do not have proper authorization to trigger a reboot (i even miss-spelled Nova's realname...) and the guy capable of deciding to react simply because it is logic to reboot a stuck server does not work on weekends.

To rule out harddisk failure in the swap area i now switched to a swap file and disabled the swap partition, the swapfile is called /lin386.swp ;)

@PMB there are CHANGELOG files on the server, just look in the root.

Cheers memed

Re: server was down all the weekend again...
 

Netfilter (iptables) doesn't rely on modules, you can build all neccessary functions statically into the kernel. Same goes for most other "modules" only a small handful of them is designed as "module-only", e.g. the new cryptographic modules of the 2.6.x kernels.

Re: server was down all the weekend again...
 

Afaik pptp with mppe is one of those module only, also beeing able to load and unload ip_table modules is a common practice for me. despite that, you words are contrary to PMB, why should all possible ip_tables modules be in the kernel (iiiieeeek), that makes it bigger and bloated. And im prety shure that some modules are (where) mutaly exclusive, resulting in the need for modules to get all funtions.
I also don't want to reinvent the wheel on a waggon allready in motion, e.g. my "forensic" connection tracker tool relies on being able to load/unload some of those modules. So i terms of security we ain't no border gate or secure login server and the kernel running now is atleast save enough for pmb and me ;-), while Nova can still recover what PMB and me failed.

cheers

p.s. when i dream of security:
If there where time and local access i would make it a GSX server running a cascaded openbsd/linux environment, preferably guarded by two aravox'es used in asynchonous mode (one for in and one for out). not to mention that i would like to have access to the onion router ring for cnnectivity to "my" border network which would be staffed with cisco's routers and pix'es running CSM on an isolated outband management station.

(apart from outband management, the number of aravox'es and the onion router ring access is did that once)

Re: server was down all the weekend again...
 

Btw (and off-topic): what is your profession memed?

Re: server was down all the weekend again...
 

care
solve
trust
is my company credo, I'm an independent IT Consultant, mostly for liunx servers [Webserver, App.-Servers, Storage-Nodes and Multimedia] and telecommunication needs [change telco provider, assist in negotiations with sub-contractors, enforce SLA's by documenting failure] and security conceps, testing and implementation [includes teaching, data-protection laws and software QA].

http://kreuzinger.biz or kreuzinger.com

but enough of advertising, i no good at that anyways.

Cheers memed
{Onno Kreuzinger}

Re: server was down all the weekend again...
 

I didn't want you to do it this way but I wanted to say that it's possible. But you're right saying that this will blow up the size of the kernel without careful planing which features are required and which are not. For a production system those features normaly don't change very frequently therefore you can go with a static kernel.
And there is much more than the kernel regarding security. E.g. stopping unneccessary services, no direct root logins etc. (hardening systems)

I'm not sure this will result in a secure system but it will result in a hard to manage system because of the complexity you described.

Btw: aravox is out of buisness since last year...
Btw2: what is a "onion router ring"?

Re: server was down all the weekend again...
 

Another kernel bug (german news site):

http://heise.de/newsticker/meldung/44755

Re: server was down all the weekend again...
 

i know, it throws a "Speicherzugriffsfehler" [core dump] on most kernels i tested. on my mandrake 9.0 it just dies w/o notice.

Re: server was down all the weekend again...
 

Debian already fixed it :-)

Re: server was down all the weekend again...
 

luckily my firewall is not affected ;-)

Re: server was down all the weekend again...
 

yes it is complex, but it was a carrier grade central internet gateway, it took about 1,5 man work years to make the concept. i was involved in the whole router and pix inband management stuff and resulting of this for all ip security but the aravox'es. all external business ip traffic (not endcustomer) went trough that single point of ip exchange.

AFAIK the aravox is still considered to be secure, it has a close-to-flawless design, but i didn't research for appliances of that scale lately.

Onion Routing is considered to be the most reliable anonymizer techniqe developed by the Navy, based on 1981 studies from David Chaum.
http://swpat.ffii.org/patents/effect.../index.en.html

Re: server was down all the weekend again...
 

Ok, THAT makes sense :-)

They were indeed secure and really fast but their configuration is a real pain in the ass. That's why they went out of business...

How does this technology fit into a security concept? I think it's counterproductiv as you as the security administrator want to know who is accessing your systems...

Re: server was down all the weekend again...
 

read the link, sender and recipient know each other, but all transit-network equipment only knows the addresse for its own path (A-B).
the trafic delegation(routing) works by encapsulating the payload for each hop into an extra packet. each step uses encryption which assures that the routers can onyl read the address for the next hop.

Jana-proxy also uses that techniqe, even cleaner implementet (why they are not afraid of any patent, which is nonsense btw because it is allready described in the 1981 studies).

cheers

Re: server was down all the weekend again...
 

I find this surprising. Why would a kernel have hardware drivers directly imbedded inside of it? That makes little sense to me. My understanding of the Unix methodology is that a layered approach is taken, where the kernel is a relatively tiny component of an overall OS. Everything is layered on top of the kernel. Something sounds wrong with having drivers embedded inside a kernel.

Ah, but as my sig says "theory != practice"

What is the BSD kernel like?

Re: server was down all the weekend again...
 

Ah, here comes the "monolithic vs modular" troll again... :D

Actually it does make sense to have a monolithic kernel tailored for your hardware. I am speaking about production machines. UNIX is supposed to be a stable OS, that you hardly ever need to reboot, which runs on a machine dedicated to it. In this sense, what are the advantages of modules (hardware drivers, crypto libraries, kernel-level binaries, whatever) over a monolithic kernel ? I don't see many, since the modules are loaded when the machine boots, and ideally, are never unloaded (since loading/unloading a kernel plugin is a critical task for the system, and most of the production systems can't afford the luxe of a system failure).

Furthermore, loading/unloading/handling modules has to be done by userland programs, executables on the hard disk, which ones are bound to user and group permissions and the filesystem's security strategy like any other userland program. There is an inner security flaw in this approach. If you haven't yet you'll soon notice that a good amount of Linux exploits concern kernel modules.

Another reason why I tend to prefer monolithic kernels, is that they typically take quite less space in memory compared to their modular equivalents (once all the modules are loaded, I mean.) And with this smaller memory footprint goes a (little) faster speed of execution. The BSD kernels are all monolithic. FreeBSD has the modularity feature, but it's not as widely used at all as in Linux, and many people (especially those who run and administrate business machines) recommend not to use it and stick with a custom kernel that is perfectly tailored to suit your hardware (although the OpenBSD guys, with their well-known focus on security, recommend to keep the default monolithic kernel that comes with the installation).

Re: server was down all the weekend again...
 

nah i wont rantle about that :-)
as PMB said, keeping the stock kernel is the safest option, for linux it's with modules, for others not.

oh and i reworked my bookmarks and found this:
http://bulk.fefe.de/
it shows some problems for the VM in OBSD.

Re: server was down all the weekend again...
 

Another small advantage of a static kernel (linux is a monolithic kernel with or without modules, ask Linus ;)) is it's resistance against most common root kits which load themself as a module...

Re: server was down all the weekend again...
 

so, please no compiling sprees and whatever nonstd stuff on the server this weekend !

Re: server was down all the weekend again...
 

ok, but that did not happen in the past. it crahed twice w/o any memory intensive task (first crash in jan., R.B. did not run) it did crash w. PMB's kernel and no memory buster, it did crash with PMB's kernel and a memory buster (realbot with memory leak) and it did crash with moderate mem usage (more apache instances, r.b. fixed) and my kernel (with modules, with some debugging).

but i won't touch it anyway's, just to make shure *g*

cheers memed

Re: server was down all the weekend again...
 

that's what I meant :)