How to debug linux crashes - Page 2

Whistler · **Re: How to debug linux crashes -** 06-01-2006

afaik -ggdb3 is exactly same as -g3.

Make sure you are attaching the paramater in the "g++" command, not the "ld" one.

KWo · **Re: How to debug linux crashes -** 06-01-2006

Might it be useful such info?

FS: Tried to FS: fprintf NULL file pointer inside valid file handle!
FS: Tried to IsOk NULL file handle!
FS: Tried to IsOk NULL file pointer inside valid file handle!
FS: Tried to Flush NULL file handle!
FS: Tried to Flush NULL file pointer inside valid file handle!
FS: Tried to ReadLine NULL file handle!
FS: Tried to ReadLine NULL file pointer inside valid file handle!
%s%s---FS: open %s %p %i
---FS: close %s %p %i
Tried to close unknown file pointer %p
File %s was never closed
%s is not a packfile%s has %i files%s out of memory allocating directoryf for %i files{}()'{}()':???15CBaseFileSystem11IFileSystemI FileSystemBaseInterfacebad_allocCFileSystem_Stdio: :Mount()CFileSystem_Stdio::Mount()CFileSystem_Stdi o::Unmount()FS: Tried to SetVBuf NULL file handle!
StdioVFileSystem00917CFileSystem_Stdio%s/%sError:%s
%s.soCreateInterfacebad_allocFunction (%s) took too long, %.3f (max %.3f)
errorlogserrorlogs/errorlog%.5d.txt

L 01/05/2006 - 23:09:56: "[POD]Matt_Damon (100)<144><BOT><TERRORIST>" killed "[POD]Mel_Gibson (100)<153><BOT><CT>" with "mp5navy"
L 01/05/2006 - 23:10:03: "[P0D]Fragnatic (100)<141><BOT><TERRORIST

Seems like problem with logging...

[NvT]_KaszpiR_ · **Re: How to debug linux crashes -** 06-01-2006

Code:

[hlds@httpmce hlds_l]$ gdb hlds_i686 core.16895
GNU gdb Red Hat Linux (6.1post-1.20040607.43rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db lib                                                                rary "/lib/tls/libthread_db.so.1".

Core was generated by `./hlds_i686 -game cstrike -debug +sv_lan 1 +log on -insec                                                                ure -nomaster -nodns +'.
Program terminated with signal 11, Segmentation fault.
#0  0x019a5689 in ?? ()
(gdb) backtrace
#0  0x019a5689 in ?? ()
Cannot access memory at address 0xfeffea50
(gdb) Quit
(gdb) quit
[hlds@httpmce hlds_l]$ gdb hlds_i686 core.16988
GNU gdb Red Hat Linux (6.1post-1.20040607.43rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `./hlds_i686 -game cstrike -debug +sv_lan 1 +log on -insecure -nomaster -nodns +'.
Program terminated with signal 11, Segmentation fault.
#0  0x05072689 in ?? ()
(gdb) backtrace
#0  0x05072689 in ?? ()
Cannot access memory at address 0xfeffea50
(gdb) quit
[hlds@httpmce hlds_l]$ gdb hlds_i686 core.20150
GNU gdb Red Hat Linux (6.1post-1.20040607.43rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `./hlds_i686 -game cstrike -debug +sv_lan 1 +log on -insecure -nomaster -nodns +'.
Program terminated with signal 11, Segmentation fault.
#0  0x018e5689 in ?? ()
(gdb) backtrace
#0  0x018e5689 in ?? ()
Cannot access memory at address 0xfeffea50
(gdb) quit
[hlds@httpmce hlds_l]$ gdb hlds_i686 core.20821
GNU gdb Red Hat Linux (6.1post-1.20040607.43rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `./hlds_i686 -game cstrike -debug +sv_lan 1 +log on -insecure -nomaster -nodns +'.
Program terminated with signal 11, Segmentation fault.
#0  0x0431f689 in ?? ()
(gdb) backtrace
#0  0x0431f689 in ?? ()
Cannot access memory at address 0xfeffea50
(gdb) quit
[hlds@httpmce hlds_l]$ gdb hlds_i686 core.21525
GNU gdb Red Hat Linux (6.1post-1.20040607.43rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `./hlds_i686 -game cstrike -debug +sv_lan 1 +log on -insecure -nomaster -nodns +'.
Program terminated with signal 11, Segmentation fault.
#0  0x02656689 in ?? ()
(gdb) backtrace
#0  0x02656689 in ?? ()
Cannot access memory at address 0xfeffea50
(gdb) quit
[hlds@httpmce hlds_l]$ gdb hlds_i686 core.7338
GNU gdb Red Hat Linux (6.1post-1.20040607.43rh)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB.  Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/tls/libthread_db.so.1".

Core was generated by `./hlds_i686 -game cstrike -debug +sv_lan 1 +log on -insecure -nomaster -nodns +'.
Program terminated with signal 11, Segmentation fault.
#0  0x00dc6689 in ?? ()
(gdb) backtrace
#0  0x00dc6689 in ?? ()
Cannot access memory at address 0xfeffea50
(gdb) quit

now we got a pattern

in addition, demos from hltv, from the crashed maps, hope it will help http://217.153.187.245:81/linux/game...ke/bots/demos/

KWo · **Re: How to debug linux crashes -** 06-01-2006

Quote:

Originally Posted by Whistler

afaik -ggdb3 is exactly same as -g3.

Make sure you are attaching the paramater in the "g++" command, not the "ld" one.

Here is my makefile

PHP Code:


			
# Console command(s) for compiling on Cygwin/MinGW:
#   Example Cygwin: make HOST=cygwin
#     http://www.xake.dk/target/tools.html

MODNAME = podbot_mm
SRCFILES = bot.cpp bot_chat.cpp bot_client.cpp bot_combat.cpp bot_globals.cpp  bot_sounds.cpp compress.cpp dll.cpp engine.cpp util.cpp waypoint.cpp


RESFILE = podbot_mm.rc

EXTRA_LIBS_LINUX =
EXTRA_LIBS_WIN32 =
EXTRA_LIBDIRS_LINUX = -Lextra/lib_linux
EXTRA_LIBDIRS_WIN32 = -Lextra/lib_win32

#EXTRA_FLAGS = -Dstrcmpi=strcasecmp

SDKTOP=../hlsdk
SDKSRC=$(SDKTOP)/multiplayer
METADIR=../metamod-p/metamod
SRCDIR=.

OBJDIR_LINUX=obj.linux
OBJDIR_WIN32=obj.win32

ifeq "$(shell uname | cut -d _ -f1)" "CYGWIN"
    HOST=cygwin
endif

ifdef ComSpec
    ifeq "$(HOST)" "cygwin"
        OS=LINUX
        MCPU=-mcpu
        PATH_WIN=/usr/local/cross-tools/i386-mingw32msvc/bin
        CLEAN=clean_linux
    else
        OS=WIN32
        MCPU=-mtune
        PATH_WIN=/mingw/bin
        CLEAN=clean_win32
    endif
else
    ifdef COMSPEC
        ifeq "$(HOST)" "cygwin"
            OS=LINUX
            MCPU=-mcpu
            PATH_WIN=/usr/local/cross-tools/i386-mingw32msvc/bin
            CLEAN=clean_linux
        else
            OS=WIN32
            MCPU=-mtune
            PATH_WIN=/mingw/bin
            CLEAN=clean_msys_w32
        endif
    else
        OS=LINUX
        PATH_WIN=/usr/local/cross-tools/i386-mingw32msvc/bin
        CLEAN=clean_linux
    endif
endif

ifeq "$(HOST)" "cygwin"
    ifeq "$(ARCH)" "amd64"
        CC_LINUX=g++-linux-x86_64
    else
        CC_LINUX=g++-linux
    endif
else
    ifeq "$(ARCH)" "amd64"
        CC_LINUX=g++-x86_64
    else
        CC_LINUX=g++
    endif
endif

ifeq "$(OS)" "LINUX"
    INSTALL=install -m 644
    LD_WINDLL= PATH=$(PATH_WIN) $(PATH_WIN)/dllwrap
    DEFAULT=linux
#    CLEAN=clean_linux
else    # WIN32
    INSTALL=cp
    CC_WIN32=g++
    LD_WINDLL= dllwrap
    DEFAULT=win32
#    CLEAN=clean_win32
endif

RES_WIN32=windres

# Just call everything i386 unless its a 64bit binary!
ifeq "$(ARCH)" "amd64"
    LIBFILE_LINUX = $(MODNAME)_amd64.so
else
    LIBFILE_LINUX = $(MODNAME)_i386.so
endif

LIBFILE_WIN32 = $(MODNAME).dll
TARGET_LINUX = $(OBJDIR_LINUX)/$(LIBFILE_LINUX)
TARGET_WIN32 = $(OBJDIR_WIN32)/$(LIBFILE_WIN32)

FILES_ALL = *.cpp *.h [A-Z]* *.rc
ifeq "$(OS)" "LINUX"
#    ASRCFILES := $(shell ls -t $(SRCFILES))
else
#    ASRCFILES := $(shell dir /b)
endif

OBJ_LINUX := $(SRCFILES:%.cpp=$(OBJDIR_LINUX)/%.o)
OBJ_WIN32 := $(SRCFILES:%.cpp=$(OBJDIR_WIN32)/%.o)
RES_OBJ_WIN32 := $(RESFILE:%.rc=$(OBJDIR_WIN32)/%.o)

# optimisation level; overridden for certain problematic files
#CCO = -O2 -fomit-frame-pointer
CCO = -O2

# architecture tuning by arch type
ifeq "$(ARCH)" "amd64"
    CCOPT_ARCH =
else
    CCOPT_ARCH = -march=i586 $(MCPU)=i686
endif

BASEFLAGS = -Dstricmp=strcasecmp -Dstrcmpi=strcasecmp

#CCOPT = $(CCO) $(CCOPT_ARCH) -fno-exceptions -fno-rtti -DNDEBUG
CCOPT = $(CCO) $(CCOPT_ARCH) -fno-exceptions -fno-rtti

INCLUDEDIRS= -I$(SRCDIR) -I$(METADIR) -I$(SDKSRC)/engine -I$(SDKSRC)/common -I$(SDKSRC)/pm_shared -I$(SDKSRC)/dlls -I$(SDKSRC) $(EXTRA_INCLUDEDIRS)
#CFLAGS=-Wall -Wno-unknown-pragmas
CFLAGS=$(BASEFLAGS) -g3
#CFLAGS=$(BASEFLAGS)
ODEF = -DOPT_TYPE=\"optimized\"
CFLAGS:=$(CCOPT) $(CFLAGS) $(ODEF)

DO_CC_LINUX=$(CC_LINUX) $(CFLAGS) -fPIC $(INCLUDEDIRS) -DLINUX -o $@ -c $<
DO_CC_WIN32=$(CC_WIN32) $(CFLAGS) $(INCLUDEDIRS) -DWIN32 -o $@ -c $<
DO_RES_WIN32=$(RES_WIN32) -I$(SRCDIR) -I$(METADIR) -i $< -O coff -o $@
LINK_LINUX=$(CC_LINUX) $(CFLAGS) -shared -ldl -lm -static-libgcc $(OBJ_LINUX) $(EXTRA_LIBDIRS_LINUX) $(EXTRA_LIBS_LINUX) -s -o $@
LINK_WIN32=$(CC_WIN32) -mdll -Xlinker --add-stdcall-alias $(OBJ_WIN32) $(RES_OBJ_WIN32) $(EXTRA_LIBDIRS_WIN32) $(EXTRA_LIBS_WIN32) -s -o $@

$(OBJDIR_LINUX)/%.o: $(SRCDIR)/%.cpp
    $(DO_CC_LINUX)

$(OBJDIR_WIN32)/%.o: $(SRCDIR)/%.cpp
    $(DO_CC_WIN32)

# compiling windows resource file
$(OBJDIR_WIN32)/%.o: $(SRCDIR)/%.rc $(INFOFILES)
    $(DO_RES_WIN) 

# compiling windows resource file
$(OBJDIR_WIN32)/%.o: $(SRCDIR)/%.rc
    $(DO_RES_WIN32)

default: $(DEFAULT)

$(TARGET_LINUX): $(OBJDIR_LINUX) $(OBJ_LINUX)
    $(LINK_LINUX)

$(TARGET_WIN32): $(OBJDIR_WIN32) $(OBJ_WIN32) $(RES_OBJ_WIN32)
    $(LINK_WIN32)

$(OBJDIR_LINUX):
    mkdir $@

$(OBJDIR_WIN32):
    mkdir $@

win32: $(TARGET_WIN32)

linux: $(TARGET_LINUX)
    
clean: $(CLEAN)

clean_linux:
    test -n "$(OBJDIR_LINUX)"
    -rm -f $(OBJDIR_LINUX)/*.o

clean_win32:
    -if exist $(OBJDIR_WIN32)\*.o del /q $(OBJDIR_WIN32)\*.o
#    -cd..
    
clean_msys_w32:
    test -n "$(OBJDIR_WIN32)"
    -rm -f $(OBJDIR_WIN32)/*.o
    
clean_both: clean_linux clean_win32

Whistler · **Re: How to debug linux crashes -** 07-01-2006

well that makefile seems to be okay.

this part:

Program terminated with signal 11, Segmentation fault.
#0 0x019a5689 in ?? ()
(gdb) backtrace
#0 0x019a5689 in ?? ()

...seems that there are some invalid memory access in the code, such as writing out of the array as the call stack is completely messed up.

KWo · **Re: How to debug linux crashes -** 07-01-2006

How to find which function does it? How to compile the sources with cygwin (not - mingw may have different switches), to get functions name to be shown in debug.log or somewhere else to narrow the problem?
During compilation process I'm getting such warnings:

bot.cpp(318 ): warning: cast to pointer from integer of different size
bot.cpp(4492): warning: cast to pointer from integer of different size
bot.cpp(4553): warning: cast to pointer from integer of different size
bot.cpp(4616): warning: cast to pointer from integer of different size

and few the same ones , but from other
Might these issues in code cause this crash?

[NvT]_KaszpiR_ · **Re: How to debug linux crashes -** 08-01-2006

it can be everything, but it looks more like getting out of bounds of the array, like you access array[10] while array got only 10 elements (the last is array[9])

on this maps i thin this is something with waypoints or calcylating the walk path.

KWo · **Re: How to debug linux crashes -** 08-01-2006

But after I did everything as Whistler says, I should get more info about crassh- if it's really podbot mm causing the crash, or I'm wrong? Can anyone fix me?

KWo · **Re: How to debug linux crashes -** 11-01-2006

Well - the crash on de_chateau I can reproduce also on my windows PC (but it takes longer time - the server is working without the crash longer time than KaszpiR's linux one and needs to be played all time de_chateau only). My windows says clearly - podbot_mm.dll is causing the crash of hlds.exe and says some offset address. How to use some debugger (gdb or anything like it) on windows to let me knwo the exact bot function name causing the crash?
Anyone can help?

Or how to use that offset address to get some helpful info?

Whistler · **Re: How to debug linux crashes -** 11-01-2006

if you have mingw, try running the drmingw.exe file in the MingW/bin directory with the parameter -i. Then when it crashed, just click the "Debug" button. You still need to compile with -g3 to know the exact place of the crash.