Bitz and Bytes , saving with fWRITE

[stefanhendriks]

I got some code while back ago from cheesemonster about writing my huge leap of data using BITS. So eventually it was not 33MB, but at max only 2 or 3 MB.

I saw at the rb forums the bots cause trouble with big experience files, and i think the function being bad here is 'fwrite'. I probably try to write to much , but there is a problem... i cannot eliminate or reduce the data. I bet i need to chop it in 2 then, but here comes my problem, i don't know how to do that.

I have checked some code first of course, but currently i have no f*cking clue if it works okay when chopping the data in 2. I probably need to create another unassigned char, with half the size of my actual 'big' array (cVisTable), then when saving, i have to assign a first piece of data on the chunk, save it, and then do the same for the other half. In theory this sounds great, but id on't know how to code this yet.

FYI, i have posted here some functions:

//---------------------------------------------------------
//CODE: CHEESEMONSTER
bool cNodeMachine::GetVisibilityFromTo ( int iFrom, int iTo )	   
{ 
	// -- BY STEFAN --
	if (iVisChecked[iFrom] == 0)
		return NULL; // we have no clue
	// -- END --
	// was int
	// work out the position  
	long iPosition = (iFrom*MAX_NODES)+iTo; 
	
	long iByte = (int)(iPosition/8); 
	unsigned int iBit = iPosition%8; 
	
	if ( iByte < g_iMaxVisibilityByte ) 
	{				   
		// Get the Byte that this is in 
		unsigned char *ToReturn = (cVisTable+iByte); 
		// get the bit in the byte 
		return ( (*ToReturn & (1<<iBit)) > 0 ); 
	} 
	
	return false; 
} 
void cNodeMachine::SetVisibilityFromTo ( int iFrom, int iTo, bool bVisible ) 
{ 
	// -- STEFAN --
	iVisChecked[iFrom] = 1; // WE HAVE CHECKED THIS ONE
	// -- END --
		
	// was int
	long iPosition = (iFrom*MAX_NODES)+iTo; 
	
	long iByte = (int)(iPosition/8); 
	unsigned int iBit = iPosition%8; 
	
	if ( iByte < g_iMaxVisibilityByte ) 
	{ 
		unsigned char *ToChange = (cVisTable+iByte); 
		
		if ( bVisible ) 
			*ToChange |= (1<<iBit); 
		else 
			*ToChange &= ~(1<<iBit); 
	} 
} 
void cNodeMachine::ClearVisibilityTable ( void ) 
{ 
	if ( cVisTable ) 
		memset(cVisTable,0,g_iMaxVisibilityByte);
} 
void cNodeMachine::FreeVisibilityTable( void )
{
	if ( cVisTable )
		free(cVisTable);
}
//---------------------------------------------------------
 
// Save Experience
void cNodeMachine::experience_save()
{
  char dirname[256];
  char filename[256];
  int i;
  // Set Directory name
  strcpy(dirname, "data/cstrike/exp/");
  strcat(dirname, STRING(gpGlobals->mapname));
  strcat(dirname, ".rbx"); // nodes file
  // writes whole path into "filename", Linux compatible
  UTIL_BuildFileNameRB(dirname, filename);
  FILE *rbl;
  // Only save if lock type is < 1
  rbl = fopen(filename, "wb");
  if (rbl != NULL)
  {
	int iVersion = FILE_EXP_VER1;
	fwrite(&iVersion, sizeof(int), 1, rbl);
   
	for (i=0; i < MAX_NODES; i++)
 { 
  fwrite(&InfoNodes[i].fDanger[0], sizeof(Vector), 1, rbl);
		fwrite(&InfoNodes[i].fDanger[1], sizeof(Vector), 1, rbl);
		fwrite(&InfoNodes[i].fContact[0], sizeof(Vector), 1, rbl);
		fwrite(&InfoNodes[i].fContact[1], sizeof(Vector), 1, rbl);
	}
	if (iMaxUsedNodes > MAX_NODES) iMaxUsedNodes=MAX_NODES;
	
	// Here write down the MAX amounts of nodes used from vis table!
	unsigned long iSize = (iMaxUsedNodes*MAX_NODES)/8;
	fwrite(&iMaxUsedNodes, sizeof(int), 1, rbl);
	// Write down the vis-table
	fwrite(cVisTable, iSize, 1, rbl);
	// write down the checked vis
	fwrite(iVisChecked, sizeof(iVisChecked), 1, rbl);
	fclose(rbl);
  }
}
// Load Danger
void cNodeMachine::experience_load()
{
  char dirname[256];
  char filename[256];
  int i;
  // Set Directory name
  strcpy(dirname, "data/cstrike/exp/");
  strcat(dirname, STRING(gpGlobals->mapname));
  strcat(dirname, ".rbx"); // nodes file
  // writes whole path into "filename", Linux compatible
  UTIL_BuildFileNameRB(dirname, filename);
  FILE *rbl;  
  rbl = fopen(filename, "rb");
  if (rbl != NULL)
  {
	int iVersion=FILE_EXP_VER1;
	fread(&iVersion, sizeof(int), 1, rbl);
	if (iVersion == FILE_EXP_VER1)
	{
		for (i=0; i < MAX_NODES; i++)
		{ 
			fread(&InfoNodes[i].fDanger[0], sizeof(Vector), 1, rbl);
			fread(&InfoNodes[i].fDanger[1], sizeof(Vector), 1, rbl);
			fread(&InfoNodes[i].fContact[0], sizeof(Vector), 1, rbl);
			fread(&InfoNodes[i].fContact[1], sizeof(Vector), 1, rbl);
		}
		fread(&iMaxUsedNodes, sizeof(int), 1, rbl);
		// make sure we never exceed the limit
		if (iMaxUsedNodes > MAX_NODES) iMaxUsedNodes=MAX_NODES;
		unsigned long iSize = (iMaxUsedNodes*MAX_NODES)/8;
		// Read table from what we know
		fread(cVisTable, iSize, 1, rbl);
		fread(iVisChecked, sizeof(iVisChecked), 1, rbl);
	}
		fclose(rbl);
  }
}

Hey, Stefan, it looks like you are writing the file data the same way I would in VB; you have an array and you loop through the array and write individual elements of the array each time the loop iterates. So each write operation shouldn't be bigger than the array element that is being written. Try modifying your file write code so that every 500 nodes or so (pick your own number here) the write routine closes the file, re-opens it in append mode (so that new data written is added to the end of the file), and then continues writing. That should clear out the write buffer and keep it from being to big without requiring massive re-coding.

[stefanhendriks]

the nodes are not the problem. The cVisTable is the problem. This is one piece of memory and i do not itterate through anythying there. I just save it as one piece

[Pierre-Marie Baty]

well if the problem is writing large amounts of data in a row, why don't you split it ? for example, start at &cVisTable[0] and write 500 bytes. Then start at &cVisTable[500] and write another 500 bytes. Then start at &cVisTable[1000] and do this again and again until you've reached the end of cVisTable, that is, iSize.

Also please note that with fread() and fwrite() there is a difference between:
fread(cVisTable, iSize, 1, rbl);
and
fread(cVisTable, 1, iSize, rbl);
in the first case you read ONE block of iSize bytes in a row, in the second case you read iSize times one byte, calling the file I/O subroutines iSize times. You might want to read smaller blocks but in a greater amount...

Ok, I'm assuming this is the line you are referring to:

fwrite(cVisTable, iSize, 1, rbl);

I'm not sure what data type it is, but there is a few strategies I would try:

If it's an array of any kind, I would make a loop and write one element at a time, like you do with the node data.

If it's more like a string, I would create a loop and write pieces of the data (write bytes 1-1000, then write bytes 1001-2000, and so on until the end of the data is reached. In VB it would go something like this:

Sub WriteReallyBigString(BigString as String)
Dim StringChunk as String 'this holds the little piece of BigString

For X = 1 to Len(BigString) Step 1000 'Go through the length of BigString in 1000 byte increments
StringChunk = Mid$(BigString, X, 1000) 'Make StringChunk be 1000 bytes of BigString starting at position X

'Write StringChunk to file here

Next X 'End of for/next loop

If there's any way of accessing a particular byte or sequence of bytes from your vis table, but it doesn't have array elements, this would be the way to go. Hope this helps.

Hehe, I see Pierre-Marie and I suggested basically the same thing, but he beat me to the post...great minds think alike.

[Lazy]

Like I suggested in IRC - loop through your vis table and write it one byte at a time.

During the loop check if the iterator is evenly divisible by 512. If it is, call fflush on your stream.

That should force the data in the stream to be written to the file, but, since I have not tested this I'm not sure.

[Pierre-Marie Baty]

oye, fflush is a dangerous facility... somehow I wouldn't use it before being *certain* that all my data is already written

[Killaruna]

fwrite( cVisTable, 1, g_iMaxVisibilityByte, rbl )

... should work. You don't need fflush() because you close the file properly.

[Cheeseh]

Hey stefan.

I have successfully got my waypoint visibility files to read and write at different sizes.

For starters, when you save the cVisTable, you may need to add an extra byte because when you divide and cast to an int you may lose some floating point info, so you need to Ceiling (round to above integer, but if x.0 then its just x) that value to get the proper size.

When you load you need to estimate the size of the file and get the filesize, if they don't match then you have to re-build the vistable etc. :/ You can get the filesize by using fseek to end of file (SEEK_END ?) and use ftell to get the filesize (in bytes) also fseek back to SEEK_CUR i think.