Original vbisam

Author: n-isaka

AUTHORS

@@ -0,0 +1,5 @@
+* Trevor van Bremen <[email protected]> wrote
+  VBISAM.
+
+* Roger While <[email protected]> autoconf'd/libtoolized
+  it. Also major code restructure.

COPYING

@@ -0,0 +1,311 @@
+2007-11-27  Roger While <[email protected]>
+
+	* Tidy code, take out caching mevhanism.
+
+2007-02-26  Roger While <[email protected]>
+
+	* 2.0 release. First release of libtoolized version.
+	  We now have 3 extra directories : bin, tests, libvbisam.
+	  The VBISAM library proper is in libvbisam. The helper
+	  programs are in bin. Test programs are in tests.
+
+	  Release version is defined in configure.ac
+
+2004-06-07  Trevor van Bremen
+
+1.03
+====
+Fixes list
+----------
+vbDataIO.c isinternal.h Makefile:
+	Mikhail pointed out that VBISAM was illegally perform pointer arithmetic
+	on voids within vbDataIO.c.  I'd thought the gcc switch -Wall would trap
+	such things but I was wrong!  Added -Wpointer-arith to Makefile CFLAGS
+isinternal.h:
+	Mikhail added the correct values for VB_ENDIAN on HPUX and AIX.  I've
+	requested Mikhail to check whether his compilers have a CPU-dependant
+	manifest constant defined rather than an OS dependant one.
+isopen.c:
+	Guido pointed out that I was still using the OLD free list format in
+	the tCountRows() function.  This was screwing up the returned value
+	for the isindexinfo() call!
+vbIndexIO.c vbCheck.c:
+	Guido ALSO pointed out that C-ISAM is not 100% C-ISAM compatible in that
+	it occassionally 'forgets' to insert the 0x7f in the free-list node
+	signature. <Sigh>
+CvtTo64.c Makefile isHelper.c isbuild.c isinternal.h isopen.c vbCheck.c vbDataIO.c vbIndexIO.c vbKeysIO.c vbNodeMemIO.c vbVarLenIO.c vbisam.h:
+	Changed the dependency on _FILE_OFFSET_BITS == 64 to ISAMMODE == 1
+	This is to allow the 64-bit file I/O system calls to function even when
+	working on C-ISAM compatible file formats thus breaking the 31-bit (2GB)
+	barrier and extending it to 41-bit!
+vbCheck.c:
+	Fixes made such that all that's required for a successful rebuild of a
+	fixed length row file is the key descriptor nodes in the index file to
+	be 'valid'.
+	Also, set the index map entry for the dictionary node AFTER it may have
+	been squashed.
+istrans.c:
+	A quick check of strace() output showed that C-ISAM is performing some
+	rudimentary locking each time it writes to the transaction log file.
+	I implemented the same scheme for compatibility.
+vbLocking.c vbLowLevel.c:
+	Changes made to iVBLock to directly deal with being interrupted by a
+	signal during the fcntl call.
+vbLocking.c:
+	Made certain that the psKeyCurr[MAXSUBS] of a table are 'valid' within
+	iVBEnter().
+isdelete.c:
+	Tidyup in preparation for 2PC (iProcessDelete function added)
+	Handle correct error return for isdelrec+isdelcurr if row isn't on file
+	Don't try to reinsert a key if iVBKeyDelete fails, it only corrupts the
+	file further.  This should be done INSIDE iVBKeyDelete!
+vbCheck.c:
+	Fixup to correctly 'free' all that was allocated
+vbNodeMemIO.c:
+	If an index had LCOMPRESS but did not have TCOMPRESS, the high value
+	entry in a node would still contain the TCOMPRESS count! (Ooops)
+vbNodeMemIO.c vbKeysIO.c isinternal.h:
+	I noticed that C-ISAM outperformed VBISAM if the index didn't
+	make use of LCOMPRESS, TCOMPRESS or DCOMPRESS.  I surmized that in
+	that case, C-ISAM is simply editing the key within the node rather than
+	completely rebuilding the node from a linked list.  I've implemented
+	a corresponding algorithm (iQuickNodeSave) in this module to bring
+	VBISAM into the lead again (performance wise).  I'm guessing that many/
+	most savvy C-ISAM developers always use SOME form of key compression and
+	thus, this will PROBABLY not affect most people, but I could not BEAR
+	the humiliation of some 25 year old piece of mature code outperforming
+	my 6 month old creation.
+	For anyone even MILDLY interested, the biggest performance gains in
+	VBISAM can be made in the following functions:
+		iVBNodeSave
+		iVBTreeLoad
+		iVBKeyCompare
+		iVBKeyInsert
+		iVBKeyDelete
+	These currently consume the MAJORITY of CPU time and thus even a 'small'
+	improvement in these functions can reap potentially disparate (large)
+	benefit to overall throughput.
+vbNodeMemIO.c:
+	Fixes to 'force' the iIsTOF and iIsEOF flags within the tree
+isHelper.c:
+	a: Initialize tValue to 0 in ldquad and lValue to 0 in ldlong
+	b: Globally reverse the VB_ENDIAN content to match BYTE_ORDER
+isbuild.c:
+	isaddindex was not returning an error if the table was not open in
+	ISEXCLLOCK mode but still did not create the new index
+MVTest.c:
+	Extended to handle multiple indexes
+	Decided to make rowlen 256 bytes too!
+README.64bit:
+	New file to describe some background for 64 bit operation
+isrecover.c:
+	Include error checking for call to iVBRollMeBack
+iswrite.c:
+	This was writing an INCORRECT transaction on fixed length files until
+	isreclen got set.  Oops!
+vbBlockIO.c:
+	Standardized all file offsets as long long (64 bit)
+vbKeysIO.c:
+	Added in a couple of extra 'checking' algorithms for DEBUG use
+	Largely rewrote the entire iVBKeyDelete function
+vbMemIO.c:
+	Added a few assertions on VBTREE / VBKEY (de-)allocation
+isread.c:
+	Guido pointed out that I wasn't setting iResult after iVBDataLock and
+	was therefore returning JUNK
+vbNodeMemIO.c:
+	The new iQuickNodeSave function was not handling keys with ISDUPS set
+vbKeysIO.c:
+	iVBKeyLocateRow was not recursing through all the possible duplicates
+	in order to find an EXACT match on the row number!
+isinternal.h isopen.c isbuild.c vbLowLevel.c:
+	Moved from using access(2) to using stat(2).
+vbLowLevel.c isbuild.c isopen.c isinternal.h vbCheck.c:
+	Implement the 'sharing' of open file handles since the close() call
+	implicitly releases *ALL* locks on the handle that were 'owned' by the
+	process.  Changes made to insert a level of indirection on:
+		Open, Close, Lseek, Read, Write, Lock
+isinternal.h isHelper.c isopen.c vbLocking.c:
+	Moved the locks from psVBFile to sVBFile and implemented table-granular
+	locking.  Note that if CISAMLOCKS is defined when compiling VBISAM, the
+	locking strategy tries to more closely mirror that of C-ISAM.  However,
+	this introduces BUGS just like in C-ISAM!  I advise against using the
+	CISAMLOCKS and would PREFER you fixed the buggy code you have!
+isopen.c:
+	Minor fixup to delay setting psVBFile handles et al to -1
+	(Was screwing up the freeing of locks!)
+isinternal.h isHelper.c isdelete.c isread.c istrans.c vbLocking.c iswrite.c isrewrite.c:
+	Change locking strategy to eliminate the concept of a 'transactional'
+	lock.  ALL locks are transactional if VBISAM has called isbegin()!
+iswrite.c:
+	Only unlock a row if a transactional lock was applied *AND* the iswrite
+	is going to FAIL!
+vbKeysIO.c:
+	Changes to keep psVBFile [iHandle]->psKeyCurr [iKeyNumber] up to date on
+	iVBKeyDelete() calls.
+	Changes to the debug routines
+	Fix iVBLocateRow (was NFG if ISDUPS was set!)
+isrewrite.c:
+	Changes to make iRowUpdate work correctly for ISDUPS indexes
+vbLocking.c:
+	Changes to iVBEnter such that it will FAIL with ENOTRANS if isbegin was
+	not called PRIOR to the VBISAM function that is calling iVBEnter.
+isdelete.c:
+	Make writing the log transation the LAST function!
+	Also, use pcWriteBuffer for the deleted row so the log WORKS
+vbNodeMemIO.c:
+	TINY change to delay resetting iIsTOF in iNodeSplit since it was being
+	screwed up by iVBNodeSave
+isrewrite.c:
+	OK, let's face it. It was a MESS (Still is, but it worx lotz betta!)
+isopen.c:
+	isindexinfo was returning an INCORRECT di_nrecords!
+Known Issues
+------------
+Uncertain (but MOSTLY in vbKeysIO.c):
+	Two-phase commit (2PC) still needs to be done
+vbLowLevel.c:
+	I believe that incoming signals are still an issue on at least the
+	tVBRead () function.
+	(write() *SHOULD* be atomic already but I know that read() is not)
+vbCheck.c:
+	Still not able to process ISVARLEN files
+	Since vbCheck doesn't call iVBExit until it's finished all its work, it
+	becomes a REAL memory hog!  (See vbMemIO.c below)
+vbMemIO.c:
+	More of a 'suggestion' than an 'issue'.  It's probably wise to 'limit'
+	the amount of RAM any given VBISAM process can allocate.  An environment
+	variable would be the logical way to do this.  Also, it'd might be cool
+	to make it so that you could allow a maximum of X bytes per table with
+	an overall limit of Y bytes for the process as a whole.
+Documentation:
+	Heck, I need to write a whole freaking BOOK!!!
+	Reminder to self to include the environment variables in the docs
+================================================================================
+06Jun2004 1.02-beta (D-Day!)
+============================
+Fixes list
+----------
+LOTS of changes!!! (Certain that I've NOT covered them all here)
+
+iswrite() was creating a row in the data file BEFORE having created the index
+entries associated with it.  If an EDUPL error resulted, the data row was not
+purged.  Fixed by delaying writing the data row till AFTER the indexes were
+added.
+
+Management of the data row free lists was a mess.  Previously deleted data rows
+were *not* being reused.
+
+Interprocess disturbances causing 105 (EBADFILE) errors were common during
+iswrite () and iscommit () operations
+
+Using the same transaction log file for multiple processes was causing 105
+(EBADFILE) errors
+
+iswrite was inserting a new row into the *WRONG* node if it was exactly
+replacing a row that had previously been deleted. (Including the dup number)
+
+iVBEnter () and iVBExit () code fixed to better handle concurrency
+(Most specifically, the handling of the iIsDictLocked to determine whether the
+iVBExit () call should update the dictionary node transaction number)
+Many source modules modified to change handling of iIsDictLocked.
+
+Minor 'touch ups' made to the internal cache handling functions iVBBlockXXXX()
+Also, externalized them into their own unique module (vbBlockIO.c) with a switch
+possible in isinternal.h to allow selection of caching. (VB_CACHE)
+
+Bug fixed where performing an isread (ISEQUAL) on a table where the index was
+ISDUPS would NOT find any matching rows if the FIRST (duplicate number 0) row
+of that key value had been deleted.
+
+I re-tested the suite 'performance' again (Thank goodness for strace) to
+see what else could be optimized...
+In doing so, I found a HUGE saving in the TreeLoad function.
+Instead of searching the linked-list of keys in the VBTREE structure, I
+implemented a simple array of keys and devised a crude but effective list
+bi-section algorithm.  VBISAM now beats the pants off the competition!
+It's possible that I can optimize this even further and, based
+upon the DRAMATIC improvements it's offered thus far, I will probably do so.
+Also, I need to phase out the old use of the psKeyFirst and psKeyLast values
+held in the VBTREE structure.
+
+Fixed up the isrecover () code to truly recover things as it should!
+
+Wrote a simple program (vbRecover) that performs the actual isrecover given the
+name of the log file on the command line
+
+Wrote the beginnings of the VBISAM equivalent to bcheck as vbCheck
+
+Known Issues
+------------
+STILL have not bothered to write iscluster ()
+I simply CANNOT see enough relevance to do so
+
+STILL have not bothered to write isaudit ()
+Does *ANYONE* out there really USE this?
+
+Need to test the functionality of deleting a row within a transaction...
+Should it be possible for an unrelated process to create a row with the SAME
+unique index values before the transaction has been 'committed'?
+Interestingly enough, C-ISAM Vn 7.2 fails MISERABLY on this issue to the point
+where the DataFree list in the index file becomes corrupted.
+See the bug files on SF.net titled "When is a bug NOT a bug?"
+Note for self to fully implement ACID within VBISAM
+This is in the process of being addressed by way of the 2PC (Two Phase Commit)
+but is NOT fully implemented in this release!
+A strong CAVEAT with respect to the 2PC code is that the index file of a given
+table will be left in an inconsistent state for much longer periods than the
+competition.  Specifically, any key deletion will *NOT* take effect until the
+transaction is completed with an isrollback or iscommit call.  A system-crash
+occuring during a transaction thus leaves the index file a little 'screwed up'.
+However, the vbCheck and isrecover system of VBISAM automagically deals with
+this issue.  Furthermore, it is always good programming practice to make any
+transaction as short as possible (thereby minimizing the effect) and it's also
+good practice to at least vbCheck (if not completely recover) the effected
+tables subsequent to a system crash.
+
+Need to fully test the row release code in isHelper.c (isrelease, isrelrec and
+isrelcurr).  Specifically, I want to make sure that they are 'transaction-safe'
+
+Many function calls are still made without testing the return value.  Most of
+these are flagged in the source with a comment containing the word 'BUG'.
+
+Still need to 'complete' the isrecover code to handle the following transaction
+types:
+	VBL_BUILD, VBL_CREINDEX, VBL_DELINDEX, VBL_CLUSTER
+	VBL_FILEERASE, VBL_RENAME, VBL_SETUNIQUE, VBL_UNIQUEID
+
+Still haven't implemented the virtual file system.  If I bother at all with
+doing so, it will come *AFTER* the 2PC code.
+
+With all the changes made in this release, I've not yet ascertained whether the
+64-bit file I/O is fully functional.  I have no specific reason to suspect to
+the contrary, but YMMV.  Let me know if you encounter any issues.
+
+Summary
+-------
+All in all, quite a valuable release this time IMNSHO. Give it a few weeks time
+to allow things to 'settle' and I'll seriously consider moving the SF.net status
+of VBISAM from BETA to whatever comes next...
+As always, many thanks to those in the virtual OSS world who have assisted me
+to 'polish' VBISAM to its current state.  Feature requests are *ALWAYS* welcome
+================================================================================
+06Jan2004 1.01Beta
+==================
+Added COPYING.LIB and License line to each module header
+(Thanks to Johann von Nepomuk for spotting this huge oversight on my part!)
+
+Changed VBBLOCK based functions in vbLowLevel.c to delay writing back to the
+disk files while the index is 'locked' for use.  (Limits the number of system
+calls performed.  This *REALLY* has a HUGE impact if ISEXCLLOCK is used!
+
+Implemented variable length row processing
+HUGE warning...  This is not 100% compatible with the competition from IBM.
+Therefore, if you use the bcheck utility from IBM, it *MAY* report issues with
+the variable length nodes in the index file.  This is because the IBM product
+uses a *STUPID* algorithm for determining which 'group' a variable length node
+with some free space left should belong to.  (Namely, IBM appears to have
+cutoffs at 200-bytes, 400-bytes, 600-bytes and 800-bytes.  I have chosen to use
+a logarithmic approach instead with cutoffs at 8-bytes, 32-bytes, 128-bytes and
+512-bytes).  To put it MILDLY, the varlen code should be considered 'UNSTABLE'
+in this release.