2021-09-02  Sergey Poznyakoff  <gray@gnu.org>

	Version 1.21 released

2021-08-27  Sergey Poznyakoff  <gray@gnu.org>

	Version 1.21

2021-08-18  Sergey Poznyakoff  <gray@gnu.org>

	Minor fix

	* src/input-argv.c ((instream_argv_read): Fix escaping of special
	characters.

2021-08-13  Sergey Poznyakoff  <gray@gnu.org>

	Revise gdbmtool error messages

	* src/gdbmshell.c: Revise error messages.  Use dberror where
	appropriate.
	* src/gdbmtool.h (dberror): New proto.
	* src/gram.y (dberror): New function.

	gdbmtool: always provide stdin stream

	* src/Makefile.am (libgdbmapp_a_SOURCES): Add input-std.c
	* src/gdbmshell.c (input_history_begin): Improve diagnostic message.
	* src/gdbmtool.c (instream_default_create): New define.
	(gdbmtool_init): Use instream_default_create.
	* src/gdbmtool.h [WITH_READLINE] (instream_readline_create_: New proto.
	(input_stream_name): New proto.
	* src/input-rl.c (instream_stdin_create): Rename to instream_readline_create.
	Fall back to instream_stdin_create if stdin is not connected to a tty.
	* src/lex.l (input_stream_name): New function.

	Re-implement input history handling to minimize linking requirements

	Applications using gbdmshell don't need to link with libreadline,
	unless they use instream_stdin_create with readline support.

	* src/gdbmshell.c (input_history_begin)
	(input_history_handler): New functions.
	(gdbmshell_run): Remove calls to input_init()/input_done().
	* src/gdbmtool.h (instream): New fields: in_history_size,
	in_history_get.
	(input_init,input_done): Remove.
	(instream_history_size,instream_history_get): New functions.
	(input_history_handler,input_history_begin): Remove.
	* src/input-rl.c: Incorporate history handling into the stream
	itself.
	* src/input-std.c (input_init,input_done): Remove placeholders.
	* src/lex.l (input_history_size,input_history_get): New functions.

2021-08-12  Sergey Poznyakoff  <gray@gnu.org>

	Update docs

	* NEWS: Document changes.
	* doc/gdbm.texi: Document new variables.

	Fix NULL dereference in gdbmshell.  Undefine "fd" upon closing the database.

	gdbmtool: setting database option affects the current database

	* src/gdbmshell.c (gdbmshell_setopt): New function.
	* src/gdbmtool.h (gdbmshell_setopt): New proto.
	* src/var.c: Provide set hooks for: cachesize, coalesce, and centfree.

	gdbmshell: get rid of remaining globals

	Use gdbmshell variables instead

	* src/gdbmshell.c (file_name, file_descr)
	(open_mode, open_format): Remove globals.
	(opendb,checkdb)
	(open_handler,import_handler)
	(status_handler): Use variables instead of globals.
	(command_tab): Mark the "open" parameter as optional.
	* src/gdbmtool.c (gdbmtool_init): Use variables instead of globals.
	* src/gdbmtool.h (file_name, file_descr)
	(open_mode, open_format): Remove globals.
	* src/lex.l: Use variables instead of globals.
	* src/var.c (variable): Rename hook to sethook.
	New field: typeconv.
	(vartab): New variables: "filename", "fd".
	(variable_get): Use typeconv if provided.
	* tests/gdbmtool/base.exp: Fix expected output.
	* tests/gdbmtool00.at: Likewise.
	* tests/gdbmtool01.at: Likewise.

	gdbmtool: bugfixes

	* src/gdbmshell.c (datum_free): New function.  Use it whenever a
	datum's dptr needs to be freed.
	(gdbmshell_run): Initialize last_cmd and last_args at the start
	and free last_args before returning.
	* src/util.c (vgetyn): Copy ap to a temporary va_list

2021-08-11  Sergey Poznyakoff  <gray@gnu.org>

	Fix duplicated mmap in gdbm_recover

	* src/recover.c (_gdbm_finish_transfer): Reuse memory mapping
	from the intermediate dbm structure.

	Minor fix

	* src/gdbmshell.c (opendb): Force GDBM_CLOERROR when calling gdbm_fd_open.

	Minor change in argv stream

	* src/input-argv.c (instream_argv_read): Don't treat = specially.

	Fix memory leaks in gdbm_fd_open

	* src/gdbmopen.c (gdbm_fd_open): Close the database before returning
	on error.

2021-08-10  Sergey Poznyakoff  <gray@gnu.org>

	Move gdbmtool shell functions to the library.

	* src/Makefile.am (libgdbmapp_a_SOURCES): Move gdbm shell
	support to the library.
	* src/gdbmtool.c: Move shell support to another file.
	* src/gdbmtool.h (file_descr): New extern.
	(gdbmshell, gdbmshell_run)
	(variables_init, variables_free): New functions.
	* src/gdbmtool.supp: New file.
	* src/var.c (VAR_IS_SET): Change definition.
	(variable): New member: init.
	(variable_set): Change meaning of VARF_INIT.
	(variables_free,variables_init): New protos.
	* src/gdbmtool.c: New file.

	gdbmtool: separate command parameters and environment

	gdbmtool: rewrite handler parameter functions; fix memory leaks

	* src/input-null.c: New file.
	* src/Makefile.am: Add input-null.c
	* src/mem.c (e2nrealloc): New function.
	* src/gdbmapp.h (e2nrealloc): New proto.
	* src/gdbmtool.c (quit_handler): Exit from parser instead of
	exiting from the program.
	(param): Remove global.
	(param_expand,param_push_arg,param_free): New functions for
	handling parameters.
	(param_free_argv): Take a single argument.  Clear all
	parameters up to argc.
	(run_command): Rewrite parameter handling using new functions.
	(main): Call input_init() after the input stream has been set
	up.
	Call yylex_destroy when finished.
	* src/gdbmtool.h (instream_null_create): New proto.
	(handler_param): Change type of argc to size_t.  New field
	argmax.
	(HANDLER_PARAM_INITIALIZER): New define.
	(yylex_destroy): New proto.
	* src/input-rl.c (history_file_name): New static.
	(get_history_file_name): Initialize history_file_name.
	(input_init): Initialize input history only in interactive mode.
	(input_done): Free input history only in interactive mode.

	Bugfixes

	* src/gdbmdefs.h (SAVE_ERRNO): Preserve both gdbm_errno and errno.
	* src/recover.c (_gdbm_finish_transfer): Transfer all cache fields
	(cache_mru was missing).

	Update gdbm.texi

2021-08-09  Sergey Poznyakoff  <gray@gnu.org>

	Update docs.  Rename some error codes.

	* doc/gdbm.texi: Update.
	* src/gdbm.h.in (GDBM_ILLEGAL_DATA): Rename to GDBM_MALFORMED_DATA.
	(GDBM_OPT_ILLEGAL): Rename to GDBM_OPT_BADVAL.
	All uses changed.
	Old constants retained for backward compatibility.

	Make parseopt reentrant.

	* src/parseopt.c (parseopt_first): Initialize option tables to NULL.

	Version 1.20.91.  Minor changes in translatable strings.

2021-08-08  Sergey Poznyakoff  <gray@gnu.org>

	Document new gdbmtool options

	gdbmtool: new option to open the database at a given file descriptor

	* src/gdbmtool.c (file_descr): New global.
	(closedb): Reset file_descr to -1.
	(opendb): Take second argument: file descriptor of the database file.
	Use gdbm_fd_open if it is > 0.
	New options: -d, -x
	* src/gdbmdefs.h: Fix comment.

2021-08-04  Sergey Poznyakoff  <gray@gnu.org>

	Fix the docs

	Fix representation of multibyte strings in gdbmtool output

	* datconv.c (f_stringz, f_string): Use multi-byte and wide
	string functions to decide whether a character is printable or
	not.

	Fix the representation of octal escapes in gdbmtool output

	* src/datconv.c (f_stringz, f_string): Use unsigned char *.

2021-08-03  Sergey Poznyakoff  <gray@gnu.org>

	gdbmtool: fix help output representation

	New macro: ARRAY_SIZE

	gdbmtool: control the format in which new databases are created

	* src/gdbmload.c (_gdbm_str2fmt): New function.
	* src/proto.h (_gdbm_str2fmt): New proto.
	* src/gdbmtool.c (open_format): New global variable.
	(opendb): Add open_format to gdbm_open flags
	* src/gdbmtool.h (open_format): New extern.
	* src/var.c: Handle the "format" variable.
	* doc/gdbm.texi: Document the changes.

	Preserve the database format across dump/restore

	* src/gdbmdump.c (_gdbm_dump_ascii): New parameter "format" holds the
	format of the original database.
	Set version number to 1.1
	* src/gdbmload.c (_gdbm_load_file): Handle the "format" parameter.

	Fix typos

2021-08-02  Sergey Poznyakoff  <gray@gnu.org>

	Various fixes

	* doc/gdbm.texi: Document manual crash recovery procedure.
	Document the gdbmtool "shell" command.
	Fix chapter headings (remove final point).
	Replace headings with sections.
	* src/gdbmopen.c (gdbm_header_avail): Remove const qualifier from
	1st argument
	* src/proto.h (gdbm_file_sync): New proto.
	* src/gdbmsync.c (gdbm_file_sync): Moved from proto.h.
	(gdbm_sync): Call _gdbm_end_update to be sure
	all changes are written to disk.
	* src/gdbmtool.c: Implement the shell command.
	(command_lookup): Fix name prefix comparison.
	* src/gram.y: New token T_SHELL.
	* src/lex.l: Sub-lexers for shell arguments.

	Fix gdbm_recover

	* src/recover.c (_gdbm_finish_transfer): Close snapshot descriptors,
	if any.
	Restore xheader, avail, and avail_size members.

	Fix numsync comparison

	* src/gdbmsync.c (gdbm_numsync_cmp): Properly handle unsigned overflow.
	* tests/gtload.c: New option -numsync.

2021-08-01  Sergey Poznyakoff  <gray@gnu.org>

	New gdbmtool command: snapshot

	* src/gdbmtool.c: New command: "snapshot"
	* doc/gdbm.texi: Document the snapshot command (gdbmtool).
	* doc/gdbmtool.1: Likewise.
	* src/gdbmsync.c: Fix a comment.

	Fix a typo.

	* src/gdbmsync.c (stat_snapshot): Use EACCES instead of EACCESS.

2021-07-31  Sergey Poznyakoff  <gray@gnu.org>

	Changes in the docs.  Minor improvements in gdbm_latest_snapshot.

	* doc/Makefile.am: Export htmlxref.cnf
	* doc/htmlxref.cnf: New file.
	* doc/gdbm.texi: Improve crash-tolerance descriptions.
	* src/gdbmsync.c (stat_snapshot): Set errno = EACCES if access bits
	of the snapshot stat are wrong.
	(gdbm_latest_snapshot): Don't touch ret if returning
	GDBM_SNAPSHOT_SUSPICIOUS.

	Update the documentation

2021-07-30  Sergey Poznyakoff  <gray@gnu.org>

	Minor fix

	* src/gdbmsync.c (gdbm_numsync_cmp): Fix signedness of na, nb

	Minor change

	* src/gdbmdefs.h (GDBM_HEADER_AVAIL_SIZE): Removed.
	(GDBM_HEADER_AVAIL_OFFSET): New macro.
	* src/avail.c: Use dbf->avail_size and GDBM_HEADER_AVAIL_OFFSET.
	* src/gdbmopen.c: Likewise.

	Avoid eventual alignment problems

	* src/gdbm.magic: Handle extended GDBM format.
	* src/gdbmdefs.h (gdbm_file_standard_header)
	(gdbm_file_extended_header): New data types.
	* src/gdbmopen.c (gdbm_header_avail): Use casts to
	gdbm_file_*_header to obtain data offsets.
	* src/gdbmtool.c (print_header_handler): Improve formatting.

2021-07-29  Sergey Poznyakoff  <gray@gnu.org>

	Minor fix

	* src/gdbmsync.c (gdbm_latest_snapshot): Restore missing break.

	Various bugfixes.

	Improve _gdbm_snapshot

	* src/gdbmsync.c (_gdbm_snapshot): Before returning, chmod previous
	snapshot write-only.  Change proposed by Terence Kelly on
	July 23, 2021.

	Mark GDBM_MAGIC constants as unsigned

	Implement snapshot selection by numsync.

	* src/gdbm.h.in (GDBM_SNAPSHOT_SUSPICIOUS): New return code.
	* src/gdbmsync.c (gdbm_latest_snapshot): Take into account numsync
	when choosing between the two snapshots.
	(gdbm_sync): Increase numsync.

	Initialize dptr to non-null storage even if size is 0.

	Test conversion to extended format.

	* src/gdbmdefs.h (gdbm_ext_header): Add new field: version.
	Pad to 8 integers.
	* tests/gtconv.c: New auxiliary program.
	* tests/conv.at: New test.
	* tests/Makefile.am: Add new tests.
	* tests/testsuite.at: Likewise.

	Simplify gdbm_file_header

	* src/gdbm.h.in (gdbm_convert): New proto.
	* src/gdbmdefs.h: Define GDBM_MAGIC and GDBM_NUMSYNC_MAGIC here.
	(gdbm_file_header): Remove the union.
	* src/gdbmopen.c (gdbm_header_avail): Return pointer to
	gdbm_ext_header in 4th argument.
	(validate_header_std,validate_header_numsync): Remove avail size
	verification.  It will be checked later in gdbm_fd_open.
	(gdbm_fd_open): Check avail table size.
	(gdbm_convert): New function.
	* src/avail.c (gdbm_avail_traverse): Change the avail table offset
	calculation.
	* src/gdbmtool.c: New commands: upgrade, downgrade and sync.
	Fix output of the "header" command.

2021-07-28  Sergey Poznyakoff  <gray@gnu.org>

	Introduce extended header.

	* src/gdbm.h.in (GDBM_NUMSYNC): New flag for gdbm_open
	* src/gdbmdefs.h (gdbm_ext_header): New struct.
	(gdbm_file_header): New member of the union v: gdbm_ext_header.
	* src/gdbmopen.c
	(gdbm_header_avail): New function.
	(validate_header): Use dbf->avail.
	(gdbm_fd_open): Read both database types.  Create GDBM_NUMSYNC_MAGIC
	database if given the GDBM_MAGIC flag.

	Prepare structs gdbm_file_header and gdbm_file_info for changes.

	* src/gdbmconst.h (GDBM_NUMSYNC_MAGIC32)
	(GDBM_NUMSYNC_MAGIC64)
	(GDBM_NUMSYNC_MAGIC32_SWAP)
	(GDBM_NUMSYNC_MAGIC64_SWAP): New constants.
	* src/gdbmdefs.h (gdbm_file_header): Move avail to a union in
	anticipation of further changes.
	(gdbm_file_info) <avail, avail_size>: New members.
	(GDBM_HEADER_AVAIL_SIZE): Redefine.
	* src/gdbmopen.c (GDBM_NUMSYNC_MAGIC): New define.
	(validate_header_std)
	(validate_header_numsync): New functions.
	(validate_header): Use one of these depending on the magic number.
	(gdbm_fd_open): Initialize new members of gdbm_file_info.

	* src/avail.c: Use the GDBM_FILE avail pointer.
	* src/falloc.c: Likewise.
	* src/gdbmtool.c: Likewise.

2021-07-22  Sergey Poznyakoff  <gray@gnu.org>

	Fix gdbm_avail_traverse()

	* src/avail.c (gdbm_avail_traverse): Apply callback to the header
	avail block as well.
	* src/gdbmtool.c (avail_list_print): Don't print header block separately.

2021-07-17  Sergey Poznyakoff  <gray@gnu.org>

	Improve fsync_to_root.  Other minor fixes.

	* src/gdbm.h.in (gdbm_latest_status): Rename to
	gdbm_latest_snapshot_status.
	* src/gdbmsync.c (fsync_to_root): Start synching from the snapshot
	itself.  Take care to open it write-only.
	(gdbm_latest_snapshot): Reorganize to improve readability.
	[!GDBM_FAILURE_ATOMIC] gdbm_latest_snapshot: Fix signature.

	Bugfix

	* src/gdbmsync.c (stat_snapshot): Fix typo (missing parentheses).

2021-07-17  gray@gnu.org  <Sergey Poznyakoff>

	Bugfixes in crash tolerance code.

	* doc/gdbm.texi: Document GDBM_SNAPSHOT_BAD
	* src/gdbm.h.in (GDBM_SNAPSHOT_BAD): New constant.
	* src/gdbmsync.c (fsync_to_root): Rewrite.  Use pointer
	to traverse the pathname in the reverse direction.  Fsync
	the root directory as well.
	(gdbm_failure_atomic): Create both snapshots in user write-only mode.
	(stat_snapshot): Fail if the file is neither readable nor writable.
	(gdbm_latest_snapshot): Fail if neither file is readable.
	Fix selection of the snapshot by mtime: prefer the latest one.
	[!GDBM_FAILURE_ATOMIC]: Provide the placeholder for
	gdbm_latest_snapshot.

2021-07-17  Sergey Poznyakoff  <gray@gnu.org>

	Document new error codes

	Document crash tolerance API

2021-07-17  gray@gnu.org  <Sergey Poznyakoff>

	Crash-tolerance based on the patch from Terence Kelly

2021-07-07  Sergey Poznyakoff  <gray@gnu.org>

	Fix https://puszcza.gnu.org.ua/bugs/?515

	* src/gdbm.h.in: Include sys/types.h

2021-06-17  Sergey Poznyakoff  <gray@gnu.org>

	Version 1.20

2021-06-14  Sergey Poznyakoff  <gray@gnu.org>

	Enable pre-reading the memory mapped regions on request.

	The commit 4fb2326a4a introduced pre-reading of memory mapped
	regions.  While speeding up searches, it has a negative impact
	on write operatons, since every remapping effectively re-reads
	the entire database.  See https://github.com/Perl/perl5/issues/18884
	for details.

	* NEWS: Document changes.
	* doc/gdbm.texi: Document the GDBM_PREREAD flag.
	* src/gdbm.h.in (GDBM_PREREAD): New flag.
	* src/gdbmdefs.h (gdbm_file_info): New member: mmap_preread.
	* src/gdbmopen.c (gdbm_fd_open): Set mmap_preread if requested.
	* src/gdbmsetopt.c (setopt_gdbm_getflags): Report GDBM_PREREAD
	flag, if dbf->mmap_preread is set.
	* src/mmap.c (_gdbm_internal_remap): Use pre-fault reading only
	if dbf->mmap_preread is set.

2021-04-23  Sergey Poznyakoff  <gray@gnu.org>

	Fix possible integer overflows and cases of undefined behavior.

	* src/avail.c (avail_comp): Prevent integer overflow.
	(gdbm_avail_table_valid_p): Likewise.
	* src/gdbmdefs.h (off_t_sum_ok): Change return type.
	Return false if any argument is negative.
	* src/gdbmopen.c (compute_directory_size): Prevent integer overflow.
	* src/hash.c (_gdbm_hash): Treat dptr elements as unsigned
	integers.

2021-03-22  Sergey Poznyakoff  <gray@gnu.org>

	Fix minor memory leaks in gdbmtool

	* src/datconv.c (dsegm_list_free): Free name if necessary.
	* src/gram.y (defid): Free $1 when no longer needed.

2021-03-21  Sergey Poznyakoff  <gray@gnu.org>

	Merge branch 'newcache': Bucket cache rewritten from scratch.

	Enable automatic cache size by default.

	This provides for the optimal performance.

	Additional GDBM_XVERIFY flag can be used when opening the database
	to request extended database verification (entire avail stack will be
	checked).
	* src/bucket.c (cache_lookup): Handle cache_size == GDBM_CACHE_AUTO.
	(_gdbm_cache_init): Likewise.
	* src/gdbm.h.in (GDBM_XVERIFY): New flag to gdbm_open.
	(GDBM_CACHE_AUTO): New constant.
	Define error constants within enum.
	Fix definition of GDBM_BAD_HASH_ENTRY.
	* src/gdbmclose.c: Fix misleading comment.
	* src/gdbmconst.h: Define DEFAULT_CACHESIZE to be GDBM_CACHE_AUTO.
	* src/gdbmopen.c (gdbm_fd_open): Call gdbm_avail_verify if
	the GDBM_XVERIFY flag is set.

	* doc/gdbm.texi: Update.
	* doc/gdbm.3: Update.

2021-03-20  Sergey Poznyakoff  <gray@gnu.org>

	More optimizations in cache tree

	* src/bucket.c (gdbm_dir_entry_valid_p): Inline
	(cache_lookup): Keep track of cache hits.
	(_gdbm_get_bucket): Check the cache_entry first.
	(gdbm_get_cache_stats): Change signature. Return the per-db
	number of cache hits as well.
	* src/cachetree.c: Inline static functions.
	(_gdbm_cache_tree_lookup): Avoid extra level of indirection
	(nodeptr) at the expense of one additional comparison.
	* src/gdbm.h.in (gdbm_get_cache_stats): Change signature.
	* src/gdbmdefs.h (gdbm_file_info) <cache_hits>: New member.

2021-03-17  Sergey Poznyakoff  <gray@gnu.org>

	Fix memory leak in gdbmtool "next" command.

	* src/gdbmtool.c (nextkey_handler): Always free the previous key
	buffer.

	Fix memory leak in gdbmtool "next" command.

	* src/gdbmtool.c (nextkey_handler): Always free the previous key
	buffer.

	New functions for traversing the available space stack

	* src/Makefile.am: Add avail.c
	* src/avail.c: New file.
	* src/gdbm.h.in (gdbm_avail_verify): New proto.
	* src/gdbmdefs.h (GDBM_HEADER_AVAIL_SIZE): New macro.
	* src/gdbmopen.c (gdbm_avail_table_valid_p)
	(gdbm_avail_block_validate)
	(gdbm_bucket_avail_table_validate): Move to avail.c
	* src/gdbmtool.c (_gdbm_avail_list_size)
	(_gdbm_print_avail_list): Rewrite using gdbm_avail_traverse.
	* src/proto.h (gdbm_avail_traverse): New proto.
	* src/systems.h: Include stddef.h.

	New functions for traversing the available space stack

	* src/Makefile.am: Add avail.c
	* src/avail.c: New file.
	* src/gdbm.h.in (gdbm_avail_verify): New proto.
	* src/gdbmdefs.h (GDBM_HEADER_AVAIL_SIZE): New macro.
	* src/gdbmopen.c (gdbm_avail_table_valid_p)
	(gdbm_avail_block_validate)
	(gdbm_bucket_avail_table_validate): Move to avail.c
	* src/gdbmtool.c (_gdbm_avail_list_size)
	(_gdbm_print_avail_list): Rewrite using gdbm_avail_traverse.
	* src/proto.h (gdbm_avail_traverse): New proto.
	* src/systems.h: Include stddef.h.

	Follow-up fixes to fd5cf245ea.

	These address https://puszcza.gnu.org.ua/bugs/?503

	* src/gdbmdefs.h (gdbm_avail_block_valid_p): Remove.
	* src/gdbmopen.c (gdbm_avail_block_validate): Use inline conditional
	instead of gdbm_avail_block_valid_p.
	(gdbm_fd_open): Revert to reading master avail_block in two passes (as
	was before fd5cf245ea).
	(validate_header): Add back master avail block consistency check.
	* src/gdbmtool.c (_gdbm_avail_list_size): Use _gdbm_avail_block_read.
	* src/recover.c (_gdbm_finish_transfer): Reset dbf->file_size.

	Follow-up fixes to fd5cf245ea.

	These address https://puszcza.gnu.org.ua/bugs/?503

	* src/gdbmdefs.h (gdbm_avail_block_valid_p): Remove.
	* src/gdbmopen.c (gdbm_avail_block_validate): Use inline conditional
	instead of gdbm_avail_block_valid_p.
	(gdbm_fd_open): Revert to reading master avail_block in two passes (as
	was before fd5cf245ea).
	(validate_header): Add back master avail block consistency check.
	* src/gdbmtool.c (_gdbm_avail_list_size): Use _gdbm_avail_block_read.
	* src/recover.c (_gdbm_finish_transfer): Reset dbf->file_size.

2021-03-16  Sergey Poznyakoff  <gray@gnu.org>

	Verify if key/pair ends at a valid offset before attempting to read it.

	This fixes https://puszcza.gnu.org.ua/bugs/index.php?502.

	* src/findkey.c (gdbm_bucket_element_valid_p): Verify also if the
	key/data pair ends at a valid file offset.
	(_gdbm_read_entry): Set up the cache data only on success.
	* src/gdbmdefs.h (gdbm_bucket_element_valid_p): Remove declaration.
	(gdbm_file_info): New member 'file_size' keeps the recently retrieved
	value of the file size.  It is invalidated wherever data is written to
	the database.
	* src/fullio.c (_gdbm_full_write,_gdbm_file_extend): Invalidate
	dbf->file_size.
	* src/gdbmopen.c (gdbm_fd_open): Initialize file_size to -1.
	(_gdbm_file_size): New function.
	* src/mmap.c (_gdbm_file_size): Remove.
	* src/proto.h (_gdbm_file_size): New proto.

	Improve reading and validation of available blocks (avail_block).

	Fixes http://puszcza.gnu.org.ua/bugs/?501.

	* src/falloc.c (_gdbm_avail_block_read): New function.
	(pop_avail_block): Use _gdbm_avail_block_read.
	* src/gdbmdefs.h: Fix typo in the comment.
	* src/gdbmopen.c (gdbm_avail_block_validate): Take size of the avail
	block in bytes as the third argument.  All uses changed.
	(PARTIAL_HEADER_SIZE)
	(HEADER_AVAIL_SIZE): New macros.
	(_gdbm_validate_header): Validate the avail block as well.
	(validate_header): Don't validate the avail block fields here.
	(gdbm_fd_open): Read partial header up to the avail field, then read
	entire avail block using _gdbm_avail_block_read.
	* src/gdbmtool.c (_gdbm_print_avail_list): Fix avail block calculation
	(one extra entry was assumed).
	Use _gdbm_avail_block_read to obtain and validate block.
	* src/proto.h (_gdbm_avail_block_read): New proto.
	(gdbm_avail_block_validate): Change signature.

	Fix memory leak in gdbm_recover

	* src/recover.c (_gdbm_finish_transfer): Free the cache.

	Fix bug #499

	See http://puszcza.gnu.org.ua/bugs/?499

	* src/gdbm.h.in (GDBM_BAD_HASH_ENTRY): New error code.
	* src/gdbmseq.c (get_next_key): Verify the computed hash and bucket
	address for the obtained key.
	* src/gdbmerrno.c (gdbm_errlist): Add error message for
	GDBM_BAD_HASH_ENTRY.

	Verify if key/pair ends at a valid offset before attempting to read it.

	This fixes https://puszcza.gnu.org.ua/bugs/index.php?502.

	* src/findkey.c (gdbm_bucket_element_valid_p): Verify also if the
	key/data pair ends at a valid file offset.
	(_gdbm_read_entry): Set up the cache data only on success.
	* src/gdbmdefs.h (gdbm_bucket_element_valid_p): Remove declaration.
	(gdbm_file_info): New member 'file_size' keeps the recently retrieved
	value of the file size.  It is invalidated wherever data is written to
	the database.
	* src/fullio.c (_gdbm_full_write,_gdbm_file_extend): Invalidate
	dbf->file_size.
	* src/gdbmopen.c (gdbm_fd_open): Initialize file_size to -1.
	(_gdbm_file_size): New function.
	* src/mmap.c (_gdbm_file_size): Remove.
	* src/proto.h (_gdbm_file_size): New proto.

	Follow-up to 40a464d322

	* src/bucket.c (bucket_read): New function.
	(_gdbm_get_bucket): Use bucket_read.  Invalidate bucket in case
	of error.

	Improve reading and validation of available blocks (avail_block).

	This fixes remaining failures in http://puszcza.gnu.org.ua/bugs/?501.

	* src/falloc.c (_gdbm_avail_block_read): New function.
	(pop_avail_block): Use _gdbm_avail_block_read.
	* src/gdbmdefs.h: Fix typo in the comment.
	* src/gdbmopen.c (gdbm_avail_block_validate): Take size of the avail
	block in bytes as the third argument.  All uses changed.
	(PARTIAL_HEADER_SIZE)
	(HEADER_AVAIL_SIZE): New macros.
	(_gdbm_validate_header): Validate the avail block as well.
	(validate_header): Don't validate the avail block fields here.
	(gdbm_fd_open): Read partial header up to the avail field, then read
	entire avail block using _gdbm_avail_block_read.
	* src/gdbmtool.c (_gdbm_print_avail_list): Fix avail block calculation
	(one extra entry was assumed).
	Use _gdbm_avail_block_read to obtain and validate block.
	* src/proto.h (_gdbm_avail_block_read): New proto.
	(gdbm_avail_block_validate): Change signature.

	Don't leave erroneous bucket in GDBM_FILE if _gdbm_get_bucket fails

	This fixes most of bug #501 (http://puszcza.gnu.org.ua/bugs/?501).

	* src/bucket.c (_gdbm_get_bucket): Reinitialize dbf->bucket to
	NULL before returning failure.

2021-02-06  Sergey Poznyakoff  <gray@gnu.org>

	Fix bug #499

	See http://puszcza.gnu.org.ua/bugs/?499

	* src/gdbm.h.in (GDBM_BAD_HASH_ENTRY): New error code.
	* src/gdbmseq.c (get_next_key): Verify the computed hash and bucket
	address for the obtained key.
	* src/gdbmerrno.c (gdbm_errlist): Add error message for
	GDBM_BAD_HASH_ENTRY.

2021-01-09  Sergey Poznyakoff  <gray@gnu.org>

	Merge branch 'master' into newcache

	Minor changes

	* src/hash.c (_gdbm_hash): Suppress address sanitizer errors.
	* src/proto.h (gdbm_file_sync): Add missing return.
	* src/lex.l (input_context_pop): Delete current buffer when tos
	is reached.

2021-01-08  Sergey Poznyakoff  <gray@gnu.org>

	Reword some messages and add translator comments.

	* src/gdbmtool.c: Avoid duplicating the GDBM_BAD_AVAIL message.
	Add translator comments.
	* src/gdbmerrno.c: Add translator comments.
	* src/util.c: Likewise.

2021-01-02  Sergey Poznyakoff  <gray@gnu.org>

	Update copyright years

	Update copyright years

2020-12-24  Sergey Poznyakoff  <gray@gnu.org>

	Merge branch 'master' into newcache

	Fix the use of unprefixed dnl in configure.ac

2020-12-23  Sergey Poznyakoff  <gray@gnu.org>

	Version 1.19

	Update the docs

	* doc/Makefile.am: Use gendocs.pl to generate web version of the
	manual.
	* doc/gdbm.texi: Update.
	* doc/gendocs.pl: New file.
	* doc/otherdoc.texi.in: New file.
	* doc/.gitignore: Ignore otherdoc.texi.
	* doc/webdoc.init: New file.
	* doc/gendocs.sh: Remove.
	* doc/gendocs_template: Remove.
	* doc/html.init: Remove.

2020-12-23  Sergey Poznyakoff  <gray@gnu.org.ua>

	Update copyright years

2020-12-23  Sergey Poznyakoff  <gray@gnu.org>

	Remove duplicate global declarations

	* src/gdbm_dump.c: Add missing I18N.
	* src/gdbm_load.c: Likewise.
	* src/parseopt.c (parseopt_program_doc)
	(parseopt_program_args): Remove duplicate declarations.

2020-10-31  Sergey Poznyakoff  <gray@gnu.org>

	Recover from ENOMEM in cache_lookup

	* src/bucket.c (cache_lookup): Try to recover from ENOMEM by freeing
	the least recently used cache entry.

2020-10-29  Sergey Poznyakoff  <gray@gnu.org>

	Minor changes.

	* src/bucket.c (_gdbm_cache_init): Minimum cache capacity is 1.
	* src/cachetree.c (_gdbm_cache_tree_lookup): Fix the description.
	(_gdbm_cache_tree_destroy): Speed up the tree traversal.
	* src/gdbmsetopt.c (setopt_gdbm_setcachesize): Don't enforce
	minimal cache capacity.

2020-10-09  Sergey Poznyakoff  <gray@gnu.org>

	Implement tagged structure initializers.

	* NEWS: Document changes.
	* doc/gdbm.texi: Document changes.
	* src/datconv.c (datum_scan_tag): Implement tagged structure
	initialization.
	* src/gdbmtool.c (kvlist_free): Made extern.
	(kvlist_find): New function.
	* src/gdbmtool.h (kvlist_find, kvlist_free)
	(dsegm_list_find): New protos.
	(dsegm_free_list): Rename to dsegm_list_free. All uses changed.
	* src/gram.y: Forbid duplicate tags in kvlist.

2020-10-08  Sergey Poznyakoff  <gray@gnu.org>

	Bugfixes.

	* doc/gdbm.texi: Fix the description of a sample content structure.
	* src/datconv.c (datum_scan_notag): Proceed to the next kvpair only
	for segments of the FDEF_FLD type.

2020-03-15  Sergey Poznyakoff  <gray@gnu.org>

	Implement the gdbm_bucket_count function.

	* src/gdbmcount.c (gdbm_bucket_count): New function.
	* src/gdbm.h.in (gdbm_bucket_count): New proto.
	* doc/gdbm.texi: Document gdbm_bucket_count.
	* src/gdbmtool.c (bucket_count): Reimplement via gdbm_bucket_count.

2020-01-27  Sergey Poznyakoff  <gray@gnu.org.ua>

	Update copyright years

	Minor changes

	* src/cachetree.c (_gdbm_node_set_elem): Remove function.
	* src/findkey.c: Finalize efaefe4c.

2019-11-16  Sergey Poznyakoff  <gray@gnu.org>

	Simplify memory allocation

	* src/gdbmdefs.h (cache_elem): Place one hash_bucket element at the
	end of the structure so it can be allocated along with it.
	* src/bucket.c (cache_elem_new): Allocate enough space for cache_elem
	and dbf->header->bucket_size hash bucket within.

2019-11-13  Sergey Poznyakoff  <gray@gnu.org.ua>

	Move cache element allocation to bucket.c.

	* src/bucket.c (_gdbm_cache_elem_new): Rename to cache_elem_new.
	Make static.
	(cache_lookup): Allocate new elem if _gdbm_cache_tree_lookup
	returned node_new.
	* src/cachetree.c (cache_tree): Remove dbf.
	(_gdbm_cache_tree_delete): Use node->adr.
	(cache_tree_lookup): Rename to _gdbm_cache_tree_lookup. Change
	signature. Return pointer to the (found or inserted) node. Don't
	allocate elem, this is responsibility of the caller.
	(_gdbm_cache_tree_alloc): Change signature.
	* src/gdbmdefs.h (cache_node): New definition (from cachetree.c)
	* src/proto.h: Fix protos.

	Namespace cleanup

	* src/proto.h (_gdbm_cache_tree_delete): Remove.
	(_gdbm_rbt_remove_node): Rename to _gdbm_cache_tree_delete.
	All uses changed

	Don't cache data pages

	* src/findkey.c (_gdbm_read_entry): Revert dc176a5c

2019-11-12  Sergey Poznyakoff  <gray@gnu.org>

	Rewrite bucket cache

	The new bucket cache uses the least recently used replacement
	policy (instead of the least recently read, implemented previously).
	It also allows for quick bucket lookups by the corresponding
	disk address. To this effect the cache entries form a red-black
	tree sorted by bucket address.

	Additionally, data buckets are also cached.

	* README: Describe the new branch.
	* src/bucket.c: Rewrite cache support.
	* src/cachetree.c: New file.
	* src/Makefile.am: Add new file.
	* src/findkey.c (_gdbm_read_entry): Use _gdbm_fetch_data.
	This ensures data pages are cached as well as buckets.
	* src/gdbm.h.in (GDBM_BUCKET_CACHE_CORRUPTED): New error code.
	(gdbm_cache_stat): New struct.
	(gdbm_get_cache_stats): New proto.
	* src/gdbmclose.c (gdbm_close): Call _gdbm_cache_free to dispose
	of the cache.
	* src/gdbmdefs.h (cache_elem_color): New data type.
	(cache_elem): New members: ca_left, ca_right, ca_node, and
	ca_hits.
	(cache_tree): New typedef.
	(gdbm_file_info): Remove bucket_cache and last_read.
	New fields: cache_num, cache_tree, cache_mru, cache_lru,
	cache_avail, cache_access_count.
	* src/gdbmerrno.c: Handle GDBM_BUCKET_CACHE_CORRUPTED.
	* src/gdbmopen.c (gdbm_fd_open): Change cache initialization.
	(_gdbm_init_cache, _gdbm_cache_entry_invalidate: Remove.
	* src/gdbmsetopt.c (setopt_gdbm_setcachesize): Cache can be
	re-initialized on the fly.
	* src/gdbmtool.c: Change bucket printing routines.
	* src/proto.h (_gdbm_read_bucket_at): Remove.
	(_gdbm_fetch_data,_gdbm_cache_init,_gdbm_cache_free)
	(_gdbm_cache_flush,_gdbm_cache_elem_new)
	(_gdbm_cache_tree_alloc,_gdbm_cache_tree_destroy)
	(_gdbm_cache_tree_delete,_gdbm_rbt_remove_node)
	(_gdbm_cache_tree_lookup): New protos.
	(_gdbm_init_cache,_gdbm_cache_entry_invalidate): Remove.
	* src/recover.c (_gdbm_finish_transfer): Adapt to the new
	cache structure.
	* src/update.c: Likewise.
	* tests/setopt00.at: Fix second GDBM_SETCACHESIZE test.

2019-10-29  Sergey Poznyakoff  <gray@gnu.org>

	Pre-read the memory mapped regions on systems that support it

	* src/mmap.c (GDBM_MMAP_FLAGS): Define to MAP_POPULATE or
	MAP_PREFAULT_READ, if available.
	(_gdbm_internal_remap): Use GDBM_MMAP_FLAGS.

	Minor changes

	* src/gdbmconst.h: Don't redefine SIZE_T_MAX
	* src/gdbmtool.c: Likewise.
	Include <termios.h>, instead of <sys/termios.h>
	* configure.ac: Don't check for termios.h
	* tests/closerr.c: Minor fixes.

2019-04-08  Sergey Poznyakoff  <gray@gnu.org>

	Update copyright years

	Preserve locking type during database reorganization

	* src/recover.c (_gdbm_finish_transfer): Preserve locking type.

2018-11-27  Sergey Poznyakoff  <gray@gnu.org>

	Fix typo in the manpage.

2018-10-27  Sergey Poznyakoff  <gray@gnu.org>

	Version 1.18.1

2018-10-18  Sergey Poznyakoff  <gray@gnu.org>

	Restore backward compatibility with older databases

	* src/gdbmopen.c (validate_header): Silently accept databases with next_block
	pointing beyond end of file.

	Attempt recovery in case of invalid next_block header field

	* src/gdbmopen.c (validate_header): Return GDBM_NEED_RECOVERY
	if next_block is invalid.
	(_gdbm_validate_header): New function.
	(gdbm_fd_open): Set need_recovery depending on return from validate_header.
	(gdbm_open): Bail out on invalid value of GDBM_OPENMASK bits.
	* src/proto.h (_gdbm_validate_header): New proto.
	* src/recover.c (check_db): Re-validate the header.
	* src/gdbmtool.c (export_handler): Fix option processing.

2018-08-31  Sergey Poznyakoff  <gray@gnu.org>

	Various bugfixes.

	* compat/dbmopen.c (ndbm_open_dir_file0): Ignore ENOENT.
	* src/falloc.c (push_avail_block): Free temporary storage no matter
	what return status.
	* src/gdbm.h.in (GDBM_FILE_TRUNCATE_ERROR): New error code.
	* src/gdbmdump.c (_gdbm_dump_ascii): Initialize rc.
	* src/gdbmerrno.c: Handle new error.code
	* src/gdbmload.c (gdbm_load_bdb_dump): Initialize rc
	* src/gdbmopen.c (_gdbm_ftruncate): New function.
	(gdbm_fd_open): Use _gdbm_ftruncate. Check its return.
	* src/gdbmseq.c (gdbm_firstkey): Initialize dsize
	* src/gdbmtool.c (command_generator): Check if cmd is NULL.
	(shouldn't happen, but anyways).
	* src/mmap.c (_gdbm_mapped_lseek): Check for vailidity of the 'whence'
	parameter.
	* src/systems.h (TRUNCATE): Remove macro.
	* src/util.c (vgetyn): Remove unnecessary assignment.
