bit Namespace Reference

A namespace containing classes for storing n-gram language models in a bit-packed trie structure. More...

Classes
class	Array
	A bit-packed array of fixed-bit-width elements. More...
class	Buffer
	Buffer storing and retrieving variable-width values (0-32 bits) in arbitrary bit positions. More...
class	CompressedArray
	Compressed array of sorted values. More...
class	invalid_argument
	Thrown to report invalid arguments to functions. More...
class	out_of_range
	Thrown to report values that exceed the expected range. More...
class	invalid_call
	Thrown to report invalid method calls. More...
class	io_error
	Thrown to report input output errors. More...
class	FloatArray
	Array for storing floats in quantized form. More...
class	Trie
	A trie structure for storing a set of integer vectors compactly. More...
class	Kneser
	Pruning algorithm for buildling variable-order models with Kneser-Ney smoothing presented in: Reinhard Kneser. More...
class	SimpleKneser
struct	Compare
class	ArpaReader
	Reads ARPA language model file. More...
class	LM
	Class for storing an n-gram language model in a bit-packed trie structure. More...
class	Perplexity
	Compute perplexity or cross-entropy of a language model. More...
class	SymbolMap
	An enumerated map of symbols. More...
Typedefs
typedef unsigned int	u32
	Unsigned 32-bit type.
typedef signed int	s32
	Signed 32-bit type.
typedef unsigned long long	u64
	Unsigned 64-bit type.
typedef unsigned long long	s64
	Signed 64-bit type.
Functions
template<class A>
u32	max (const A &array)
	Find the maximum element from an array.
template<class A>
u64	binary_search (const A &array, u32 value, u64 first, u64 limit)
	Find the first index of a value from a sorted array.
template<class A>
u64	lower_bound (const A &array, u32 value, u64 first, u64 limit)
	Find the smallest index at which value could be inserted maintaining array in sorted order.
template<class A>
u64	last_leq (const A &array, u32 value)
	From a sorted array, find the largest index that has a equal or smaller value than specified.
template<class A>
u64	last_leq (const A &array, u32 value, u64 first, u64 limit)
	From a sorted array range, find the largest index that has a equal or smaller value than specified.
template<class A>
u64	last_leq_naive (const A &array, u32 value, u64 first, u64 limit)
u64	bytes_required (u64 num_elems, unsigned int bits_per_elem)
	The number of 8-bit bytes required to store a number of fixed-width elements.
unsigned int	highest_bit (u64 value)
	The number of highest non-zero bit.
unsigned int	float2uint (float f)
	Convert float to 32-bit integer representation.
float	uint2float (unsigned int i)
	Restore float from 32-bit integer representation.
float	unquantize_float (u32 i, double step)
	Restore a quantized float value using linear quantization centered around zero.
u32	quantize_float (float f, double step)
	Quantize float value using linear quantization centered around zero.
Variables
static const unsigned int	max_bits_per_value = 32
	Maximum value width in bits.
static const u32	max_u32 = (u32)-1
	Maximum value of u64 type.
static const s32	max_s32 = 0x7fffffff
	Maximum value of s32 type.
static const u64	max_u64 = (u64)-1
	Maximum value of u64 type.
static const unsigned int	one_masks []
	Bit-masks containin n lowest bits set to one.
static const int	shift [] = { 0, 8, 16, 24, 32 }
	Bit-shifts for byte positions.

---

Detailed Description

A namespace containing classes for storing n-gram language models in a bit-packed trie structure.

---

Typedef Documentation

typedef signed int bit::s32

Signed 32-bit type.

typedef unsigned long long bit::s64

Signed 64-bit type.

typedef unsigned int bit::u32

Unsigned 32-bit type.

typedef unsigned long long bit::u64

Unsigned 64-bit type.

---

Function Documentation

template<class A> u64 bit::binary_search (  const A &  array, u32  value, u64  first, u64  limit )

Find the first index of a value from a sorted array. Parameters: array  = the array to be examined value  = the value to be searched first  = the first index in the range considered limit  = the index limiting the range (not included in the range) Returns: the index of the value or limit if not found Exceptions: bit::invalid_argument  if range is negative bit::out_of_range  if range exceeds array size

u64 bit::bytes_required (  u64  num_elems, unsigned int  bits_per_elem )  [inline]

The number of 8-bit bytes required to store a number of fixed-width elements. The bits_per_elem must be 1-32 and the required size must be less than 2^64 bits.

unsigned int bit::float2uint (  float  f  )  [inline]

Convert float to 32-bit integer representation.

unsigned int bit::highest_bit (  u64  value  )  [inline]

The number of highest non-zero bit.

template<class A> u64 bit::last_leq (  const A &  array, u32  value, u64  first, u64  limit )

From a sorted array range, find the largest index that has a equal or smaller value than specified. Parameters: array  = the array to be examined value  = the candidate value first  = the start of the range limit  = the limit of the range (not included in range) Returns: the largest index containing equal or smaller value; max_u64 if there exists no such index inside range (value is smaller than first value in the range); Exceptions: bit::invalid_argument  if negative range bit::out_of_range  if limit greater than number of elements

template<class A> u64 bit::last_leq (  const A &  array, u32  value )

From a sorted array, find the largest index that has a equal or smaller value than specified. Parameters: array  = the array to be examined value  = the candidate value Returns: the largest index containing equal or smaller value; max_u64 if there exists no such index inside range (value is smaller than first value in the range);

template<class A> u64 bit::last_leq_naive (  const A &  array, u32  value, u64  first, u64  limit )

template<class A> u64 bit::lower_bound (  const A &  array, u32  value, u64  first, u64  limit )

Find the smallest index at which value could be inserted maintaining array in sorted order. Parameters: array  = the array to be examined value  = the candidate value first  = the first index in the range considered limit  = the index limiting the range (not included in the range) Returns: the index at which the value should be inserted Exceptions: bit::invalid_argument  if range is negative bit::out_of_range  if range exceeds array size

template<class A> u32 bit::max (  const A &  array  )

Find the maximum element from an array.

u32 bit::quantize_float (  float  f, double  step )  [inline]

Quantize float value using linear quantization centered around zero. The least-significant bit is used for sign to keep near-zero values with least number of bits. Parameters: f  = float to quantize step  = length of the quantization step (0 = no quantization) Returns: the quantized value Exceptions: invalid_argument  if quantized value overflows linear scale

float bit::uint2float (  unsigned int  i  )  [inline]

Restore float from 32-bit integer representation.

float bit::unquantize_float (  u32  i, double  step )  [inline]

Restore a quantized float value using linear quantization centered around zero. The least-significant bit is used for sign to keep near-zero values with least number of bits. Parameters: i  = integer to unquantize step  = length of the quantization step (0 = no quantization) Returns: the restored float value

---

Variable Documentation

const unsigned int bit::max_bits_per_value = 32 [static]

Maximum value width in bits.

const s32 bit::max_s32 = 0x7fffffff [static]

Maximum value of s32 type.

const u32 bit::max_u32 = (u32)-1 [static]

Maximum value of u64 type.

const u64 bit::max_u64 = (u64)-1 [static]

Maximum value of u64 type.

const unsigned int bit::one_masks[] [static]

Initial value: { 0x00000000, 0x00000001, 0x00000003, 0x00000007, 0x0000000f, 0x0000001f, 0x0000003f, 0x0000007f, 0x000000ff, 0x000001ff, 0x000003ff, 0x000007ff, 0x00000fff, 0x00001fff, 0x00003fff, 0x00007fff, 0x0000ffff, 0x0001ffff, 0x0003ffff, 0x0007ffff, 0x000fffff, 0x001fffff, 0x003fffff, 0x007fffff, 0x00ffffff, 0x01ffffff, 0x03ffffff, 0x07ffffff, 0x0fffffff, 0x1fffffff, 0x3fffffff, 0x7fffffff, 0xffffffff } Bit-masks containin n lowest bits set to one.

const int bit::shift[] = { 0, 8, 16, 24, 32 } [static]

Bit-shifts for byte positions.

---