bit Namespace Reference

A namespace containing classes for storing n-gram language models in a bit-packed trie structure. More...


Classes

class  Array
 A bit-packed array of fixed-bit-width elements. More...
class  Buffer
 Buffer storing and retrieving variable-width values (0-32 bits) in arbitrary bit positions. More...
class  CompressedArray
 Compressed array of sorted values. More...
class  invalid_argument
 Thrown to report invalid arguments to functions. More...
class  out_of_range
 Thrown to report values that exceed the expected range. More...
class  invalid_call
 Thrown to report invalid method calls. More...
class  io_error
 Thrown to report input output errors. More...
class  FloatArray
 Array for storing floats in quantized form. More...
class  Trie
 A trie structure for storing a set of integer vectors compactly. More...
class  Kneser
 Pruning algorithm for buildling variable-order models with Kneser-Ney smoothing presented in: Reinhard Kneser. More...
class  SimpleKneser
struct  Compare
class  ArpaReader
 Reads ARPA language model file. More...
class  LM
 Class for storing an n-gram language model in a bit-packed trie structure. More...
class  Perplexity
 Compute perplexity or cross-entropy of a language model. More...
class  SymbolMap
 An enumerated map of symbols. More...

Typedefs

typedef unsigned int u32
 Unsigned 32-bit type.
typedef signed int s32
 Signed 32-bit type.
typedef unsigned long long u64
 Unsigned 64-bit type.
typedef unsigned long long s64
 Signed 64-bit type.

Functions

template<class A>
u32 max (const A &array)
 Find the maximum element from an array.
template<class A>
u64 binary_search (const A &array, u32 value, u64 first, u64 limit)
 Find the first index of a value from a sorted array.
template<class A>
u64 lower_bound (const A &array, u32 value, u64 first, u64 limit)
 Find the smallest index at which value could be inserted maintaining array in sorted order.
template<class A>
u64 last_leq (const A &array, u32 value)
 From a sorted array, find the largest index that has a equal or smaller value than specified.
template<class A>
u64 last_leq (const A &array, u32 value, u64 first, u64 limit)
 From a sorted array range, find the largest index that has a equal or smaller value than specified.
template<class A>
u64 last_leq_naive (const A &array, u32 value, u64 first, u64 limit)
u64 bytes_required (u64 num_elems, unsigned int bits_per_elem)
 The number of 8-bit bytes required to store a number of fixed-width elements.
unsigned int highest_bit (u64 value)
 The number of highest non-zero bit.
unsigned int float2uint (float f)
 Convert float to 32-bit integer representation.
float uint2float (unsigned int i)
 Restore float from 32-bit integer representation.
float unquantize_float (u32 i, double step)
 Restore a quantized float value using linear quantization centered around zero.
u32 quantize_float (float f, double step)
 Quantize float value using linear quantization centered around zero.

Variables

static const unsigned int max_bits_per_value = 32
 Maximum value width in bits.
static const u32 max_u32 = (u32)-1
 Maximum value of u64 type.
static const s32 max_s32 = 0x7fffffff
 Maximum value of s32 type.
static const u64 max_u64 = (u64)-1
 Maximum value of u64 type.
static const unsigned int one_masks []
 Bit-masks containin n lowest bits set to one.
static const int shift [] = { 0, 8, 16, 24, 32 }
 Bit-shifts for byte positions.


Detailed Description

A namespace containing classes for storing n-gram language models in a bit-packed trie structure.

Typedef Documentation

typedef signed int bit::s32
 

Signed 32-bit type.

typedef unsigned long long bit::s64
 

Signed 64-bit type.

typedef unsigned int bit::u32
 

Unsigned 32-bit type.

typedef unsigned long long bit::u64
 

Unsigned 64-bit type.


Function Documentation

template<class A>
u64 bit::binary_search const A &  array,
u32  value,
u64  first,
u64  limit
 

Find the first index of a value from a sorted array.

Parameters:
array = the array to be examined
value = the value to be searched
first = the first index in the range considered
limit = the index limiting the range (not included in the range)
Returns:
the index of the value or limit if not found
Exceptions:
bit::invalid_argument if range is negative
bit::out_of_range if range exceeds array size

u64 bit::bytes_required u64  num_elems,
unsigned int  bits_per_elem
[inline]
 

The number of 8-bit bytes required to store a number of fixed-width elements.

The bits_per_elem must be 1-32 and the required size must be less than 2^64 bits.

unsigned int bit::float2uint float  f  )  [inline]
 

Convert float to 32-bit integer representation.

unsigned int bit::highest_bit u64  value  )  [inline]
 

The number of highest non-zero bit.

template<class A>
u64 bit::last_leq const A &  array,
u32  value,
u64  first,
u64  limit
 

From a sorted array range, find the largest index that has a equal or smaller value than specified.

Parameters:
array = the array to be examined
value = the candidate value
first = the start of the range
limit = the limit of the range (not included in range)
Returns:
the largest index containing equal or smaller value; max_u64 if there exists no such index inside range (value is smaller than first value in the range);
Exceptions:
bit::invalid_argument if negative range
bit::out_of_range if limit greater than number of elements

template<class A>
u64 bit::last_leq const A &  array,
u32  value
 

From a sorted array, find the largest index that has a equal or smaller value than specified.

Parameters:
array = the array to be examined
value = the candidate value
Returns:
the largest index containing equal or smaller value; max_u64 if there exists no such index inside range (value is smaller than first value in the range);

template<class A>
u64 bit::last_leq_naive const A &  array,
u32  value,
u64  first,
u64  limit
 

template<class A>
u64 bit::lower_bound const A &  array,
u32  value,
u64  first,
u64  limit
 

Find the smallest index at which value could be inserted maintaining array in sorted order.

Parameters:
array = the array to be examined
value = the candidate value
first = the first index in the range considered
limit = the index limiting the range (not included in the range)
Returns:
the index at which the value should be inserted
Exceptions:
bit::invalid_argument if range is negative
bit::out_of_range if range exceeds array size

template<class A>
u32 bit::max const A &  array  ) 
 

Find the maximum element from an array.

u32 bit::quantize_float float  f,
double  step
[inline]
 

Quantize float value using linear quantization centered around zero.

The least-significant bit is used for sign to keep near-zero values with least number of bits.

Parameters:
f = float to quantize
step = length of the quantization step (0 = no quantization)
Returns:
the quantized value
Exceptions:
invalid_argument if quantized value overflows linear scale

float bit::uint2float unsigned int  i  )  [inline]
 

Restore float from 32-bit integer representation.

float bit::unquantize_float u32  i,
double  step
[inline]
 

Restore a quantized float value using linear quantization centered around zero.

The least-significant bit is used for sign to keep near-zero values with least number of bits.

Parameters:
i = integer to unquantize
step = length of the quantization step (0 = no quantization)
Returns:
the restored float value


Variable Documentation

const unsigned int bit::max_bits_per_value = 32 [static]
 

Maximum value width in bits.

const s32 bit::max_s32 = 0x7fffffff [static]
 

Maximum value of s32 type.

const u32 bit::max_u32 = (u32)-1 [static]
 

Maximum value of u64 type.

const u64 bit::max_u64 = (u64)-1 [static]
 

Maximum value of u64 type.

const unsigned int bit::one_masks[] [static]
 

Initial value:

 
  { 0x00000000, 
    0x00000001, 0x00000003, 0x00000007, 0x0000000f,
    0x0000001f, 0x0000003f, 0x0000007f, 0x000000ff,
    0x000001ff, 0x000003ff, 0x000007ff, 0x00000fff,
    0x00001fff, 0x00003fff, 0x00007fff, 0x0000ffff,
    0x0001ffff, 0x0003ffff, 0x0007ffff, 0x000fffff,
    0x001fffff, 0x003fffff, 0x007fffff, 0x00ffffff,
    0x01ffffff, 0x03ffffff, 0x07ffffff, 0x0fffffff,
    0x1fffffff, 0x3fffffff, 0x7fffffff, 0xffffffff }
Bit-masks containin n lowest bits set to one.

const int bit::shift[] = { 0, 8, 16, 24, 32 } [static]
 

Bit-shifts for byte positions.


Generated on Mon Jan 8 15:51:04 2007 for bit by  doxygen 1.4.6