FTokenizer Class

Purpose:

The File Tokenizer (FTokenizer) returns a single token (via the extraction operator) from a file.

The extraction operator of the FTokenizer object is repeatedly called to grab the  next token in the file.

When no more token can be found in the string, more() and done() will return false and true respectively. 

FTokenizer Class:

• Receives a file name via its constructor
• Opens the file and reads a block of characters (ifstream::read) and sets up the STokenizer member object.
• As the extraction operator is called repeatedly, it receives a token from STokenizer and returns it.
• When _stk runs out of characters to process, it will refill _stk via its set_string function.
• If there are no more characters in the file to process, more() will return false.

File Tokenizer Design:

class FTokenizer
{
public:
    const int MAX_BLOCK = MAX_BUFFER;
    FTokenizer(char* fname);
    Token next_token();
    bool more();        //returns the current value of _more
    int pos();          //returns the value of _pos
    int block_pos();     //returns the value of _blockPos
    friend FTokenizer& operator >> (FTokenizer& f, Token& t);
private:
    bool get_new_block(); //gets the new block from the file
    std::ifstream _f;   //file being tokenized
    STokenizer _stk;     //The STokenizer object to tokenize current block
    int _pos;           //Current position in the file
    int _blockPos;      //Current position in the current block
    bool _more;         //false if last token of the last block
                        //  has been processed and now we are at
                        //  the end of the last block.
};

The Calling Function:

void test_f_tokenize_simple(){
    Token t;
    FTokenizer ftk("solitude.txt");
    ftk>>t;
    int token_count = 0;
    while (ftk.more()){
        if (t.type_string()=="ALPHA" ){
            token_count++;
            cout<<setw(10)<<token_count
                <<setw(3)<<left<<":"<<setw(25)<<left<<t.token_str()
                <<t.type_string()<<endl;
        }
        ftk>>t;
    }
    cout<<"Tokens Found: "<<token_count<<endl;
    cout<<"=========="<<endl;

}

Test File:

solitude.txt