Lab: Tokenizer

Objective

In this lab you will familiarize yourself with the task of tokenizing a simple input by implementing the core methods in the Tokenizer utility class.

Setup

To get started, import the project for this lab, Tokenizer, from the Tokenizer.zip file available at this link. If you don't remember how to do this, see the Setup instructions in an earlier lab.

Method

1. First carefully look through the file Tokenizer.java in the src folder and make sure you understand all the given members and the main method in particular. Complete the body of the static methods nextWordOrSeparator and tokens by pasting the code you wrote for the homework into the skeleton provided.
2. Run the main program to test your implementation of nextWordOrSeparator and tokens. You can use the SampleProgram.bl and SampleInput.txt files in the data folder in your project as test inputs. The main program generates two output files (both in the data folder): expected-output.txt that contains the correct output for the given input and actual-output.txt that is the output generated by your implementation. Use Eclipse's file compare feature you learned about in the previous labs to compare the two outputs. Fix your code as needed.