Homework: Tokenizer

This homework is necessary preparation for the lab. Make sure you type your code and you bring the file to the lab so that you will not have to waste time entering it during the lab.

1. Carefully review the slides on Tokenizing.
2. Complete the body of the following private static method.

       /**
        * Returns the first "word" (maximal length string of characters not in
        * {@code SEPARATORS}) or "separator string" (maximal length string of
        * characters in {@code SEPARATORS}) in the given {@code text} starting at
        * the given {@code position}.
        * 
        * @param text
        *            the {@code String} from which to get the word or separator
        *            string
        * @param position
        *            the starting index
        * @return the first word or separator string found in {@code text} starting
        *         at index {@code position}
        * @requires 0 <= position < |text|
        * @ensures <pre>
        * nextWordOrSeparator =
        *   text[position, position + |nextWordOrSeparator|)  and
        * if entries(text[position, position + 1)) intersection entries(SEPARATORS) = {}
        * then
        *   entries(nextWordOrSeparator) intersection entries(SEPARATORS) = {}  and
        *   (position + |nextWordOrSeparator| = |text|  or
        *    entries(text[position, position + |nextWordOrSeparator| + 1))
        *      intersection entries(SEPARATORS) /= {})
        * else
        *   entries(nextWordOrSeparator) is subset of entries(SEPARATORS)  and
        *   (position + |nextWordOrSeparator| = |text|  or
        *    entries(text[position, position + |nextWordOrSeparator| + 1))
        *      is not subset of entries(SEPARATORS))
        * </pre>
        */
       private static String nextWordOrSeparator(String text, int position) {...}
   

   This is a modified version of the method you wrote in a homework and lab in Software I. Feel free to reuse your own code (but not someone else's) for this homework. Note that the method here has one less parameter than in the Software I version. The "separators" are defined as a String constant as follows:

       /**
        * Definition of whitespace separators.
        */
       private static final String SEPARATORS = " \t\n\r";
   

3. Complete the body of the following public static method. You should use nextWordOrSeparator in your solution because (except for END_OF_INPUT as explained below) "non-separator token" and "non-whitespace token" here mean "word" as used in nextWordOrSeparator.

       /**
        * Tokenizes the entire input getting rid of all whitespace separators and
        * returning the non-separator tokens in a {@code Queue<String>}.
        * 
        * @param in
        *            the input stream
        * @return the queue of tokens
        * @updates in.content
        * @requires in.is_open
        * @ensures <pre>
        * tokens =
        *   [the non-whitespace tokens in #in.content] * <END_OF_INPUT>  and
        * in.content = <>
        * </pre>
        */
       public static Queue<String> tokens(SimpleReader in) {...}
   

   The END_OF_INPUT token used in the ensures clause is defined as a String constant as follows:

       /**
        * Token to mark the end of the input. This token cannot come from the input
        * stream because it contains whitespace.
        */
       public static final String END_OF_INPUT = "### END OF INPUT ###";