Java - StreamTokenizer ordinaryChars(int low, int hi) method

---

---

Description

The Java StreamTokenizer ordinaryChars(int low, int hi) method specifies that all characters c in the range low <= c <= high are "ordinary" in this tokenizer. See the ordinaryChar method for more information on a character being ordinary.

Declaration

Following is the declaration for java.io.StreamTokenizer.ordinaryChars(int low, int hi) method.

public void ordinaryChars(int low, int hi)

Parameters

• low − The low end of the range.

• hi − The high end of the range.

Return Value

This method does not return a value.

Exception

NA

Example - Usage of StreamTokenizer ordinaryChars(int low, int hi) method

The following example shows the usage of StreamTokenizer ordinaryChars(int low, int hi) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Reader;
import java.io.StreamTokenizer;

public class StreamTokenizerDemo {
   public static void main(String[] args) {
      String text = "Hello. This is a text \n that will be split "
         + "into tokens. 1 + 1 = 2";
         
      try {
         // create a new file with an ObjectOutputStream
         FileOutputStream out = new FileOutputStream("test.txt");
         ObjectOutputStream oout = new ObjectOutputStream(out);

         // write something in the file
         oout.writeUTF(text);
         oout.flush();

         // create an ObjectInputStream for the file we created before
         ObjectInputStream ois = new ObjectInputStream(new FileInputStream("test.txt"));

         // create a new tokenizer
         Reader r = new BufferedReader(new InputStreamReader(ois));
         StreamTokenizer st = new StreamTokenizer(r);

         // set chars a to e as ordinary
         st.ordinaryChars('a', 'e');

         // print the stream tokens
         boolean eof = false;
         
         do {
            int token = st.nextToken();
            
            switch (token) {
               case StreamTokenizer.TT_EOF:
                  System.out.println("End of File encountered.");
                  eof = true;
                  break;
                  
               case StreamTokenizer.TT_EOL:
                  System.out.println("End of Line encountered.");
                  break;
                  
               case StreamTokenizer.TT_WORD:
                  System.out.println("Word: " + st.sval);
                  break;
                  
               case StreamTokenizer.TT_NUMBER:
                  System.out.println("Number: " + st.nval);
                  break;
                  
               default:
                  System.out.println((char) token + " encountered.");
                  
                  if (token == '!') {
                     eof = true;
                  }
            }
         } while (!eof);

      } catch (Exception ex) {
         ex.printStackTrace();
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result −

Word: AH
e encountered.
Word: llo.
Word: This
Word: is
a encountered.
Word: t
e encountered.
Word: xt
Word: th
a encountered.
Word: t
Word: will
b encountered.
e encountered.
Word: split
Word: into
Word: tok
e encountered.
Word: ns.
Number: 1.0
+ encountered.
Number: 1.0
= encountered.
Number: 2.0
End of File encountered.

Example - Treat digits 0 - 9 as ordinary characters

The following example shows the usage of StreamTokenizer ordinaryChars(int low, int hi) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "value123";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.ordinaryChars('0', '9'); // Make all digits ordinary

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else {
            System.out.println("Char: " + (char) tokenizer.ttype);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: value
Char: 1
Char: 2
Char: 3

Explanation

• Normally, 123 would be part of a number or word token.

• By using ordinaryChars('0', '9'), each digit is returned as a separate character token.

Example - Treat punctuation marks as ordinary characters

The following example shows the usage of StreamTokenizer ordinaryChars(int low, int hi) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Hello! How are you?";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.ordinaryChars('!', '?'); // Make '!' and '?' ordinary

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else {
            System.out.println("Punctuation: " + (char) tokenizer.ttype);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: Hello
Punctuation: !
Word: How
Word: are
Word: you
Punctuation: ?

Explanation

• ! and ? are treated as separate punctuation tokens, not ignored or grouped into words.