Java - StreamTokenizer ordinaryChar(int ch) method

---

---

Description

The Java StreamTokenizer ordinaryChar(int ch) method specifies that the character argument is "ordinary" in this tokenizer. It removes any special significance the character has as a comment character, word component, string delimiter, white space, or number character. When such a character is encountered by the parser, the parser treats it as a single-character token and sets ttype field to the character value.

Making a line terminator character "ordinary" may interfere with the ability of a StreamTokenizer to count lines. The lineno method may no longer reflect the presence of such terminator characters in its line count.

Declaration

Following is the declaration for java.io.StreamTokenizer.ordinaryChar(int ch) method.

public void ordinaryChar(int ch)

Parameters

ch − The character.

Return Value

This method does not return a value.

Exception

NA

Example - Usage of StreamTokenizer ordinaryChar(int ch) method

The following example shows the usage of StreamTokenizer ordinaryChar(int ch) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Reader;
import java.io.StreamTokenizer;

public class StreamTokenizerDemo {
   public static void main(String[] args) {
      String text = "Hello. This is a text \n that will be split "
         + "into tokens. 1 + 1 = 2";
         
      try {
         // create a new file with an ObjectOutputStream
         FileOutputStream out = new FileOutputStream("test.txt");
         ObjectOutputStream oout = new ObjectOutputStream(out);

         // write something in the file
         oout.writeUTF(text);
         oout.flush();

         // create an ObjectInputStream for the file we created before
         ObjectInputStream ois = new ObjectInputStream(new FileInputStream("test.txt"));

         // create a new tokenizer
         Reader r = new BufferedReader(new InputStreamReader(ois));
         StreamTokenizer st = new StreamTokenizer(r);

         // set \n as an ordinary char
         st.ordinaryChar('\n');

         // print the stream tokens
         boolean eof = false;
         
         do {
            int token = st.nextToken();
            
            switch (token) {
               case StreamTokenizer.TT_EOF:
                  System.out.println("End of File encountered.");
                  eof = true;
                  break; 
                  
               case StreamTokenizer.TT_EOL:
                  System.out.println("End of Line encountered.");
                  break;
                  
               case StreamTokenizer.TT_WORD:
                  System.out.println("Word: " + st.sval);
                  break;
                  
               case StreamTokenizer.TT_NUMBER:
                  System.out.println("Number: " + st.nval);
                  break;
                  
               default:
                  System.out.println((char) token + " encountered.");
                  
                  if (token == '!') {
                     eof = true;
                  }
            }
         } while (!eof);

      } catch (Exception ex) {
         ex.printStackTrace();
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result −

Word: AHello.
Word: This
Word: is
Word: a
Word: text
End of Line encountered.
Word: that
Word: will
Word: be
Word: split
Word: into
Word: tokens.
Number: 1.0
+ encountered.
Number: 1.0
= encountered.
Number: 2.0
End of File encountered.

Example - Treat = as a separate symbol

The following example shows the usage of StreamTokenizer ordinaryChar(int ch) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "x=10";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.ordinaryChar('='); // Make '=' an ordinary character

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println("Number: " + tokenizer.nval);
         } else {
            System.out.println("Symbol: " + (char) tokenizer.ttype);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: x
Symbol: =
Number: 10.0

Explanation

• By default, = is treated as part of a word or ignored.

• ordinaryChar('=') forces it to be a standalone symbol token.

Example - Treat + and - as ordinary characters

The following example shows the usage of StreamTokenizer ordinaryChar(int ch) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "a + b - c";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.ordinaryChar('+');
      tokenizer.ordinaryChar('-');

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else {
            System.out.println("Symbol: " + (char) tokenizer.ttype);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: a
Symbol: +
Word: b
Symbol: -
Word: c

Explanation

• The + and - symbols are not treated as operators or part of a number/word.

• They're returned as individual symbol tokens.