Java - StreamTokenizer eolIsSignificant(boolean flag) method

---

---

Description

The Java StreamTokenizer eolIsSignificant(boolean flag) method determines whether or not ends of line are treated as tokens. If the flag argument is true, this tokenizer treats end of lines as tokens; the nextToken method returns TT_EOL and also sets the ttype field to this value when an end of line is read.

A line is a sequence of characters ending with either a carriage-return character ('\r') or a newline character ('\n'). In addition, a carriage-return character followed immediately by a newline character is treated as a single end-of-line token.

If the flag is false, end-of-line characters are treated as white space and serve only to separate tokens.

Declaration

Following is the declaration for java.io.StreamTokenizer.eolIsSignificant(boolean flag) method.

public void eolIsSignificant(boolean flag)

Parameters

flag − true indicates that end-of-line characters are separate tokens; false indicates that end-of-line characters are white space.

Return Value

This method does not return a value.

Exception

NA

Example - Usage of StreamTokenizer eolIsSignificant(boolean flag) method

The following example shows the usage of StreamTokenizer eolIsSignificant(boolean flag) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.BufferedReader;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStreamReader;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.Reader;
import java.io.StreamTokenizer;

public class StreamTokenizerDemo {
   public static void main(String[] args) {      
      String text = "Hello. This is a text \n that will be split "
         + "into tokens. 1 + 1 = 2";
      
      try {
         // create a new file with an ObjectOutputStream
         FileOutputStream out = new FileOutputStream("test.txt");
         ObjectOutputStream oout = new ObjectOutputStream(out);

         // write something in the file
         oout.writeUTF(text);
         oout.flush();

         // create an ObjectInputStream for the file we created before
         ObjectInputStream ois = new ObjectInputStream(new FileInputStream("test.txt"));

         // create a new tokenizer
         Reader r = new BufferedReader(new InputStreamReader(ois));
         StreamTokenizer st = new StreamTokenizer(r);

         // set that end of line is significant
         st.eolIsSignificant(true);

         // print the stream tokens
         boolean eof = false;
         
         do {
            int token = st.nextToken();
            
            switch (token) {
               case StreamTokenizer.TT_EOF:
                  System.out.println("End of File encountered.");
                  eof = true;
                  break;
                  
               case StreamTokenizer.TT_EOL:
                  System.out.println("End of Line encountered.");
                  break;
                  
               case StreamTokenizer.TT_WORD:
                  System.out.println("Word: " + st.sval);
                  break;
                  
               case StreamTokenizer.TT_NUMBER:
                  System.out.println("Number: " + st.nval);
                  break;
                  
               default:
                  System.out.println((char) token + " encountered.");
                  
                  if (token == '!') {
                     eof = true;
                  }
            }
         } while (!eof);
         
      } catch (Exception ex) {
         ex.printStackTrace();
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result −

Word: Hello.
Word: This
Word: is
Word: a
Word: text
End of Line encountered.
Word: that
Word: will
Word: be
Word: split
Word: into
Word: tokens.
Number: 1.0
+ encountered.
Number: 1.0
= encountered.
Number: 2.0
End of File encountered.

Example - eolIsSignificant(true) − Detect end of lines

The following example shows the usage of StreamTokenizer eolIsSignificant(boolean flag) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Hello 123\nWorld 456\nDone";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.eolIsSignificant(true); // Enable EOL tokens

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         switch (tokenizer.ttype) {
            case StreamTokenizer.TT_WORD:
               System.out.println("Word: " + tokenizer.sval);
               break;
            case StreamTokenizer.TT_NUMBER:
               System.out.println("Number: " + tokenizer.nval);
               break;
            case StreamTokenizer.TT_EOL:
               System.out.println("End of line");
               break;
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: Hello
Number: 123.0
End of line
Word: World
Number: 456.0
End of line
Word: Done

Explanation

• With eolIsSignificant(true), the tokenizer reports each line break.

• You'll see "End of line" printed after each line of tokens.

Example - eolIsSignificant(false) − Ignore line breaks

The following example shows the usage of StreamTokenizer eolIsSignificant(boolean flag) method.

StreamTokenizerDemo.java

package com.tutorialspoint;

import java.io.IOException;
import java.io.Reader;
import java.io.StreamTokenizer;
import java.io.StringReader;

public class StreamTokenizerDemo {
   public static void main(String[] args) throws IOException {
      String input = "Apple 10\nBanana 20\nCherry 30";

      Reader reader = new StringReader(input);
      StreamTokenizer tokenizer = new StreamTokenizer(reader);

      tokenizer.eolIsSignificant(false); // Ignore EOL tokens

      System.out.println("Tokens:");
      while (tokenizer.nextToken() != StreamTokenizer.TT_EOF) {
         if (tokenizer.ttype == StreamTokenizer.TT_WORD) {
            System.out.println("Word: " + tokenizer.sval);
         } else if (tokenizer.ttype == StreamTokenizer.TT_NUMBER) {
            System.out.println("Number: " + tokenizer.nval);
         }
      }
   }
}

Output

Let us compile and run the above program, this will produce the following result−

Tokens:
Word: Apple
Number: 10.0
Word: Banana
Number: 20.0
Word: Cherry
Number: 30.0

Explanation

• Even though the input has multiple lines, the tokenizer treats it as a continuous stream.

• No EOL tokens are reported.