Thursday, June 22, 2017

Java Command-Line Interfaces (Part 2): args4j

In my previous post, I looked at parsing command-line arguments in Java applications using Apache Commons CLI. In this post, I look at doing the same using a different library: args4j.

args4j takes a different approach to specifying which command-line arguments the Java application should expect than that used by Commons CLI. While Commons CLI expects objects representing the options to be individually and explicitly instantiated, args4j uses custom annotations to facilitate this "definition" stage of command-line arguments processing. Command-line options are expected to be instance-level fields on the class and are annotated with the @org.kohsuke.args4j.Option annotation. The characteristics of each command-line argument are included as attributes of this @Option annotation.

The simple application demonstrated in this post is similar to that used in my previous post and focuses on an optional and valueless -v option for specifying verbosity and a required -f option which expects a value that represents the file path and name. The next code listing demonstrates use of args4j's @Option annotation to set up these command-line argument as annotation on class data members.

args4j Definition of Command-line Arguments via @Option Annotations

@Option(name="-v", aliases="--verbose", usage="Print verbose status.")
private boolean verbose;

@Option(name="-f", aliases="--file", usage="Fully qualified path and name of file.", required=true)
private String fileName;

As the above code listing demonstrates, it is easy to specify the name of the options, their usage, and whether they are required or not (default is optional). The presence of the private modifier above makes it obvious that these are attributes defined at a class level. Because there is no static modifier, we see that these are instance variables that have been annotated.

To parse the command-line options, one simply needs to instantiate a CmdLineParser and pass the command-line arguments to its parseArguments(String...) method:

Parsing Command-line Arguments in args4j

final CmdLineParser parser = new CmdLineParser(this);
try
{
   parser.parseArgument(arguments);
}
catch (CmdLineException clEx)
{
   out.println("ERROR: Unable to parse command-line options: " + clEx);
}

In the first line of Java code just shown, this is the reference to the instance of the class in which the member variables shown above are defined and annotated with the @Option annotation. In this case, I used this because the same class that defines those options is the one calling this parsing method. To do this in the same class, I needed to have an instance (non-static) method called doMain defined in the class and invoked by the class's main function (this is shown in the complete code listing toward the end of this post). The command-line arguments as received from the class's main(final String[]) function are the array of Strings passed to the parseArguments(String[]) method.

The next two screen snapshots demonstrate application of the described code based on args4j to parsing the command-line arguments. The first image shows combinations of the short and long options for the two options. The second image shows the automatic reporting of the case where a required command-line argument was not provided.

An important feature of a command line parsing library is the ability to display usage or help information. The next code listing demonstrates an example of doing this with args4j's CmdLineParser.printUsage(OutputStream) method.

Printing Usage Information with args4j

final CmdLineParser parser = new CmdLineParser(this);
if (arguments.length < 1)
{
   parser.printUsage(out);
   System.exit(-1);
}

The usage information printed out by default by args4j is depicted in the next screen snapshot.

This post has demonstrated using arg4j to achieve some of the most common functionality related to command-line parsing in Java applications including option "definition", command-line arguments "parsing", "interrogation" of the parsed command-line arguments, and help/usage details related to the command-line arguments. The full code listing for the class partially represented above in code listings is shown now.

Full Code Listing for args4j Demonstration Main.java

package examples.dustin.commandline.args4j;

import static java.lang.System.out;

import org.kohsuke.args4j.CmdLineException;
import org.kohsuke.args4j.CmdLineParser;
import org.kohsuke.args4j.Option;

import java.io.IOException;

/**
 * Demonstrate args4j.
 */
public class Main
{
   @Option(name="-v", aliases="--verbose", usage="Print verbose status.")
   private boolean verbose;

   @Option(name="-f", aliases="--file", usage="Fully qualified path and name of file.", required=true)
   private String fileName;

   private void doMain(final String[] arguments) throws IOException
   {
      final CmdLineParser parser = new CmdLineParser(this);
      if (arguments.length < 1)
      {
         parser.printUsage(out);
         System.exit(-1);
      }
      try
      {
         parser.parseArgument(arguments);
      }
      catch (CmdLineException clEx)
      {
         out.println("ERROR: Unable to parse command-line options: " + clEx);
      }
      out.println("The file '" + fileName + "' was provided and verbosity is set to '" + verbose + "'.");
   }

   /**
    * Executable function demonstrating Args4j command-line processing.
    *
    * @param arguments Command-line arguments to be processed with Args4j.
    */
   public static void main(final String[] arguments)
   {
      final Main instance = new Main();
      try
      {
         instance.doMain(arguments);
      }
      catch (IOException ioEx)
      {
         out.println("ERROR: I/O Exception encountered: " + ioEx);
      }
   }
}

Here are some additional characteristics of args4j to consider when selecting a framework or library to help with command-line parsing in Java.

  • args4j is open source and licensed with the MIT License.
  • Current version of args4j (2.33) requires J2SE 5.
  • args4j does not require any third-party libraries to be downloaded or referenced separately.
  • The args4j 2.33 main JAR (args4j-2.33.jar) is approximately 152 KB in size.
  • The Maven Repository shows 376 dependencies on args4j including OpenJDK's JMH Core and Jenkins (not surprising given Kohsuke Kawaguchi's involvement in both).
  • args4j has been around for a while; its 2.0.3 release was in January 2006 and it's been around in some form since at least 2003.
  • args4j allows a command-line parameter to be excluded from the usage output via "hidden" on the @Option annotation.
  • args4j allows for relationships between command-line arguments to be specified and enforced. This includes the ability to specify when two arguments cannot be supplied at the same time ("forbids") and when the presence of an argument only makes sense when another argument is also provided ("depends").
  • args4j supports use of enum-typed class attributes for cases where a finite set of values is applicable to the option. The @Option documentation describes how to do this under the "Enum Switch" section.
  • args4j provides extensibility and customizability of command-line arguments parsing via its OptionHandler class.

The args4j library is easy to use and allows for highly readable code. Perhaps the biggest consideration when deciding whether to use args4j is deciding how comfortable one is with using annotations for specifying the command-line parameters' definitions.

Additional References

Tuesday, June 20, 2017

Java Command-Line Interfaces (Part 1): Apache Commons CLI

Although I typically use Groovy to write JVM-hosted scripts to be run from the command-line, there are times when I need to parse command-line parameters in Java applications and there is a plethora of libraries available for Java developers to use to parse command-line parameters. In this post, I look at one of the best known of these Java command line parsing libraries: Apache Commons CLI.

I have blogged on Apache Commons CLI before, but that post is over eight years old and describes Apache Commons CLI 1.1. Two classes that I demonstrated in that post, GnuParser and PosixParser, have since been deprecated. The examples in this current post are based on Apache Commons CLI 1.4 and use the newer DefaultParser that was introduced with CLI 1.3 to replace GnuParser and PosixParser.

The Apache Commons CLI documentation's "Introduction" explains how Commons CLI accomplishes the "three stages [of] command line processing" ("definition", "parsing", and "interrogation"). These three stages map in Commons CLI to classes Option and Options ("definition"), to interface CommandLineParser ("parsing"), and to class CommandLine ("interrogation").

For the examples built here with Apache Commons CLI, the expected command-line arguments are relatively simple. One argument is optional and, when specified, indicates that verbose output is enabled. The other argument is required and is used to specify a file to be processed by the imaginary application. The optional argument does not have a value associated with the flag and is expressed as -v or --verbose. The required argument should be followed by a value which is the path and name of the file. This flag is either -f or --file. The next code listing demonstrates using Commons CLI's Option.Builder (introduced with Commons CLI 1.3) to build up the expected options as part of the "definition" stage.

Example of Using Apache Commons CLI Option.Builder for "Definition Stage

/**
 * "Definition" stage of command-line parsing with Apache Commons CLI.
 * @return Definition of command-line options.
 */
private static Options generateOptions()
{
   final Option verboseOption = Option.builder("v")
      .required(false)
      .hasArg(false)
      .longOpt(VERBOSE_OPTION)
      .desc("Print status with verbosity.")
      .build();
   final Option fileOption = Option.builder("f")
      .required()
      .longOpt(FILE_OPTION)
      .hasArg()
      .desc("File to be processed.")
      .build();
   final Options options = new Options();
   options.addOption(verboseOption);
   options.addOption(fileOption);
   return options;
}

The "Builder" pattern implemented for Apache Commons CLI as shown in the above example features the benefits of the builder pattern such as creating an Option in a fully completed state in one statement and use of highly readable builder methods to set that instance's various fields. My older post on Apache Commons CLI demonstrates use of the alternate traditional constructor approach to instantiating Option instances.

With the command-line options defined, it's time to move to the "parsing" stage and the next code listing demonstrates how to parse with Apache Commons CLI by simply invoking the method CommandLinePaser.parse().

Parsing Command-line Options with Commons CLI

/**
 * "Parsing" stage of command-line processing demonstrated with
 * Apache Commons CLI.
 *
 * @param options Options from "definition" stage.
 * @param commandLineArguments Command-line arguments provided to application.
 * @return Instance of CommandLine as parsed from the provided Options and
 *    command line arguments; may be {@code null} if there is an exception
 *    encountered while attempting to parse the command line options.
 */
private static CommandLine generateCommandLine(
   final Options options, final String[] commandLineArguments)
{
   final CommandLineParser cmdLineParser = new DefaultParser();
   CommandLine commandLine = null;
   try
   {
      commandLine = cmdLineParser.parse(options, commandLineArguments);
   }
   catch (ParseException parseException)
   {
      out.println(
           "ERROR: Unable to parse command-line arguments "
         + Arrays.toString(commandLineArguments) + " due to: "
         + parseException);
   }
   return commandLine;
}

Note that this code using a newer version of Apache Commons CLI instantiates a DefaultParser for doing the parsing rather than a PosxParser or GnuParser as was done in the older code.

With the command-line objects defined and the command-line parsed, it is time for the interrogation stage. The next code listing demonstrates Apache Commons CLI's support for command-line interrogation.

Interrogating Command-line with Commons CLI

final boolean verbose =
   commandLine.hasOption(VERBOSE_OPTION);
final String fileName =
   commandLine.getOptionValue(FILE_OPTION);
out.println("The file '" + fileName + "' was provided and verbosity is set to '" + verbose + "'.");

The above code listing demonstrates use of CommandLine.hasOption() to determine if an option's particular flag is present without regard for whether a value is provided for that flag (appropriate for -v/--verbose in our example). Likewise, the code shows that CommandLine.getOptionValue() can be used to obtain the value associated with the the provided command-line flag (appropriate for the -f/--file option in our example).

The next screen snapshot demonstrates the output from the simple example whose code listings were shown above and they demonstrate the support for the verbosity and file path/location command-line options described above.

The second screen snapshot demonstrates Commons CLI' output when the command-line parameters don't include a required command-line argument.

A useful piece of functionality for any framework for building Java command-line parsing is the ability to support usage and help information. This is accomplished via Commons CLI's HelpFormatter. The next code listing demonstrates using HelpFormatter for printing help and usage information and the screen snapshot following the code listing demonstrates the appearance of the help and usage when employed.

Acquiring "usage" and "help" Details with Commons CLI

/**
 * Generate usage information with Apache Commons CLI.
 *
 * @param options Instance of Options to be used to prepare
 *    usage formatter.
 * @return HelpFormatter instance that can be used to print
 *    usage information.
 */
private static void printUsage(final Options options)
{
   final HelpFormatter formatter = new HelpFormatter();
   final String syntax = "Main";
   out.println("\n=====");
   out.println("USAGE");
   out.println("=====");
   final PrintWriter pw  = new PrintWriter(out);
   formatter.printUsage(pw, 80, syntax, options);
   pw.flush();
}

/**
 * Generate help information with Apache Commons CLI.
 *
 * @param options Instance of Options to be used to prepare
 *    help formatter.
 * @return HelpFormatter instance that can be used to print
 *    help information.
 */
private static void printHelp(final Options options)
{
   final HelpFormatter formatter = new HelpFormatter();
   final String syntax = "Main";
   final String usageHeader = "Example of Using Apache Commons CLI";
   final String usageFooter = "See http://marxsoftware.blogspot.com/ for further details.";
   out.println("\n====");
   out.println("HELP");
   out.println("====");
   formatter.printHelp(syntax, usageHeader, options, usageFooter);
}

This post has demonstrated using Apache Commons CLI to achieve some of the most common functionality related to command-line parsing in Java applications including option "definition", command-line arguments "parsing", "interrogation" of the parsed command-line arguments, and help/usage details related to the command-line arguments. Here are some additional characteristics of Apache Commons CLI to consider when selecting a framework or library to help with command-line parsing in Java.

  • Apache Commons CLI is open source and licensed with the Apache License, Version 2.0.
  • Current version of Apache Commons CLI (1.4) requires J2SE 5 or later.
  • Apache Commons CLI does not require any third-party libraries to be downloaded or referenced separately.
  • The Apache Commons CLI 1.4 main JAR (commons-cli-1.4.jar) is approximately 53 KB in size.
  • Apache Groovy provides out-of-the-box command-line parsing capabilities based on Apache Commons CLI via CliBuilder.
  • The Maven Repository shows almost 1800 dependencies on Apache Commons CLI including Apache Groovy.
  • Apache Commons CLI has been around for a while; its initial 1.0 release was in November 2002.
  • Commons CLI supports both long and short syntax for command-line arguments. Its main page lists support for these types of option formats (and that page includes examples of each style):
    • "POSIX like options" (single hyphen)
    • "GNU like long options" (double hyphen)
    • "Java like properties" (using -D)
    • "Short options with value attached"
    • "Long options with single hyphen"

For me, one of the biggest advantages of Apache Commons CLI when implementing command-line interfaces in simple Java applications is that I'm already familiar with Groovy's built-in use of CliBuilder. Because I use Groovy far more often for simple command-line based scripts and tools than I use Java, this Groovy familiarity with the basic Apache Commons CLI usage is helpful when moving back to Java.

Additional References

Monday, June 5, 2017

jhsdb: A New Tool for JDK 9

I like to use the command-line tools provided with the JDK in the early steps of analyzing performance and other issues with Java-based applications and have blogged on tools such as jcmd, jps, jstat, jinfo, jhat and jmap, jrunscript, jstack, and jdeps. JDK 9 is bringing new command-line tools with multiple tools specifically related to new JDK 9 features such as modularity (jlink and jmod) and enhanced deprecation (jdeprscan). In this post, I focus on a new command-line tool delivered with JDK 9 for dealing with performance and serviceability issues: jhsdb.

The jhsdb tool is described on its Oracle JDK 9 Documentation Early Access page, "You use the jhsdb tool to attach to a Java process or to launch a postmortem debugger to analyze the content of a core-dump from a crashed Java Virtual Machine (JVM)." The tool comes with several "modes" and several of these modes correspond in name and function with individual command-line tools available in previous JDK distributions. The jhsdb tool not only provides a single tool that encompasses functionality of multiple other tools, but it also provides a single, consistent approach to applying these different functions. For example, the jhsdb command-line syntax for getting help for each of the "modes" is identical.

The jhsdb tool can be attached and applied to a running JVM (including one that is hanging) via its process identifier (PID) similar to how several other tools (including jcmd) work. The jhsdb tool can also be used to analyze core information associated with a crashed JVM if the core file and executable are provided. As an example of the consistency jhsdb provides, all of its mode support the "common options" --pid (to specify target JVM's process ID), --exe (to specify target executable), --core (to specify target core dump file), and --help (to display options specific to each mode).

The next series of snapshots demonstrates use of the --help option with the main jhsdb command and with each of several of jhsdb's "modes." One observation that can be made is that the common options --pid, --core, and --exe are offered by all the modes. The obvious inference from this is that the specific functions supported in each mode are those other than those "common" options.

jhsdb "jstack" Mode

The --help for the jhsdb mode jstack has two specific functionality options: --locks and --mixed. These are demonstrated in the following two screen snapshots.

The screen snapshots just shown demonstrate that the jstack mode of the jhsdb tool provides us with deadlock detection details, information on thread locks, and an overview of the native frames and Java frames.

jhsdb "jmap" Mode

The --help for jhsdb mode jmap shows several functions supported by that mode. When jhsdb jmap is executed with only the --pid or only with the --exe/--core combination, the output is similar to that provided by the Linux pmap command.

As one would expect, the jmap mode of the jhsdb provides functions similar to those provided by the separate but similarly named jmap command. These include heap dump (--heap), class histogram (--histo), classloader statistics (--clstats), and finalizer information (--finalizerinfo) and are demonstrated in the following four screen snapshots.

jhsdb "jinfo" Mode

Not surprisingly, the jinfo mode of the jhsdb command provides functionality that overlaps with that provided by the jinfo command. Specifically, the jhsdb's jinfo mode allows one to see the targeted JVM's flags and system properties.

There are three main options used with jhsdb jinfo: --flags to see JVM flags, --sysprops to see the system properties, or no argument to see both the flags and the system properties. The next two screen snapshots demonstrate use of jhsdb jinfo --flags and jhsdb jinfo --sysprops. Running jhsdb jinfo without any arguments shows the system properties first followed by the flags, but is not shown here.

jhsdb "jsnap" Mode

The jhsdb mode jsnap provides access to information previously provided by the internal class sun.jvm.hotspot.tools.JSnap which was previously available in lib/sa-jdi.jar and which has been added to jhdsdb for JDK 9. There are two options for output from jhsdb jsnap based on whether no mode-specific argument is provided or if the --all mode-specific argument is provided. The next two screen snapshots demonstrate these two options.

These screenshots demonstrate that jhsdb jsnap with no mode-specific option lists information such as events related to threads and class loading/unloading along with core JVM properties. Adding the --all option lists these same properties, but in addition adds far more properties and, according to the documentation, "Prints all performance counters."

By the way, Marcus Hirt's Using the JVM Performance Counters provides an interesting look at how to apply JMX and custom MBeans to achieve a tool "similar to the PerformanceCounters MBean available in JRockit." I believe that jhsdb jsnap --all brings simple ability to see the same type of information in HotSpot as Hirt talked about being available in JRockit with jrcmd -l.

jhsdb Debug Modes

The three jhsdb modes hsdb (graphical user interface for interactive debugging), clhsdb (command-line interface for interactive debugging), and debugd (remote debug server) are related to debug operations. I may take a closer look at these modes in a future post, but for now I simply show some screen snapshots that demonstrate the graphical interaction using jhsdb hsdb. The GUI was started with jhsdb hsdb --pid <pid> and most of the options displayed here were run by selecting the specific menu option under "Tools".

As can be seen in the article HotSpot's Hidden Treasure, the serviceability debugger GUI has been available before JDK 9 and jhsdb, but this article also shows how much more difficult it was to find and start this tool before JDK 9's introduction of jhsdb.

Relationship of jhsdb to jcmd and to Other Command-line JDK Tools

I summarized the relationship of general-purpose tool jcmd to other JDK-provided command-line tools in the blog post jcmd: One JDK Command-Line Tool to Rule Them All. I adapt that table here to add jhsdb to the mix.

FunctionalityjhsdbjcmdSimilar Tool
Listing Java Processes N/A1 jcmd jps -lm
Heap Dumps jhsdb jmap --binaryheap jcmd <pid> GC.heap_dump jmap -dump <pid>
Heap Usage Histogram jhsdb jmap --histo jcmd <pid> GC.class_histogram jmap -histo <pid>
Thread Dump jhsdb jstack --locks
(subset of locked thread frames)
jcmd <pid> Thread.print jstack <pid>
List System Properties jhsdb jinfo --sysprops jcmd <pid> VM.system_properties jinfo -sysprops <pid>
List VM Flags jhsdb jinfo --flags jcmd <pid> VM.flags jinfo -flags <pid>

1 You use jcmd or jps -lm to identify PID upon which to have jhsdb, jcmd, and many other tools act if working against a running JVM. I used jcmd in this post to identify the PID, but the current jhsdb documentation demonstrates using jps to acquire the JVM PID.

The jhsdb tool is a mostly command-line tool that does also have an optional interactive GUI available that supports reporting of many of the commonly desired attributes of a JVM that is hung or has crashed. It provides a consistent interface across its modes and the command-line interaction allows for interactive help requests such that very little syntax must be known or remembered before applying the tool. If one can remember "jhsdb", one can start using the tool effectively. The jhsdb tool is new to JDK 9, but brings functionality into one tool that was previously available from several different tools.

Wednesday, May 17, 2017

Kotlin and Android: JetBrains and Google Behind One Language

Google I/O 2017 had several major announcements, but one of the most interesting to me is "first-class support for Kotlin" on Android.

The Kotlin blog post on this announcement discusses the benefits this brings to Kotlin users:

In case you are concerned about other platforms that Kotlin supports (Kotlin/JVM for server and desktop, Kotlin/JS and Kotlin/Native), please be sure that they are as important for us as ever. Our vision here is to make Kotlin a uniform tool for end-to-end development of various applications bridging multiple platforms with the same language. ... First-class support on Android will likely bring more users to Kotlin, and we expect the community to grow significantly. This means more libraries and tools developed in/for Kotlin, more experience shared, more Kotlin job offerings, more learning materials published, and so on. We are excited to see the Kotlin ecosystem flourish!

An interesting point related to and included in this announcement is mentioned in the Kotlin on Android FAQ:

What's the future of Kotlin? JetBrains' thoughtful work on Kotlin's design is one of the reasons we're embracing the language. Google is partnering with JetBrains to ensure a wonderful overall developer story—from language, to framework, to tools. And, we are excited to be working together to move the Kotlin language into a non profit foundation.

The Kotlin blog post describes the future of Kotlin with Android in greater detail:

We will be partnering with Google to create a non-profit foundation for Kotlin. Language development will continue to be sponsored by JetBrains, and the Kotlin team (over 40 people and second largest team at the company) will operate as usual. Andrey Breslav remains the Lead Language Designer, and Kotlin will be developed under the same principles as before.

Android 3.0 Canary 1 (preview) can currently be downloaded at https://developer.android.com/studio/preview/index.html at includes Kotlin support among its new features that include Java 8 support.

In the 2016 edition of my annual post on significant software developments of the year, I listed Kotlin-related developments in the "Honorable Mention" section. With this announcement of Kotlin being officially supported on Android and the language being placed in nonprofit foundation and enjoying support of Google and JetBrains, I think it's very possible that Kotlin will make my top ten list in 2017 for significant software language development developments. Besides these announcements, Kotlin has already seen JavaScript support released in Kotlin 1.1 in 2017 and a preview of Kotlin/Native.

Monday, May 15, 2017

JVM Statistics with jstat

I have written about several command-line tools provided with the Oracle and/or OpenJDK Java Development Kits (JDKs) in the past, but I've never written exclusively about the jstat tool. The Oracle JDK 9 Documentation Early Access states that jstat is used "to monitor Java Virtual Machine (JVM) statistics." There is also a warning, "This command is experimental and unsupported." Although I quoted the JDK 9 documentation, jstat has been a part of the Sun/Oracle JDK in some form (known at one time as jvmstat) in Java SE 8, Java SE 7, Java SE 6, and J2SE 5. Instrumentation for the HotSpot JVM was introduced with Java 1.4.1 (only enabled when -XX:+UsePerfData was set) and has provided "always-on instrumentation" since Java 1.4.2.

Much of the information that jstat provides can be gleaned from visual tools such as VisualVM, JMX and platform MBeans, garbage collection logs or via JVM options. However, jstat provides advantages when compared to each of these alternatives. Its advantages include those common to command-line tools such as the ability to execute from scripts and run without need of developers or others being present. It also is useful to be able to apply jstat to an already running Java process to start monitoring its JVM statistics rather than having the specify the monitoring of those options when starting the JVM.

For my examples in this post, I am using Oracle JDK 9 build 164. The next screen snapshot shows this version and also shows one of the first flags to apply when starting to use jstat: the -options flag.

As demonstrated in the screen snapshot, and as stated in the jstat documentation, jstat -options is used to "display the list of options for a particular platform installation." In my example being shown here, the following options are available:

  • -class
  • -compiler
  • -gc
  • -gccapacity
  • -gccause
  • -gcmetacapacity
  • -gcnew
  • -gcnewcapacity
  • -gcold
  • -gcoldcapacity
  • -gcutil
  • -printcompilation

I'll only be looking at a small subset of these available options in this post, but the jstat documentation provides a single sentence describing each jstat option and the command-line use of each option is very similar to all other options. In fact, once one learns a few minor things regarding use of jstat, the execution of various options becomes the easy part. The difficult part of using jstat is often interpreting the data provided by jstat.

The jstat -help option prints a simple usage as shown in the next screen snapshot.

From the jstat usage message, we learn that the jstat command-line tool is executed by running the name of the command first (jstat) with the hyphenated option name next, followed by the optional -t and/or -h flags, followed by a vimid, and concluding with an optional interval and optional count of the number of times to execute the command on the provided interval. Examples are clearer than descriptive text and some examples are shown in this post and in the jstat documentation.

For monitoring "local" JVM statistics, the vmid is simply the process ID of the JVM process. This is the same PID returned by the hip jcmd (or stodgy jps) for Java processes. The next screen snapshot demonstrates use of jcmd to identify the PID (8728 in this case) of the Java application I'm monitoring in my examples (JEdit in this case).

The "Virtual Machine Identifier" section of the jstat documentation provides significantly more details regarding the vmid because a more complicated vmid (for remote monitoring of JVM statistics) can include a protocol, the local targeted machine's vmid, host, and port. Although all of my examples in this post will use jstat and a simple Java PID (vmid), the jstat documentation does provide examples of use of a more detailed vmid for remote monitoring of JVM statistics.

For the remaining examples in this post, I wanted a Java application that was a bit more interesting from a JVM statistics monitoring perspective than JEdit is. I decided to use the "PigInThePython" sample application from Nikita Salnikov-Tarnovski's post "Garbage Collection: increasing the throughput" on the highly recommended Plumbr Blog. See that post if you're interested in seeing the source code for PigInThePython.

For my first example of using jstat, I use one of its most commonly used options: -gcutil. Besides demonstrating the -gcutil option, I'll use this first example to also demonstrate and explain the jstat output options that generally apply to other jstat options besides gcutil.

The following screen snapshot demonstrates use of jcmd to acquire the PID of the PigInThePython application (5096 in this case) and running of the simplest form of jstat -gcutil.

In its simplest form (with no other options), jstat -gcutil displays a single line of output with no timestamp. The column headings are described in the jstat documentation section "-gcutil option" which also describes the -gcutil option as, "Summary of garbage collection statistics." This documentation explains, for example, that several of the columns indicate percentages of usage of different spaces' allocations while other columns indicate number of garbage collection events and total garbage collection time.

We often want to associate the statistics that jstat provides with the time that other events are occurring in the monitored system to identify correlations between those events and effects on the JVM. The jstat -t option will prepend a timestamp at the beginning of the output. This timestamp is the number of seconds since the JVM that is being monitored was started. Although this is not as convenient for humans to read as other formats, it does make it possible to correlate the JVM statistics with timeframes in which the JVM has been running and with garbage collection logs that had time stamps included. The next screen snapshot demonstrates -t in action:

It is typically useful to monitor JVM statistics such as those presented by jstat -gcutil more than once. The next screen snapshot demonstrates use of a specified interval (100 milliseconds intervals as specified with 100ms to have these results captured and shown every 100 milliseconds.

The output in the last screen snapshot never repeats the header with the column acronyms after showing it the first time. If one wants to have that header repeated after a certain number of lines to make it easier to know which numbers belong with which columns much farther down in the output, the -h option can be used to specify the number of results after which the column headers are displayed again. In the next screen snapshot, -h20 is used to see the header after every 20 rows.

There may be times when it's desirable to have jstat provide its data every so often and only for a certain number of times. The interval allows one to specify the duration of time between the results and any integer following that interval specification acts as the limit on the total number of times the results will be displayed. In the next screen snapshot, the 15 at the end of the command limits the output to 15 total rows.

The jstat -gccause option returns the same information as -gcutil, but also adds information about what caused the monitored garbage collections. The following screen snapshot demonstrates this.

In the above screen snapshot, we see that the "cause of last garbage collection" (LGCC) was "G1 Humongous Allocation" and that "cause of current garbage collection" (GCC) is "No GC" (there is no garbage collection currently underway).

The next screen snapshot demonstrates use of jstat -class to see "Class loader statistics" number of classes loaded ("Loaded"), number of kilobytes loaded (first "Bytes"), number of classes unloaded ("Unloaded"), and number of kilobytes unloaded (second "Bytes"), and "time spent performing class loading and unloading operations" ("Time").

The command jstat -printcompilation indicates "Java HotSpot VM compiler method statistics" and is demonstrated in the next screen snapshot.

The -printcompilation option displays columns "Compiled" ("Number of compilation tasks performed by the most recently compiled method"), "Size" ("Number of bytes of byte code of the most recently compiled method"), "Type" ("Compilation type of the most recently compiled method"), and "Method" (name of class/method of most recently compiled method expressed in format consistent with HotSpot VM option -XX:+PrintCompilation).

The jstat -compiler command allows us to see "Java HotSpot VM Just-in-Time compiler statistics" such as number of compilation tasks performed ("Compiled"), number of failed compilation tasks ("Failed"), number of invalidated compilation tasks ("Invalid"), time spent on compilation ("Time"), type and class/method name of last failed compilation ("FailedType" and "FailedMethod"). This is demonstrated in the next screen snapshot.

There are several more options available with jstat and most of them are specific to different perspectives on garbage collection in the monitored JVM.

The jstat documentation warns that the tool is experimental and subject to change or removal in future versions of the JDK. The documentation also warns against writing scripts and tools that parse the output of jstat as the output content or format might change in the future. However, I can see how some might take the risk and write parsing code because this command-line tool is easily accessible from scripts and the parsing code needed would not be very complex.

This post has been an introduction to jstat, but there is much more to learn about the tool. Interpretation of the tool's results is more complex than using the tool and the analysis of the data provided by jstat is likely to be more challenging than the effort to collect the numbers. Several additional resources are listed below to provide more information on the collecting of and analyzing of Java Virtual Machine statistics with the jstat tool.

Additional jstat Resources

Wednesday, May 10, 2017

Java Platform Module System Public Review Fails to Pass

There has been an unusual level of drama, intrigue, and politics in the world of Java over the past few weeks that culminated in this week's JSR 376 Java Platform Module System Public Review Ballot. Java modularity [including the Java Platform Module System (JPMS)] has been arguably the most significant piece of JDK 9 and so it's not surprising that it has received so much attention. In addition to the typical publicly available mailing list traffic, there have been blog posts and open letters to further advertise the contention and debate surrounding JPMS (JSR 376), described as "a central component of Project Jigsaw."

The final vote was, according to the JSR #376 Java Platform Module System Public Review Ballot page, 10 votes for and 13 votes against, so "the EC has not approved this ballot." The comments that go along with the votes in the text area at the bottom of the ballot page are telling. In particular, I thought it interesting how many reviewers vote no mainly because they were uncomfortable with the other, more vocal reviewers not approving. It's also interesting that the Public Review Ballot for JSR #379 Java SE 9 Release Contents (an "umbrella" JSR) passed overwhelmingly on the same day this one failed.

It will be interesting to see how this continues to unfold over the coming days and weeks and what impact is has on the release date for JDK 9. Rather than rehash the arguments on both sides, I reference posts from key contributors to the discussion below.

References: Executive Committee Participants/Representatives

References: Opinions/Forums

References: Other Overviews

References: Since the Vote

Since the Vote: Java Platform Module System (JSR 376) EG minutes

Monday, May 8, 2017

Java's Observer and Observable are Deprecated in JDK 9

In the blog post Applying JDK 9 @Deprecated Enhancements, I discussed additions of the optional elements (methods) forRemoval() and since() to the @Deprecated annotation in JDK 9. I stated in that post, "The application of new JDK 9 @Deprecated methods on the Java SE API can also be instructive in how they are intended to be used." In this post, I look at the application of the enhanced @Deprecated annotation to the JDK class java.util.Observable.

The class java.util.Observable has been with us almost since the beginning (since Java 1.0). As of JDK 9, however, it will be marked as deprecated. The following screen snapshot shows a portion of this class's Javadoc representation in a web browser.

This is an example of a class that in the category "Deprecated With No Plans for Removal" described in my previous post. The presence of since() provides information on when it was deprecated (JDK 9) and the absence of forRemoval() indicates lack of concrete plans to actually remove the class. The java.util.Observer interface has also been deprecated in a similar manner and its documentation references the documentation for the Observable class.

Not only does the Observable documentation relay when it was deprecated, but it also documents the problems with Observable that make deprecation desirable and provides important information on alternatives that might be used instead of Observable:

This class and the Observer interface have been deprecated. The event model supported by Observer and Observable is quite limited, the order of notifications delivered by Observable is unspecified, and state changes are not in one-for-one correspondence with notifications. For a richer event model, consider using the java.beans package. For reliable and ordered messaging among threads, consider using one of the concurrent data structures in the java.util.concurrent package. For reactive streams style programming, see the Flow API.

This is a good example of how Java developers can use the Javadoc tag @deprecated to provide much deeper detail related to deprecation than can be provided even with the enhanced @Deprecated annotation. JEP 277 ("Enhanced Deprecation") explicitly listed unifying of Javadoc tag @deprecated and annotation @Deprecated as a "non-goal": "It is not a goal of this project to unify the @deprecated Javadoc tag with the @Deprecated annotation."

Additional details justifying the deprecation of Observable and Observer can be found in JDK-8154801 ("deprecate Observer and Observable"). There is a quote in there from Josh Bloch as part of JDK-4180466 ("Why is java.util.Observable class not serializable.") dated February 1999:

This class is no longer under active development. It is largely unused in the JDK, and has, for the most part, been superseded by the 1.1 Beans/AWT event model. ... Observable has fallen into disuse and is no longer under active development.

For the most part, it seems that Observer and Observable are not used much, so deprecation shouldn't be much of an issue, especially given no definitive plans to remove these altogether.

Monday, April 24, 2017

Java and Docker: Now and the Future

Sharat Chander's blog post Official Docker Image for Oracle Java and the OpenJDK Roadmap for Containers provides a high-level overview of the "Official Docker Image for Oracle Java," an introduction to Docker and why containers like Docker are desirable, and a peek at things to come for Java on Docker. The post also provides a link to the "Docker image for Oracle Server JRE" that "is now available on Docker Store."

Chander briefly discusses attempts to set "up a consistent, reproducible environment that scales to thousands/millions of instances" for the cloud and how operating system tools and hardware virtualization have been used in this way. He then introduces Docker as another approach and provides a brief description of the benefits of Docker.

Chander's post introduces Alpine Linux and quotes their web page, "Small. Simple. Secure. Alpine Linux is a security-oriented, lightweight Linux distribution based on musl libc and busybox." Chander associates Alpine Linux's use of musl libc (described as "a new general-purpose implementation of the C library" that "is lightweight, fast, simple, free and aims to be correct in the sense of standards-conformance and safety") and busybox (combination of "tiny versions of many common UNIX utilities into a single small executable") to Java and paraphrases the message list post announcing Project Portola: "The goal of OpenJDK 'Project Portola' is to provide a port of the JDK to Alpine Linux, and in-particular the 'musl' C library."

The FAQ section of Chander's post has some interesting questions and answers. For example, there is information on Oracle's "recommendations for running Oracle Java SE on Docker" available on GitHub. There is information on Java 8 Update 131 enhancements that enable "better memory and processor integration between Java and Docker" and it's interesting to note that in "JDK 9 the JVM will be container-aware" instead of "the thread and memory settings [coming] from the host OS."

The OpenJDK Docker Repository is available at https://hub.docker.com/_/openjdk/.

It's interesting to see what is being done and what's planned for using Java with Docker.

Tuesday, April 18, 2017

Oracle JDK 9 Early Access Documentation Updated

Raymond Gallardo's 4 April 2017 post Early Access documentation for Oracle JDK 9 has been updated today announces updates to the Oracle JDK9 Documentation Early Access page. Gallardo highlights a few of the updated sections including What's New in Oracle JDK 9, Oracle JDK 9 Migration Guide, HotSpot Virtual Machine Garbage Collection Tuning Guide (including Garbage-First Garbage Collector Tuning), javapackager tool for "packaging Java and JavaFX applications," and XML Catalog API.

The Main Tools to Create and Build Applications section features information on jlink (JEP 282), jmod (create and show contents of Project Jigsaw JMOD files), and jdeprscan (static analysis tool that scans ... for uses of deprecated API elements). There is also information on jhsdb for obtaining "specific information from a hanging or crashed JVM."

I summarized some of the tools and features being removed from the JDK with Java 9 in the post JDK 9 is the End of the Road for Some Features. Some of these are spelled out with additional details in the section Removed Tools. The Changes to Garbage Collection section similarly shows changes and removals in Java 9 of options and commands related to garbage collection. The section Changes to the Installed JDK/JRE Image outlines removal of things such as rt.jar and tools.jar, the extension mechanism, and the endorsed standards override mechanism.

The section Enable Logging with the JVM Unified Logging Framework demonstrates how to "use -Xlog option to configure or enable logging with the Java Virtual Machine (JVM) unified logging framework." The Changes to GC Log Output section explains how to use this -Xlog with gc (-Xlog:gc) for logging related to garbage collection and points out that "the -XX:+PrintGCDetails and -XX:+PrintGC options have been deprecated."

I previously blogged on the future of the Concurrent-Mark-Sweep garbage collector. The Java SE 9 HotSpot Virtual Machine Garbage Collection Tuning Guide states, "The CMS collector is deprecated. Strongly consider using the Garbage-First collector instead."

The main Oracle JDK 9 Documentation Early Access page is at http://docs.oracle.com/javase/9/index.html and its Developer Guides link provides direct references to guides such as Troubleshooting Guide, HotSpot Virtual Machine Garbage Collection Tuning Guide, Java Scripting Programming Guide, Java Language Updates, and Migration Guide.

Monday, April 17, 2017

Implications of the Presence of StringBuffer

When I am working on legacy code and run across instances of StringBuffer, I typically replace them with instances of StringBuilder. Although a performance advantage can be gained from this change, I often change it in places I know will have little noticeable effect in terms of performance. I feel it's worth making the change for a variety of reasons in addition to the potential for performance benefit. There's rarely a reason to not choose StringBuilder over StringBuffer (API expectations are the most common exception) and the existence of StringBuffer in code misleads and provides a bad example to those new to Java.

In the book The Pragmatic Programmer: From Journeyman to Master, Andy Hunt and David Thomas discuss "the importance of fixing the small problems in your code, the 'broken windows'." Jeff Atwood touched on this subject in the post The Broken Window Theory and it has been more recently addressed in the posts Software Rot, Entropy and the Broken Window Theory and Don't leave broken windows. The presence of StringBuffer implies a staleness in the code. In effect, use of StringBuffer may not be a "broken window," but it's a really old, leaky single-pane window that should be replaced with a modern, energy-efficient double-pane window.

I found Peter Lawrey's recent blog post StringBuffer, and how hard it is to get rid of legacy code to be an interesting take on other implications of StringBuffer that still exist in code. Lawrey quotes the last paragraph of the StringBuffer class Javadoc documentation, "As of release JDK 5, this class has been supplemented with an equivalent class designed for use by a single thread, StringBuilder. The StringBuilder class should generally be used in preference to this one, as it supports all of the same operations but it is faster, as it performs no synchronization." Lawrey then uses simple Java methods and jmap to demonstrate that instances of StringBuffer are still used in classes and libraries delivered with the JDK even as late as Java 8.

Lawrey points out that the presence of StringBuffer in frequently used Java code more than a decade after the introduction of "drop-in replacement" StringBuilder is evidence of how difficult it is to "clean up legacy code." Lawrey's full conclusion states, "Using StringBuffer on startup doesn’t make much difference, but given it has a well known, drop in replacement, and it is still used, even in new functionality more than ten years later shows how hard it can be to clean up legacy code or to change thinking to get people to use best practice libraries."

I decided to try out one of Lawrey's simplest examples when compiled with Java 8 Update 121 and when compiled with a recent release of OpenJDK 9. I (slightly) adapted Lawrey's example to the simple "Main" class listing shown next.

Main.java

import java.io.IOException;

/**
 * (Slightly) adapted class from blog post
 * "StringBuffer, and how hard it is to get rid of legacy code" at
 * https://vanilla-java.github.io/2017/04/13/String-Buffer-and-how-hard-it-is-to-get-rid-of-legacy-code.html
 */
public class Main
{
   /**
    * Main function that instantiates this Java "application" and does nothing
    * else until "ENTER" is pressed.
    */
   public static void main(final String[] args) throws IOException
   {
      System.out.println("Waiting [press ENTER to exit] ..");
      System.in.read();
   }
}

The following screen snapshot shows the output of using jcmd with its -all option (includes unreachable objects in the inspection) to show the number of instances of StringBuffer and StringBuilder in the simple Java application when compiled and run against three different versions of Java (Java 8 Update 102, Java 8 Update 121, and OpenJDK 9.0 ea+164). The execution of jcmd is performed in PowerShell and so Select-String is used similarly to how grep is used in Linux.

Although the versions of the class compiled and executed with versions of Java 8 had instances of StringBuffer, the version compiled with and executed against Java 9 only had instances of StringBuilder. It looks like the resolution of JDK-8041679 ("Replace uses of StringBuffer with StringBuilder within core library classes") and JDK-8043342 ("Replace uses of StringBuffer with StringBuilder within crypto code") have had their intended effect.