Starfish v0.3.0 Tutorial: Job History Collection

Sample files

Here is an example directory that contains the job histories for two jobs: download. Unzip them and try the Analyzer:)

What are job history files?

There are two files per job, which are created when the job is launched:

  • The Job Configuration XML file: it contains the job configuration as specified when the job is launched, i.e. it contains the merged defaults with the user settings for the job that override the defaults.

    File name example: job_201209062254_0003_conf.xml
  • The Job Status File: it is created when the job starts but continually written to until the job finishes. It contains what is also visible after a job has finished, i.e. the counters, status, start/stop time, task attempt details etc.

    File name example: job_201209062254_0003_1346981800709_hadoop_TeraSort

The two files are usually stored as pairs unless noted otherwise. Please see the Cloudera blog post for more details.

How to collect these files?

  • From the master node: If you have access to the master node, then you can easily collect the job history files for completed jobs.

    • First check the directory specified by hadoop.job.history.location, which defaults to the JobTracker's local directory ${hadoop.log.dir}/history.

      Example: /Users/jay23jack/hadoop-1.0.3/logs/history/
    • Otherwise, check the directory specified by mapred.job.tracker.history.completed.location, which defaults to the JobTracker's local directory ${hadoop.log.dir}/history/done. Files are organized by the date.

      Example: /Users/jay23jack/hadoop-1.0.3/logs/history/done/version-1/localhost_1353818432170_/2012/11/25/000000
    • Note: On Elastic MapReduce, the job history files are stored under /mnt/var/log/hadoop/history on the master node.
  • From the client node

    • First check with your Hadoop admins whether those files are kept in a publicly accessible place, like HDFS.
    • Otherwise, specify hadoop.job.history.user.location to either a local directory e.g. file:///tmp/history, or a HDFS directory e.g. /tmp/history, and you'll find a new directory _log under it after the job completes.

      If this parameter is left blank (by default), then it equals to the job's output directory. Note if it is set to none, then the job histories are not collected to anywhere outside the master node.

How to use Starfish to analyze these files

Once you collect the job history files on your desktop, put all of them under a history directory, and pass the parent directory to Starfish visualizer. For example, if all files are under /Users/jay23jack/history, then you need to enter /Users/jay23jack to the Starfish visualizer.