Kastang Ramblings of a Geek

31Dec/100

Parsing the WoW Armory without XML

A month or so ago Blizzard moved the WoW Armory to Battle.net servers. Currently, the new WoW Armory does not offer XML feeds for the data.  I spent a few hours working with PHP and DOM to create a 'parser' for the new Armory.  The below script is a trimmed down version of what is currently being used for the We Know Roster. I am only providing the back end script that will do the parsing and store the information in a MySQL database. Front end displaying can easily be achieved by querying the database with the stored results.

The script will pull the following information for each member in a specified guild: Name, Level, Class, Rank, Achievement Points, Profession 1 Name+Level, Profession 2 Name+Level, Talent1, and Talent2.

The scripts below require modifications to work properly. I recommend having knowledge of PHP/CLI before working with this script. I will develop a more user friendly version of this script only if Blizzard does not supply useful XML or JSON feeds in a reasonable amount of time.

The Bash Script:

The bash script pulls the newest HTML Roster file from the new Armory. This could probably be pulled via the PHP script, but since the file is several thousand lines long, I found it more efficient to save the file first and read it locally.

Please pay special attention to the paths, they will need to be altered in order to work correctly.

#!/bin/bash
#Replace YOUR_GUILD_NAME_HERE with your guild name. If your Guild Name is two or more words, it should be in the format
#of Your%20Guild%20Name
wget --directory-prefix=/path/to/your/desired/directory/ http://us.battle.net/wow/en/guild/YOUR_SERVER_HERE/YOUR_GUILD_NAME_HERE/roster
mv /path/to/your/desired/directory/roster /path/to/your/desired/directory/roster.html
php /path/to/php/file/ParseRoster.php

The SQL Dump:

Import this into a MySQL database.

--
-- Table structure for table `roster`
--
 
CREATE TABLE IF NOT EXISTS `roster` (
  `id` INT(11) NOT NULL AUTO_INCREMENT,
  `name` VARCHAR(255) NOT NULL,
  `race` VARCHAR(255) NOT NULL,
  `class` VARCHAR(255) NOT NULL,
  `level` VARCHAR(255) NOT NULL,
  `rank` VARCHAR(255) NOT NULL,
  `ap` VARCHAR(255) NOT NULL,
  `prof1name` VARCHAR(255) DEFAULT NULL,
  `prof1value` VARCHAR(255) DEFAULT NULL,
  `prof2name` VARCHAR(255) DEFAULT NULL,
  `prof2value` VARCHAR(255) DEFAULT NULL,
  `talent1` VARCHAR(255) DEFAULT NULL,
  `talent2` VARCHAR(255) DEFAULT NULL,
  PRIMARY KEY (`id`)
) ENGINE=MyISAM  DEFAULT CHARSET=latin1 AUTO_INCREMENT=1 ;

The PHP Backend:

The PHP file should be fairly straight forward.

A few notes:

  • I have a config file that holds information for my database, if you have a similar file you should include it, otherwise add in the proper mysql_connect() information.
  • Make sure the path to the Roster.html file is correct.
 
< ?php
/**
 * This script will parse the new WoW Armory without an XML file.
 * This script will currently pull the Name, Level, Class, Race,
 * Achievement Points, Professions, and Talents of every member
 * In a specified guild. The script works for me but may not work
 * as expected on every system. Use at your own risk.
 *
 * @author Josh Grochowski (josh[dot]kastang[at]gmail[dot]com)
 *
 */
 
set_time_limit(8000);
 
include("/path/to/config/file.php");
 
getRosterInformation();
 
function getRosterInformation() {
 
    $roster = file_get_contents("/path/to/roster/file/roster.html");
 
    $dom = new domDocument;
    $dom->loadHTML($roster);
    $dom->preserveWhiteSpace = false;
 
    //The first tbody tag marks the start of the actual
    //'roster' part of the html.
    $roster = $dom->getElementsByTagName('tbody');
 
    //Each Character has its own tr block.
    $char = $roster->item(0)->getElementsByTagName('tr');
 
    foreach ($char as $c) {
 
        //Character information is split into individual
        //td blocks.
        $charInfo = $c->getElementsByTagName('td');
        $charImages = $c->getElementsByTagName('img');
 
        //I only care about active characters. Inactive characters
        //will display 0 Achievement points.
        if((int)$charInfo->item(5)->nodeValue > 0) {
 
                $name = $charInfo->item(0)->nodeValue;
                $race = $charImages->item(0)->getAttribute('src');
                $class = $charImages->item(1)->getAttribute('src');
                $level = $charInfo->item(3)->nodeValue;
                $rank = trim($charInfo->item(4)->nodeValue);
                $ap = trim($charInfo->item(5)->nodeValue);
 
                //Returns an array containing the professions name/level and
                //talents of each individual character.
                $charArray = getCharacterInformation($name);
                $query = "INSERT INTO roster(name,race,class,level,rank,ap,prof1name,prof1value,prof2name,prof2value,talent1,talent2)
                                VALUES('$name','$race','$class','$level','$rank','$ap','$charArray[profName1]','$charArray[profValue1]',
                                '$charArray[profName2]','$charArray[profValue2]','$charArray[talent1]','$charArray[talent2]')";
 
                mysql_query($query) or die(mysql_error());
 
                //Wait 5 seconds inbetween queries to keep from getting banned from WoW Armory servers.
                //This can probably be adjusted to three or four seconds, but if you do get banned, it can
                //last las long as 48 hours.
                sleep(5);
        }
    }
}
 
function getCharacterInformation($charName) {
 
    //link to characters page on WoW Armory
    $charInfo = file_get_contents("http://us.battle.net/wow/en/character/eitrigg/".$charName."/simple");
 
    $dom = new domDocument;
    $dom->loadHTML($charInfo);
    $dom->preserveWhiteSpace = false;
 
    //Profession Names
    $xpath = new DOMXPath($dom);
    $profName = $xpath->query('//span[@class="profession-details"]/span[@class="name"]');
 
    //Profession Values
    $profValue = $xpath->query('//span[@class="profession-details"]/span[@class="value"]');
 
    //Talents
    $talents = $xpath->query('//span[@class="name-build"]/span[@class="name"]');
 
    $charArray = array("profName1" => $profName->item(0)->nodeValue,
                        "profValue1" => $profValue->item(0)->nodeValue,
                        "profName2" => $profName->item(1)->nodeValue,
                        "profValue2" => $profValue->item(1)->nodeValue,
                        "talent1" => $talents->item(0)->nodeValue,
                        "talent2" => $talents->item(1)->nodeValue);
 
    return $charArray;
}
 
?>
4Dec/100

Using Python to Parse WoW Server XML.

THIS CODE IS OBSOLETE. Blizzard changed the way information can be grabbed from their servers. Please view my PHP Class which will work.

The below Python script will parse the WoW Server XML file and return status, type, name, and load about a specified WoW Server. It also stores the Server XML file into a cache.txt file so constant requests to the script will not result in an IP ban on Blizzards servers.

Python 2.6+ must be used for this script to function correctly. It will work out of the box with no third party libraries required.

Please change the SERVER_NAME (on line 32) variable to your servers name.

#!/usr/bin/python
 
"""
Author: Josh Grochowski (Kastang)
 
This script will parse the WoW Server Status XML file and return
information about a specified server. This Python script works
for me, but it may not work for you. Use this script at your
own risk. 
 
Python 2.6+ must be used.
"""
 
"""
The below four lines assume this script will be loaded onto a
webserver. If you are planning to run this script in a CLI
enviornment, feel free to remove the below four lines of code.
"""
import cgitb
cgitb.enable()
print "Content-Type: text/html;charset=utf-8"
print
 
#Imports
from xml.etree.ElementTree import parse
import urllib, os, time
 
#WoW Server Status Path
WOW_XML = "http://www.worldofwarcraft.com/realmstatus/status.xml"
 
#Change SERVER_NAME to the your server.
SERVER_NAME = "Eitrigg"
 
#Display information for the Realm information.
rStatus = ["Offline", "Online"]
rType = ["", "PVE", "PVP", "RP", "RP PVP"]
rLoad = ["", "Low", "Medium", "High"]
 
#Modification time of the cache file.
modTime =  os.stat("cache.txt").st_mtime
 
#Current system time.
currTime =  time.time()
 
"""
If this function is called, the cache file will be
updated with the newest information from the WoW
Server Status XML file.
"""
def updateCache():
    cache = open("cache.txt", "w")
    cache.writelines(urllib.urlopen(WOW_XML))
 
"""
Given a Realm name and XML file, it will find the
XML string containing information about the specified
Realm and return it to the requesting function.
"""
def findRealm(realm, xml):
        for l in xml.findall('rs/r'):
                if l.get('n') == realm:
                        return l
 
        return False
 
#Displays information about the specified Server.
def printOutput(server):
    print "Realm Name: " + server.get('n')
    print "Realm Type: " + rType[int(server.get('t'))]
    print "Realm Status: " + rStatus[int(server.get('s'))]
    print "Realm Load: " + rLoad[int(server.get('l'))]
 
"""
If the cache file has not been updated for 10 minutes,
it will be updated. Otherwise, the existing file will
be used to find current information about the specified
realm. 
 
If the specified Realm is not a valid WoW Realm, nothing
will be returned. 
 
"""
def main():
    if (currTime - modTime) &gt; 600:
        updateCache()
 
    xml = parse("cache.txt").getroot()
    server = findRealm(SERVER_NAME, xml)
 
    if server != False:
        printOutput(server)
 
if __name__ == "__main__":
    main()

A sample output can be seen below:

# python realmstatus.py
Realm Name: Eitrigg
Realm Type: PVE
Realm Status: Online
Realm Load: High
Tagged as: , , No Comments
21Aug/100

HowTo: AWStats and phpMyAdmin to Display Properly Using Nginx and Apache

I am now using Nginx to handle the static content on my server while using Apache to handle the dynamic content. Overall I am very happy with the switch. One issue I had was the images in AWStats and phpMyAdmin were not displaying because the static file regular expression I am using was overriding my /awstats location rule. The fix is quite easy. A new rule needs to be setup to override the original static file regular expression.

This code will only work if you are using Nginx to forward non static requests to Apache.

 location ~* ^/(awstats|phpmyadmin) {
                proxy_pass http://127.0.0.1:8080;
                include /etc/nginx/proxy.conf;
        }

If you are already setup using Nginx + Apache, chances are you have a file similar to this below. If you have the proxy configurations copied and pasted into your files I would recommend making an includes file to keep your configuration files clean and readable.

proxy_redirect off;
 
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
 
client_max_body_size 10m;
client_body_buffer_size 128k;
 
proxy_connect_timeout 90;
proxy_send_timeout 90;
proxy_read_timeout 90;
 
proxy_buffer_size 32k;
proxy_buffers 8 16k;
proxy_busy_buffers_size 64k;
proxy_temp_file_write_size 64k;

Quite simple to accomplish, though I will admit it took me a few minutes to figure out. Hope this helps.

13Aug/100

Push Daily Weather to your iPhone using Prowl

Below is a PHP script that will pull your daily local weather forecast from Weather Bug and push it to your iPhone using Prowl. One requirement is you have the Prowl iPhone app on your iPhone or iTouch. Prowl is $2.99 in the App Store. This script also requires PHP to be installed on your server/desktop machine. I have tested this script with a handful of random Zip Codes around the US. If I happen to come up with any special conditions, I will update the script below.

Before running this script, make sure you change the XXXXX's in $xml to your local zip code and change YOUR_API_KEY_HERE in $prowl to your Prowl API code. You will also need to change includes('API/ProwlPHP') to the location of ProwlPHP on your server.

 
/**
 * This script pulls a local XML weather forcast from
 * Weatherbug. Pushes the current days weather to a
 * device running prowl.
 *
 * This script requires ProwlPHP, and a valid Prowl API code.
 *
 * @author Kastang (josh at kastang.com)
 */
    include('API/ProwlPHP.php');
 
    //Change the XXXXX's to your local zip code.
    $xml = simplexml_load_file("http://feeds.weatherbug.com/rss.aspx?zipcode=XXXXX&feed=fcst");
 
    //Parse the description from Weatherbug.
    prowl(substr((ltrim(strip_tags($xml->channel->item[0]->description))),0,-22));
 
    //Push the weather to Prowl.
    function prowl($weather) {
        $prowl =new Prowl('YOUR_API_KEY_HERE');
 
        $prowl->push(array(
                        'application'=>'Weather',
                        'event'=>'Today',
                        'description'=>$weather,
                        'priority'=>0,
                    ),true);
    }

I have this setup daily to run at 8am using cron. Below is an example you should add to crontab if you want to do the same thing:

0 8 * * * /path/to/weather.php
11Aug/100

Push Twitter Replies to your iPhone using Prowl

Yesterday I bought Prowl for the iPhone. Prowl is similar to Growl on OSX systems. Prowl pushes notifications to your iPhone or iTouch. One of the first uses I thought to use Prowl for was pushing my @replies and mentions from Twitter to my iPhone. I wanted to run the script from my Ubuntu server rather then keeping my desktop on 24/7. I did a quick search online and didn't find any command line options for achieving such a task. I decided to code a quick and dirty PHP script to accomplish what I wanted to do.

This is a quick hack. There are probably more efficient ways of accomplishing this task. I used ProwlPHP to link to the Prowl API.

I assume you know PHP and know how to navigate in Terminal. This is a command line app, it will run on any computer/server running PHP. I have extensively commented the script. Hopefully it is easy to follow along.

Quick Instructions:

  1. Download the newest version of ProwlPHP and copy it a directory.
  2. Create a file lastreply.txt and stick in in the same directory as the below code will be copied into. This file needs to have read and write permissions.
  3. Copy the below code into another file. Alter the ProwlPHP includes directory on line 2, and add your Prowl API key, Twitter Username, and Twitter Password in the constructor.
//CHANGE THIS PATH TO WHERE ProwlPHP IS LOCATED ON
//YOUR SERVER
include('../API/ProwlPHP.php');
$t = new Twitter();
 
/**
 * @class Twitter
 * This class integrates Twitter and Prowl notifications with the
 * iPhone. When this script runs it checks to see if any new
 * Twitter mentions have occured on your account since last check.
 * If any exist, the Tweet is sent to Prowl and will be notified
 * via push on your iPhone.
 *
 * This class uses ProwlPHP located at:
 * 			http://github.com/Fenric/ProwlPHP
 *
 * @author Kastang (josh dot kastang at gmail dot com)
 */
class Twitter {
 
    var $xml;
    var $lastID;
    var $tUser;
    var $tPass;
    var $prowl;
 
    /**
     * Constructor for Twitter Class. Three lines need to be edited
     * below before running the file: prowl, tUser, and tPass.
     */
    function __construct() {
 
        //EDIT THE 3 LINES BELOW
        $this->prowl = new Prowl('YOUR PROWL API KEY');
        $this->tUser = "TWITTER USERNAME";
        $this->tPass = "TWITTER PASSWORD";
 
        //XML info loaded from Twitter API.
        $this->xml = simplexml_load_string($this->getReplies());
 
        //Opens the lastreply file which contants the
        //id of the last mentioned tweet.
        $this->lastID = file_get_contents("lastreply.txt");
 
        //If the file is empty (probably the first time
        //you are using the script). It will pull the
        //newest mention id from your twitter feed and
        //store it in the file.
        if ($this->lastID == null) {
            $this->lastID = $this->xml->status[0]->id;
            $full = "@" . $this->xml->status[0]->user->screen_name .
                    ": " . $this->xml->status[0]->text;
            $this->prowl($full);
            $this->updateNewest($this->xml->status[0]->id);
        }
 
        //Checks for new Twitter Mentions.
        $this->checkForUpdates();
    }
 
    /**
     * Checks for updates.
     */
    function checkForUpdates() {
        //first run boolean.
        $first = true;
 
        //For each mention in the XML array, check to see if
        //the current ID is greater then the last recorded ID.
        //If it is, push the current Tweet to Prowl, if it isn't,
        //check to see if it is the first run, if it is, break out
        //of the for loop, if it isn't the first run, update the
        //lastreply.txt file and break from the forloop.
        for ($i = 0; $i &lt; 10; $i++) {             $curr = $this->xml->status[$i]->id;
            if ($curr > $this->lastID) {
                $first = false;
                $full = "@" . $this->xml->status[$i]->user->screen_name .
                        ": " . $this->xml->status[$i]->text;
                $this->prowl($full);
            } else {
                if ($first) {
                    break;
                } else {
                    $this->updateNewest($this->xml->status[($i - 1)]->id);
                    break;
                }
            }
        }
    }
 
    /**
     * Writes the newest @reply id to a file.
     */
    function updateNewest($id) {
        $file = fopen("lastreply.txt", "w");
        fwrite($file, $id);
        fclose($file);
    }
 
    /**
     * Push the Tweet to Prowl. This code is modified from
     * example.php in the ProwlPHP API Wrapper.
     */
    function prowl($tweet) {
 
        $this->prowl->push(array(
            'application' => 'Twitter',
            'event' => 'Reply',
            'description' => $tweet,
            'priority' => 0,
                ), true);
    }
 
    /**
     * Gets replies from Twitter. In order to grab replies, you
     * must be authenticated.
     */
    function getReplies() {
        $twitterHost = "http://twitter.com/statuses/mentions.xml";
        $curl;
        $curl = curl_init();
        curl_setopt($curl, CURLOPT_CONNECTTIMEOUT, 2);
        curl_setopt($curl, CURLOPT_HEADER, false);
        curl_setopt($curl, CURLOPT_HTTPAUTH, CURLAUTH_BASIC);
        curl_setopt($curl, CURLOPT_RETURNTRANSFER, 1);
        curl_setopt($curl, CURLOPT_USERPWD, "$this->tUser:$this->tPass");
        curl_setopt($curl, CURLOPT_HTTP_VERSION, CURL_HTTP_VERSION_1_1);
        curl_setopt($curl, CURLOPT_URL, $twitterHost);
        $result = curl_exec($curl);
        curl_close($curl);
        header('Content-Type: application/xml; charset=ISO-8859-1');
        return $result;
    }
}

I chose to add an entry in my crontab to run this script every 10 minutes. The time can be adjusted to suit your needs. Personally I do not see a need to ping Twitter more then once every ten minutes.

*/10 * * * * curl /path/to/replies.php

If all goes well, you should see something like this when someone mentions you in a tweet:

Prowl pushing a notification to the iPhone.

Prowl pushing a notification to the iPhone.

It is possible to setup a redirection within Prowl to automatically launch your Twitter client of choice when you get a Twitter based notification. You can also open Prowl and view all notifications:

Prowl Main Screen

Prowl Main Screen

The script should be fairly easy to modify. I will probably add Direct Messaging next to my Prowl Push Notifications.

~