All About Web Services

Like many gainfully employed penetration testers I test internal networks and external networks. While there are many ways to scope them, these days external network testing largely consists of web application assessments. Because I did not come from a webdev background I have taken the time to get comfortable with as many of the fundamentals of modern web apps as possible. In this blog post I would like to tackle one of these topics that took me some time to wrap my head around when I first bumped up against it. This post will be centered mostly on concepts and the next one in the series will jump into some strategies for penetration testing.

Before I get too far a quick shout out to my friend Tom Porter for pointing out that adding a few links into my posts were useful. I went back and added a few to my previous post as well!

Modern Web Services

A word of caution. I am NOT a web developer so keep in that in mind as you parse through my ramblings on these topics. There are number of terms that are inescapable when reading about modern web applications:

  • REST
  • SOAP
  • JSON
  • WSDL

Let's start with the broadest perspective and work to the more narrow to get a feel for what these things are.

Web Services

A long time ago in a browser far far away the most exciting thing a web page could do was make the title blink (yay DHTML). Browsers have become much more capable and web applications have scaled up to take advantage of the new functionality. Now days our web applications can do all kinds of sophisticated things. Web services can allow an application to publish functionality and to share data with other applications. Thinking back to the last post about XML we covered the idea of saving data in a way that would be usable by different applications. This is an extension of that same concept. Web services are kind of a neutral way to move data between apps or systems that everyone can agree to use. On your local operating system you might think of this sort of like RPC. Another way to say this is that Web services are a platform independent way to make functions of an application usable over the internet.

Now we have danced around the concept for a bit let's look at some real world applications of the idea. That's where SOAP and REST come in.

SOAP

Ok so the first question, What is SOAP? SOAP stands for Simple Object Access Protocol and as for what it is? It is more or less a format for sending messages over HTTP. This is a good thing because HTTP is an established protocol that browsers understand. SOAP employs XML as a format for messages, but a very specifically formatted XML document called an envelope. Let's take a quick look at an example:

<?xml version="1.0"?>
<soap:Envelope
xmlns:soap="http://www.w3.org/2001/12/soap-envelope"
soap:encodingStyle="http://www.w3.org/2001/12/soap-encoding">

  <soap:Header>
...
  </soap:Header>

  <soap:Body>
...
  <soap:Fault>
...
  </soap:Fault>
  </soap:Body>

</soap:Envelope>

A few key peices to point out. The 'Envelope' element is always the root and you can see the soap namespace. This value should always be as it is in the example above. Next up we see a 'Header', 'Body, and 'Fault' tag. IF there is a header element it will always be first. There are a few key attributes that are common here but since we aren't actually designing a web service I'll let you take some time to go read more about those one your own. After the header you see the 'Body' element which is where all of our stuff is going to go. If this is all sounding crazy just hang in there and we will play around with some examples shortly. The last piece is the 'Fault' element. It does exactly what you thing it does. It holds error info for a message. I'm not going to burn too much time going over the peices of this. You can go learn more about this over at w3schools http://www.w3schools.com/webservices/ws_soap_syntax.asp ["w3schools"]

The next thing that will help us wrap our hands around SOAP is that it is used on both sides of the communication, that is for both sending and receiving data. Let's look at a quick example of what a request and a response might look like in an application.

Request:

POST /Quotation HTTP/1.1
Host: www.track-your-nutrition.org
Content-Type: text/xml; charset=utf-8
Content-Length: nnn

<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2001/12/soap-envelope"
soap:encodingStyle="http://www.w3.org/2001/12/soap-encoding"

<soap:Body xmlns:m="http://www.track-your-nutrition.org/food" >

  <m:GetFood>
     <m:FoodName>Hot Dog</m:FoodName>
  </m:GetFood>

</soap:Body>

</soap:Envelope>

Response:

HTTP/1.0 200 OK
Content-Type: text/xml; charset=utf-8
Content-Length: nnn

<?xml version="1.0"?>
<soap:Envelope xmlns:soap="http://www.w3.org/2001/12/soap-envelope"
soap:encodingStyle="http://www.w3.org/2001/12/soap-encoding"

  <soap:Body xmlns:m="http://www.track-your-nutrition.org/food" >

    <m:GetFoodResponse>
      <m:Food id="201">
        <m:Name>Hot Dog</m:Name>
        <m:Cals>300</m:Cals>
      </m:Food>
    </m:GetFoodResponse>

  </soap:Body>

</soap:Envelope>

WSDL

Ok we have one more critical piece to cover before we get into some demos and that is the all knowing WSDL. What is a WSDL?

WSDL stands for Web Services Description Language. WSDL is a specification of what requests can be made, with which parameters, and what they will return. It is a complete specification of your API.

Basically a WSDL is a document you will find at a website that describes all the services that site offers. That way people that want to write code that will interact with your web service will know how to format their requests and what to expect in response. So what IS a WSDL?

  • A WSDL document is just a simple XML document.
  • It contains set of definitions to describe a web service.

A WSDL has a number of key elements in it that I won't cover here, however, we can use my favorite programming language to pull down a WSDL and go over some details of the associated web service.

Now that we've gone over a few pieces of SOAP lets go to my favorite programming language and play around a bit. Let's start by getting to know the WSDL a little bit. For this we can use the simple wasabi gem (which I believe is part of the savon package which we will use shortly). Lets go ahead and require wasabi.

pry(main)> require 'wasabi'
=> true

For this demo I will be using the Shakespeare web service at http://www.xmlme.com/WSShakespeare.asmx. From the site:

This Web Service takes a phrase from the plays of William Shakespeare and returns the associated speech, speaker, and play.

You will notice we can view the WSDL by appending '?WSDL' to the end of the URL. Let's use wasabi to import the WSDL and assign it to a variable.

wsdl = Wasabi.document("http://www.xmlme.com/WSShakespeare.asmx?WSDL")
=> #<Wasabi::Document:0x007ff562916180 @adapter=nil, @document="http://www.xmlme.com/WSShakespeare.asmx?WSDL">

I chose this service because it is very simple. Let's take a look at the available actions.

pry(main)> wsdl.soap_actions
=> [:get_speech]

Get a map of SOAP action Symbols, their input tag and original SOAP action name:

pry(main)> wsdl.operations
=> {:get_speech=>
{:action=>"http://xmlme.com/WebServices/GetSpeech", :input=>"GetSpeech", :output=>"GetSpeechResponse", :namespace_identifier=>"tns", :parameters=>{:Request=>{:name=>"Request", :type=>"string"}}}}


response = client.call( :get_speech, message: { "Request" => "My kingdom for a horse" })

What does this do? First of all I highly recommend you take a few minutes to peruse the savon site. The documentation does a good job of at least getting you started on the basics (even though the documentation was lacking in a few places for me). Essentially savon is building a SOAP request for us and sending it to the web service then grabbing the response. I want to point out once more this is a hugely simplified version of ways you might do this in a more robust script but I think it gets the point across.

This is kind of a critical piece here where we are using a wonderful gem to do all the work for us. It is very possible you are as lazy as me and don't feel like firing up a proxy to inspect this web traffic so you know what's going on here so I am going to give you an idea of what the actual HTTP request looked like when it was sent.

POST /WSShakespeare.asmx HTTP/1.1
Host: www.xmlme.com
Content-Type: text/xml; charset=utf-8
Content-Length: length
SOAPAction: "http://xmlme.com/WebServices/GetSpeech"

<?xml version="1.0" encoding="utf-8"?>
<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
  <soap:Body>
   <GetSpeech xmlns="http://xmlme.com/WebServices">
    <Request>My kingdom for a horse</Request>
   </GetSpeech>
  </soap:Body>
</soap:Envelope>

I'll leave it you your imagination to figure out what the response might look like (or maybe you can go look at it yourself here http://www.xmlme.com/WSShakespeare.asmx?op=GetSpeech). I am a huge proponent of using libraries when they exist, however, if you are struggling to wrap your head around all the things going on here it might be a fun exercise to implement a simple version of the SOAP client yourself using nokogiri and an HTTP library. Otherwise, let us move on to looking at the response.

response.body
=> {:get_speech_response=>
{:get_speech_result=>
"<SPEECH><PLAY>KING RICHARD III</PLAY><SPEAKER>KING RICHARD III</SPEAKER>Slave, I have set my life upon a cast, And I will stand the hazard of the die: I think there be six Richmonds in the field; Five have I slain to-day instead of him. A horse! a horse! my kingdom for a horse!</SPEECH>",
:@xmlns=>"http://xmlme.com/WebServices"}}

Now we see what we got back we will grab out the XML and use our sweet nokogiri skills to parse the name of the play.

xml = response.to_hash[:get_speech_response][:get_speech_result]

p xml.search('PLAY').text
"KING RICHARD III"
=> "KING RICHARD III"

Boom, that was a good time!

REST

Ok so REST feels like the hot chick that showed up to the party and everybody is talking about without being too obvious. ReST if talked about a lot but those conversations don't seem to address what ReST really is. Unlike the other things we have talked about up till now it is more of an idea than a specific technology. Again from W3C schools:

REST stands for Representational State Transfer. (It is sometimes spelled "ReST".) It relies on a stateless, client-server, cacheable communications protocol -- and in virtually all cases, the HTTP protocol is used.

ReST is really just a way to build a networked application (eg. the internet). We've been talking about different ways that applications can exchange data between machines without causing problems relating to different browsers or operating systems etc. ReST is just another idea that addresses this issue. Typically you can assume that ReST means using HTTP to accomplish this. If that doesn't make sense just hang with me. A web application implementing ReST for exchanging data uses HTTP requests to get, create, update, and delete data. You might be familiar with the HTTP verbs that mimic the actions (GET, POST, PUT, DELETE).

In a ReSTful application we just use a URI to to work with a particular resource. In our last post we were talked about an application that tracked out meals for us. So lets say we are talking about one of those meals. If we wanted to look at it we might access

/patrick/meals?id=105

There is no rule about what format a REST request returns but we are talking about JSON next so let's just pretend that's what our application is using now. Keep in mind that any format could be used (XML included).

{"meal":[
    {"food":"Carrots"},
    {"food":"Steak"},
    {"food":"Water"}
]}

Now these will be within the context of an HTTP request but I think you can get the idea. I'm getting lazy so if this section was less than helpful I would like to point you to a good resource for reading more about the subject.

JSON

Ok into my last topic. My hope was to front load this post in terms of ideas. The intention being of you can grasp the concepts presented earlier in the topic that by the time you get here we are more focused on semantics. If that was even partially successful then JSON should be pretty easy to approach. XML and JSON are ways of serializing data and are functionally equivalent. XML is used in a LOT of different places and thus has a lot of standards designed around it. The disadvantage (as I understand it) is in it's verbosity. You need more text to do the same amount of stuff compared to JSON. JSON is a simpler format which defines a few basic rules. Let's look at some specifics and talk about exactly what JSON is.

From W3Schools

JSON stands for JavaScript Object Notation and is is a lightweight data-interchange format which is language independent,"self-describing", and easy to understand

I don't know why self-describing is in quotations there but I will just roll with it. There is a fair amount to know about the formatting and rules but I just want to touch on the basics so that when you see JSON you understand how to read it.

When we put something in curly braces (as best I can tell that is the official name for {} ) we indicate an object. Inside the object, we can declare any number of properties using a "name": "value" pairing, separated by commas. That's pretty easy right? Let's go back to our example from talking about REST.

{"meal":[
    {"food":"Carrots"},
    {"food":"Steak"},
    {"food":"Water"}
]}

Here we have a 'meal' object when a few properties being the foods in that meal. These can be nested Alright, as is my way I want to do a little demo using Ruby and a few handy gems. This time we need to require 'net/http' and 'json'. For our demo we will use the handy example json web service at http://api.geonames.org/ which returns placenames for a given postal code. Go take a look at http://www.geonames.org/export/ajax-postalcode-autocomplete.html to see more about the service. It's a neat example of using JSON and gives some good info on why you might use it over SOAP. Moving on...

Let's use net/http to pull down the data. First we will require it:

require 'net/http'
=> true

Next we need to figure out what our URI is. Obviously in anything approach a real example we might want to assign the variables for postal code and country dynamically but I think you get the point if you take a second to review the URL we use below:

uri = URI.parse('http://api.geonames.org/postalCodeLookupJSON?postalcode=93908&country=US&username=demo')
=> #<URI::HTTP:0x007febdc2f2568 URL:http://api.geonames.org/postalCodeLookupJSON?postalcode=93908&country=US&username=demo>

Earlier I pointed out that a ReSTful web service might point at a URI to get a resource. Well now we can see some data we might get back. Let's send a request to to the URI and assign the result to the 'json' variable.

json = Net::HTTP.get(uri)
=> "{\"postalcodes\":[{\"adminName2\":\"Monterey\",\"adminCode2\":\"053\",\"postalcode\":\"93908\",\"adminCode1\":\"CA\",\"countryCode\":\"US\",\"lng\":-121.672861,\"placeName\":\"Salinas\",\"lat\":36.601122,\"adminName1\":\"California\"}]}"

Sweet! Now we can use the json library to parse the data.

pry(main)> require 'json'
=> true

And tada!

pry(main)> results = JSON.parse(json)
=> {"postalcodes"=>
  [{"adminName2"=>"Monterey",
    "adminCode2"=>"053",
    "postalcode"=>"93908",
    "adminCode1"=>"CA",
    "countryCode"=>"US",
    "lng"=>-121.672861,
    "placeName"=>"Salinas",
    "lat"=>36.601122,
    "adminName1"=>"California"}]}

Clearly there is a lot you can do from here but it's pretty straight forward so I will leave the rest up to your imagination.

Conclusion

Yes, there is a ton of important shit I left out, but the point is to cover enough to give you a jumping off point and to talk about penetration testing against web services. That being said I highly encourage anyone with interest to take time to experiment on their own and learn more. I found the best way to learn is to practice on your own. Perhaps build your own service. Write a script to interact with some ReSTful and SOAP web services. In any case there my next post will be to talk about penetration testing against web services.