This blog post is the first one in a series of posts covering how to send free SMS messages using the Ung1881 web site. This first post discuss how to sign into a "Forms Authentication" protected ASP.NET web site, store an authentication cookie, and programmatically post web forms. Even if you're not interested in the free SMS messages you might still find the article interesting. The next part will be on how to use the .NET Class Library in a Windows Vista Gadget.
Unlimited free SMS messages, exposed as a .NET Class Library… Sounds to good to be true? Well, It's not, but as with everything else that's free (except Open Source?) it has a catch. You can only send messages containing 130 characters. The last 30 characters are reserved for commercials. The good thing is that you can send as many messages as you want, and your own number show up as the sender. The service is offered by "Opplysningen 1881" (Norwegian Number Enquiry) through their new portal "Ung1881", a page aimed at young people. In order to use their SMS service you need to be a have a Norwegian social security number, and a phone number. They need your social to do a lookup and verify that you're a real person. Even though you might not be able to register for an account at "Ung1881" you might still find the code and article useful since the concepts apply for any web application.
I got interested in the "Ung1881" after Håvard pointed me to it some weeks back. After doing some re-search it turned out someone had written a Mac OSX Widget letting you send SMS messages directly from your Mac desktop. My immediate idea was to port this widget to the Windows Vista Sidebar, but after looking at the JavaScript code of the widget I soon realized this was a bad idea. The Mac OSX Widget is depending on a third party server doing the screen scrapping server side, so all your SMS messages (including your username and password) are sent through a third party using HTTP GET like this: http://www.theretard.net/smss.pl?u=username&p=password&n=number&m=message. I don't know the guys behind theretard.net, but there's no way I'm trusting them with my username, password and every SMS message I ever send. Just imagine the password harvesting possibilities! On top of this they don't even use HTTPS for communication.
I decided I had to write a .NET Assembly wrapping the "Ung1881" service, so that I could re-use the functionality in other applications, like a Windows Vista Sidebar Gadget, a console application, Outlook 2007, PowerShell etc.
The technique of accessing web sites programmatically is commonly refereed to as screen scrapping. Since the Ung1881 portal doesn't offer any kind of programming interface, you need to tap in to it at the HTTP level, and "pretend" that you're a regular user accessing the page with a browser. Since you're depending on the actual HTML structure page you're application might become unstable, and if the owner of the source you're scrapping change their HTML it might break your application. You also need to be careful that you're not violating copyright regulations, by for instance downloading weather information and displaying it as your own. In the case of "Ung1881" terms don't say anything about "automatic access", and as long as you have a valid account they still get their 30 character commercials included in every SMS message you send. I don't think you'll get into trouble by using this code, but if you do don't blame me!
The "Ung1881" portal is running on the .NET-based Content Management System EPiServer, and uses ASP.NET Forms Authentication to authenticate users on the portal. Forms Authentication normally works by using an authentication cookie. When you logon the server authenticates your username and password. If your credentials are valid, it issues a cookie your browser sends with subsequent requests to the site. So in order to wrap the SMS service in a C# class you basically need to write a "non visual" Web Browser. You need to make a request to their login page, use HTTP POST to post a username and password to the server and store the authentication cookie sent back in the response. On subsequent requests you need to attach the authentication cookie in order to access protected pages (like the send SMS page).
The first thing I did when implementing the "Ung1881" proxy class was to analyze the HTTP traffic going between server and client. There are several tools you can use to monitor network traffic at different levels. My preferred HTTP monitor is Fiddler. Fiddler act as a HTTP Proxy, so all you're HTTP requests are routed through Fiddler. Using Fiddler you can look at the raw requests, the headers, the posted form values etc. New in Fiddler v2.0 is support for HTTPS, which is excellent since the "Ung1881" portal is using HTTPS for secure communication. Another excellent tool for network monitoring is oSpy, written by Ole André Vadla Ravnås - a brilliant programmer, reverse engineer and in general a really nice guy!
The above screenshot shows all the HTTP traffic between Internet Explorer and the "Ung1881" web server. As you can see in the left column the site is quite "chatty" because of all the pictures, advertisements, script, style sheets etc. If you enter your username, password and hit the login button, and look at the request made to the URL https://www.ung1881.no/default____3.aspx you can figure out which form elements that are being posted to the server. The site contains quite a few form elements, but the interesting parts are the following:
Content-Disposition: form-data; name="defaultframework:login:tbxUsername"
myusername
Content-Disposition: form-data; name="defaultframework:login:tbxPassword"
mypassword
Content-Disposition: form-data; name="defaultframework:login:LoginButton.x"
0
Content-Disposition: form-data; name="defaultframework:login:LoginButton.y"
0
Content-Disposition: form-data; name="__VIEWSTATE"
dDw5NDE0MjMxMzc7O2w8ZGVmYXVsdGZyYW1ld29yazpsb2dpbjpMb2dpbk
J1dHRvbjtkZWZhdWx0ZnJhbWV3b3JrOnNlbmRzbXM6YnRuU2VuZDs+Pp5c
kvQYH64y3L+4/9ebcQzi7GmX
I've highlighted the names of the form elements. You could have figured out most of this just by looking at the HTML source, but I find it easier to use Fiddler. The thing that might not be too obvious just by looking at the HTML source, is what form value get posted when you click the login image button. The HTML fragment looks like this:
<input type="image" name="defaultframework:login:LoginButton" id="defaultframework_login_LoginButton" class="..." src="..." alt="" />
If you look at the HTTP POST values above you can see that there are two values posted back to the server when you click the login button, defaultframework:login:LoginButton.x and defaultframework:login:LoginButton.y. Both have the value 0. I took a few seconds before I figured what was happening, but apparently the HTML specifications says that <input type="image"> should post back the x and y values with the coordinates of where the user clicked the image. For accessibility reasons this is deprecated, and if you need x and y values from an image you should use an <imgemap> element instead. In order to confirm with the HTML standard the browser posts an x and y value back to the server.
Another important thing to note is that "Ung1881" is an ASP.NET application, and therefore is depending on the "__VIEWSTATE" form element. This hidden form value holds an encoded string representing the state of the application on the server. If we don't include this value in our scrapper the ASP.NET application can't figure out what state it's in, and how to process the request correctly.
If you look at the response sent back from the server you see that the server issues three cookies:
ASP.NET_SessionId=jnq5hh45zuuu0xyigc1jtm45; path=/
.EPiServerLogin=6E1DBAC98D4073F956DA2000EBF122056E335441CCC7756F98B2
A229387EC47E446701A29D36C4497553A5409F40322142C84B140E93C25CFC468C81
5B25CCC35648CE9B03C96XXXXXXXXXXXX; path=/;HttpOnly
ung1881.no=3568924299.20480.0000; expires=Sun, 25-Mar-2007 16:22:15 GMT; path=/
The interesting part is the .EPiServerLogin-cookie which is the authentication cookie. By adding this cookie to my subsequent requests, the server can tell that you're an authenticated user.
Now that we know what's going on when login into the page it's time to send an SMS message. I click the "Send SMS" link on the page, which takes me to the following URL: https://www.ung1881.no/Templates/SMS____24.aspx. I enter a phone number and message, and hit the button to send the message. The requests gets picked up by Fiddler, and by analyzing the request I figure out which form elements I need to post to the server in order to send a message:
Content-Disposition: form-data; name="defaultframework:_ctl2:Smssend:txtPhonenumber"
myNumber
Content-Disposition: form-data; name="defaultframework:_ctl2:Smssend:txtText"
myMessage
Content-Disposition: form-data; name="defaultframework:_ctl2:Smssend:butSend.x"
46
Content-Disposition: form-data; name="defaultframework:_ctl2:Smssend:butSend.y"
9
Content-Disposition: form-data; name="__VIEWSTATE"
*LARGE CHUNK OF STATE*
So after analyzing the application I now know the following about the communication between browser and server:
- The login URL is: https://www.ung1881.no/default____3.aspx
- The username form element is named: defaultframework:login:tbxUsername
- The password form element is named: defaultframework:login:tbxPassword
- The login button posts two values: defaultframework:login:LoginButton.x and defaultframework:login:LoginButton.y
- The server issues an authentication cookie named .EPiServerLogin
- The send SMS URL is: https://www.ung1881.no/Templates/SMS____24.aspx
- The number form element is named: defaultframework:_ctl2:Smssend:txtPhonenumber
- The message form element is named: defaultframework:_ctl2:Smssend:txtText
- The send SMS button posts two values: defaultframework:_ctl2:Smssend:butSend.x and defaultframework:_ctl2:Smssend:butSend.y
Now that I know everything I need to know about the form elements It's time to write a C# class wrapping the site. The code is fairly well commented, so I won't describe it in detail. The key classes involved in the scrapper are the HttpWebRequest, HttpWebResponse, CookieContainer, StreamReader and StreamWriter classes. Another interesting part of the code is the private GetViewState method which extracts the view state of the page so that you can post it back to the server.
// Copyright (c) 2007, Jonas Follesø
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright
// notice, this list of conditions and the following disclaimer in the
// documentation and/or other materials provided with the distribution.
// * Neither the name of the Jonas Follesø nor the
// names of its contributors may be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY Jonas Follesø ``AS IS'' AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
// DISCLAIMED. IN NO EVENT SHALL Jonas Follesø BE LIABLE FOR ANY
// DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
// LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
// ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Collections.Generic;
using System.Text.RegularExpressions;
using System.Security.Authentication;
namespace Ung1881
{
/// <summary>
/// Proxy class wrapping the Ung1881 site for free SMS messaging.
/// </summary>
public class Ung1881Proxy
{
/// <summary>
/// Variable used to store the authentication cookie.
/// </summary>
private CookieContainer cookies;
/// <summary>
/// Variable used to store the base uri of the page.
/// </summary>
private string baseUri = "https://www.ung1881.no/";
/// <summary>
/// Default constructor.
/// </summary>
public Ung1881Proxy()
{
cookies = new CookieContainer();
}
/// <summary>
/// Login on Ung1881 using username and password.
/// Authenticates against the site and keeps the auth cookie for next request.
/// </summary>
/// <param name="username">Username used to logon.</param>
/// <param name="password">Password used to logon.</param>
public void Login(string username, string password)
{
//Validate arguments.
if (username == null || username.Length == 0)
throw new ArgumentException("You must provide a username!", "username");
if (password == null || password.Length == 0)
throw new ArgumentException("You must provide a password!", "password");
string loginUri = baseUri + "default____3.aspx";
// Perform the first http request against the asp.net application login site.
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(loginUri);
// Get the response object, so that we may get the session cookie.
HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Populate the cookie container.
request.CookieContainer = cookies;
response.Cookies = request.CookieContainer.GetCookies(request.RequestUri);
// Read the incoming stream containing the login dialog page.
StreamReader reader = new StreamReader(response.GetResponseStream());
string loginDlgPage = reader.ReadToEnd();
reader.Close();
// Extract the viewstate value from the login dialog page.
// We need to post this back, along with the username and password
string viewState = GetViewState(loginDlgPage);
// Build postback string.
// This string will vary depending on the page. The best way to find out what your postback
// should look like is to monitor a normal login using a utility like Fiddler.
string postback = String.Format("__VIEWSTATE={0}&defaultframework:login:tbxUsername={1}" +
"&defaultframework:login:tbxPassword={2}" +
"&defaultframework:login:LoginButton.x=0&defaultframework:login:LoginButton.y=0",
viewState, username, password);
// Our second request is the POST of the username / password data.
request = (HttpWebRequest)WebRequest.Create(loginUri);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.CookieContainer = cookies;
// Write our postback data into the request stream
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(postback);
writer.Close();
reader = new StreamReader(request.GetResponse().GetResponseStream());
loginDlgPage = reader.ReadToEnd();
reader.Close();
int index = loginDlgPage.IndexOf("Logg ut", StringComparison.OrdinalIgnoreCase);
if (!CheckAuthenticationCookies() || index == -1)
{
throw new AuthenticationException("Unable to authenticate user. Check your username and password.");
}
}
/// <summary>
/// Method sending a new SMS message trough Ung1881.
/// </summary>
/// <param name="phoneNumber">The phone number of the receiver.</param>
/// <param name="message">The message to send.</param>
/// <returns>True if the message was sendt sucsessfully.</returns>
public void SendSms(string phoneNumber, string message)
{
//Validate arguments
if (phoneNumber == null || phoneNumber.Length == 0)
throw new ArgumentException("You need to provide a phone number!");
if (message == null || message.Length == 0)
throw new ArgumentException("You need to provide a message!");
//Check that we have an authentication cookie.
if (!CheckAuthenticationCookies())
throw new AuthenticationException("User not authenticated. Call Login first!");
//our third request is for the actual webpage after the login.
string smsUrl = baseUri + "/Templates/SMS____24.aspx";
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(smsUrl);
request.CookieContainer = cookies;
//and read the response
StreamReader reader = new StreamReader(request.GetResponse().GetResponseStream());
string page = reader.ReadToEnd();
string viewState = GetViewState(page);
string postback = String.Format("__VIEWSTATE={0}&defaultframework:_ctl2:Smssend:txtPhonenumber={1}" +
"&defaultframework:_ctl2:Smssend:txtText={2}&defaultframework:_ctl2:Smssend:butSend.x=0" +
"&defaultframework:_ctl2:Smssend:butSend.y=0", viewState, phoneNumber, message);
reader.Close();
request = (HttpWebRequest)WebRequest.Create(smsUrl);
request.Method = "POST";
request.ContentType = "application/x-www-form-urlencoded";
request.CookieContainer = cookies;
//Write our postback data into the request stream
StreamWriter writer = new StreamWriter(request.GetRequestStream());
writer.Write(postback);
writer.Close();
//Execute the request and read the response.
reader = new StreamReader(request.GetResponse().GetResponseStream());
page = reader.ReadToEnd();
reader.Close();
//Verify that the message was sent.
if (!page.Contains("SMS sendt"))
{
throw new Ung1881Exception("Unable to send SMS message. Unknown error.");
}
}
/// <summary>
/// Check if the user is authenticated.
/// </summary>
/// <returns>True if the user is authenticated.</returns>
private bool CheckAuthenticationCookies()
{
bool authenticated = false;
//Check that we have cookies.
if (cookies != null)
{
//Check all cookies for the EpiServerLogin cookie.
foreach (Cookie cookie in cookies.GetCookies(new Uri(baseUri)))
{
if (cookie.Name != null &&
cookie.Name.Equals(".EPiServerLogin", StringComparison.OrdinalIgnoreCase))
authenticated = true;
}
}
return authenticated;
}
/// <summary>
/// Extract the viewstate data from a page.
/// </summary>
/// <param name="aspxPage">The raw HTML of the page.</param>
/// <returns>Thew viewstate data of the page.</returns>
private string GetViewState(string aspxPage)
{
Regex regex = new Regex("(?<=(__viewstate\".value.\")).*(?=\"./>)", RegexOptions.IgnoreCase);
Match match = regex.Match(aspxPage);
return System.Web.HttpUtility.UrlEncode(match.Value);
}
}
}
To test the class I've created a simple console application accepting a username, password, number and message as it's argument. Using the console application you can send SMS messages by typing the command: "SMS username password number message". If you add the folder containing the .EXE to your environment path you can send SMS messages from any command prompt.
// Copyright (c) 2007, Jonas Follesø
// All rights reserved.
//
// Redistribution and use in source and binary forms, with or without
// modification, are permitted provided that the following conditions are met:
// * Redistributions of source code must retain the above copyright
// notice, this list of conditions and the following disclaimer.
// * Redistributions in binary form must reproduce the above copyright
// notice, this list of conditions and the following disclaimer in the
// documentation and/or other materials provided with the distribution.
// * Neither the name of the Jonas Follesø nor the
// names of its contributors may be used to endorse or promote products
// derived from this software without specific prior written permission.
//
// THIS SOFTWARE IS PROVIDED BY Jonas Follesø ``AS IS'' AND ANY
// EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
// WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
// DISCLAIMED. IN NO EVENT SHALL Jonas Follesø BE LIABLE FOR ANY
// DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
// LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND
// ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS
// SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
using System;
using System.IO;
using System.Net;
using System.Text;
using System.Collections.Generic;
using System.Text.RegularExpressions;
using Ung1881;
namespace ConsoleScraper
{
class Program
{
static void Main(string[] args)
{
try
{
//Check that we have at least four arguments (username, password, number, message)
if (args.Length < 4)
{
Console.WriteLine("Usage: SMS username password number message");
}
else
{
//Extract variables from arguments.
string username = args[0];
string password = args[1];
string number = args[2];
string message = string.Empty;
//Build up the message.
for (int i = 3; i < args.Length; ++i)
{
message += args[i];
//Add space if this isn't the last word of the message.
message += (i == args.Length - 1) ? string.Empty : " ";
}
Console.WriteLine("Sending message \"{0}\" to number {1}", message, number);
//Send the message.
Ung1881Client client = new Ung1881Client(username, password);
client.SendMessage(number, message);
Console.WriteLine("Message \"{0}\" sent to number {1}", message, number);
}
}
catch (Exception ex)
{
//Simple exception handling.
Console.WriteLine(ex.ToString());
}
}
}
}
So that's about it. A simple C# library allowing you to send unlimited free SMS messages through "Ung1881". You can choose to either download the compiled binaries or the source code. In the next part of this article/post I'll discuss how to use the library in a Windows Vista Sidebar Gadget.
If you have any questions, comments or bugs, or find the library useful, please drop me a comment!