Screen Scrapping with HTTP request.

Hi All,

This posts is about the screen scrapping with out using any third party tool by using HttpWebRequest object.

So guys download fidler 2 here to get started. "http://www.fiddler2.com/fiddler2/"

Open up fidler 2 and click launch IE button in the top right corner. when the browser is launched open up the URL in the browser which needs to be scrapped.

and click on the inspector tab in right top column. there you will see several tabs of header , text view , web form for the request generated.
and in the right bottom column you will notice response in various formats select the tab web view in the bottom.

Create a .net application write the following code in MyPage.aspx.cs file

strURL = "http://www.rppsales.com/Properties.aspx";
string startDate = DateTime.Now.ToShortDateString();
string endDate = DateTime.Now.AddDays(1).ToShortDateString();
HttpWebRequest request = (HttpWebRequest)WebRequest.Create(strURL);
request.Method = "POST";
request.CookieContainer = new CookieContainer();
request.CookieContainer.Add(new Uri(strURL), new Cookie("cookieName", "CookieValue"));

In this piece of code you will notice that i am using a cookie container for adding cookies that are made while requesting the site. you can check the cookies in Request > header Tab of filder under the tree view cookies / login

When the cookies are placed in the container we will try getting the request data to get that open up the second tab textview of fidler and use it in the code like.

string postData = "__EVENTTARGET=&__EVENTARGUMENT=&__LASTFOCUS="; //Paste data of textview here in double qoutes

then you have to post the data and will read the response. by using the following peice of code.

byte[] byteArray = Encoding.UTF8.GetBytes(postData);
// Set the ContentType property of the WebRequest.
request.ContentType = "application/x-www-form-urlencoded";
// Set the ContentLength property of the WebRequest.
request.ContentLength = byteArray.Length;
// Get the request stream.
Stream dataStream = request.GetRequestStream();
// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);
// Close the Stream object.
dataStream.Close();
// Get the response.
WebResponse response = request.GetResponse();
// Display the status.
this1.Text = ((HttpWebResponse)response).StatusDescription;
// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();
// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);
// Read the content.
string responseFromServer = reader.ReadToEnd();
// Display the content.

this1.Text += responseFromServer;
////Console.WriteLine(responseFromServer);
// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();



At this stage the response of the server is in your pocket. Apply regular expression to extract the data that you need. and save it any where you want.

Happy Coding.

Cheers.

Action Script 3.0 and .NET --- A Bridge

hi All,

In this post i will use AMF.net to create a bridge between action script(AS) and .NET code.

as an over view we will create the front end of the application in action script and will call the .net classes which are connected to the Database and will get the results back to action script after that we will be parsing the results to display in AS.

Step 1:

Create a .NET class library project with a class defined in it. Let us call it "MyClass"

namespace MyClasses
{
    public class MyClass
    {

        #region Constructor
        public MyClass()
        {

        }

        #endregion

        public static List<myclass> getAllNotes()
        {

            SqlParameter[] prams = null;
            List<myclass> words = new List<myclass>();
            IDataReader reader = null;

            using (DbManager db = DbManager.GetDbManager())
            {
                try
                {
                    // DB logic to return a List 
                }
                catch (Exception)
                {
                    // Exception logging needed
                }
                finally
                {
                    if (reader != null)
                        reader.Close();
                }
                return MyClass;
            }
        }
    }
}

After the class is done build the library project and add reference of it to the ASP.net Web site.

Step 2:

http://amfnet.openmymind.net/ Go to the URL and download the AMF.net classes and follow the first two steps mentioned in http://amfnet.openmymind.net/howto/index.php.

Step 3:

As we will be using AS 3.0 so we can not use the step 3 and 4 code mentioned in above link.

Create a flash application and write the following code in MXML file

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute"
backgroundColor="#FFFFFF"
applicationComplete="init();">
<mx:Script>
<![CDATA[
import mx.messaging.channels.StreamingAMFChannel;

import flash.net.Responder;

import mx.collections.ArrayCollection;
import mx.controls.Label;

private var rs:RemotingService;
private var _myArrayCollection:ArrayCollection;


private function init():void

{


rs = new RemotingService("http://localhost:50020/MyWeb/gateway.php");

var responder:Responder = new Responder(onResult, onFault);

var params:Object = new Object();

params.arg1 = "1";


rs.call("MyClasses.MyClass, Library.getAllNotes", responder);



}

private function onResult(result:Object):void

{

trace(result);

var s:String = "";


var length:int = result.length;
var tempArray:Array = [];



for(var i:int=0; i<length; i++)

{

s += result[i].word + " " + result[i].Property1;
s += result[i].word + " " + result[i].Property2;
//var tempObj:Object = {
//name:result[i].word,
//birthday:result[i].WordCategory
//}


//tempArray.push(tempObj);

}

myLabel.text = s.substring(0,50);
//_myArrayCollection = new ArrayCollection(tempArray);


}


private function onFault(fault:Object):void

{

trace(fault.toString());

}


]]>
</mx:Script>
<mx:Label id="myLabel" visible="true" x="200" y="150" color="#111111" text="thisss" >

</mx:Label>
<mx:Text id="myText" x="200" y="550" >

</mx:Text>
</mx:Application>

Step4:

In the above code the highlighted region is to be replaced with fully Qualified name that is "NameSpace.ClassName, LibararyName.FunctionToBeCalled"


Step 5:

rs.Call will have all the parameters that will be send to the function if the function accpets parameters.

Step 6:

In the function onResult() we will get the response object as in the above case we will be having a list which we will parse to show the desired result in string.


Hope it simplifies the Things

Happy coding.