Top

Multi-threaded Stack Class for PHP

SEMLabs.co.uk / General  / Multi-threaded Stack Class for PHP

Multi-threaded Stack Class for PHP

This is a PHP class to quickly download single URLs and stacks of URLs using multiple threads. It uses the OO cURL wrapper.

One of the most common URI-related tasks is to create stacks of URLs (multi-dimensional arrays), to be downloaded and stored. The main purpose of the class is to provide a simplistic way to do this.

Downloading a Single URL

The above demonstrates downloading a single URL using the get method of the cURL Request class.

Arguments

  1. The URL to downloaded
  2. An array of cURL options, see the PHP reference for a list of options

Returns

An associative array containing the response of the request and information about the request.

Downloading Stacks of URLs

The above demonstrates downloading an array of URLs using the getThreaded method. There are 13 URLs and 5 threads being used. This means that 5 URLs are downloaded at a time in 3 stacks.

Arguments

  1. Array of URLs
  2. An array of cURL options, see the PHP reference for a list of options
  3. Number of threads to be use, i.e. the number of concurrent downloads

The number of threads can also be set using the $threads class variable.

Returns

An associative array containing the content of each URL and its request info. Note that, the request will be returned using the same key as was specified in the array of URLs passed to getThreaded. This will allow you to keep track of requests. For example, if you are downloading URLs from a database, you can use it to keep track of the record Id.

Optimal Threads

When you are downloading 1,000s or more URLs it is best to find out the optimum number of threads – to reduce the execution time of your script. I find it is best to use one thread for every 5-10 kbps of bandwidth for HTML. The optimal number of threads will depend on what you are downloading though.

Share