DeveloperSide.NET Forums
March 18, 2019, 05:38:09 PM *
Welcome, Guest. Please login or register.

Login with username, password and session length
News:
 
   Home   Help Search Login Register  
Pages: [1]
  Print  
Author Topic: DeveloperSide.NET Log Results, 3 years worth...  (Read 27057 times)
0 Members and 1 Guest are viewing this topic.
admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« on: September 12, 2006, 10:00:00 PM »

I've run 'analog' through 08,2003 -- 08,2006 [3 years of logs].

These results are filtered, no partials (206) for downloads and no moved (301, 302) requests for urls.

Some users use a Download Manager that can create 100s and sometimes 1000s of connections...

Partials (206) can increase your request numbers, in downloads, by a factor or 10x-100x AND MORE [what happens when a 20 downloads per day site gets just 1 DM with 1k-10k spread connections; or a bunch of 100s or even 10s?]. And I have certainly seen some outrageous claims and results on other unnamed sites.

Also, it is wise to take into account dropped downloads, which are still HTTP Status code '200'. I have just kept the results that were reported in the 10mb-100mb category, did the math, and took out 10k for good measure...

Not to mention that 'bytes out' [%b], under the regular Apache logging mechanism, reports the size of the file and not the amount of data transferred [%O -- mod_logio]. I have always used mod_logio [%O].

Results...
For the last 3 years...

Web-Developer/DeveloperSide.NET Web-Server Suite has been downloaded over 60,000+ times.

URL '/download/', which has always contained all the Suites and the extra modules has had over 170,000 requests for files.

MSN Search referrer requests are non-existent, its all google with us.

Successful requests: over 6.2 million

Distinct hosts served: over 750,000

Data transferred: 2.05 terabytes -- probably more like 3 terabytes at the router... Average month for DeveloperSide.NET has been 60GB to 200GB at the router.

Again, these results have been filtered with a custom configuration file; otherwise, by the standard of other websites, some would be at least 10x greater, and even more.

And now with the site re-design and updates taking place, the number of requests and downloads will increase... It usually take about 2 weeks for Google to update, so I will be checking again to see what kind of numbers I'm getting for the week.

BTW, this is only for http://www.devside.net/ and does not include http://forums.devside.net/
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
Jorge
Customer, Basic Support
Guru
*****
Posts: 544

jorge_schrauwen@msn.com
View Profile WWW
« Reply #1 on: September 13, 2006, 07:09:05 AM »

Nice man!

I supose you could reduce that though.

By writing a small program C, C+, Delphi (runtime less) that downloads components.

So you can download only httpd with ssl

but if later you need php or so you can simply add it and write the include line for it at the end of httpd.conf. same for mod_perl, mod_xxx, perl, ...
Logged

admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« Reply #2 on: September 13, 2006, 01:06:35 PM »

My bandwidth allowance is 1200GB per month, 12.5Mb link. Dedicated server. I suppose I do not really need it, but I do have it. My goal for DeveloperSide.NET is to fill that 1200GB per month quota. With a little work, I think it is doable.
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« Reply #3 on: September 13, 2006, 01:44:24 PM »

Here is what partials will do to the results...

Quote

[root@www devside.net]# grep ' 206 ' 200*/*/access_log.* | wc -l
 419680


Thats an extra 420,000 downloads that others would report.

Lets say 1/2 of that is for the Suite [more like 80%, at least]. But lets just keep it at 1/2.

With partials and also no mod_logio %O...
210,000 x the size of the download, 33MB average, and you get an extra 7 Terabytes.
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« Reply #4 on: September 13, 2006, 01:59:57 PM »

I would be really curious to see how sourceforge and other sites like download.com hadle the reported numbers. mod_logio is not even built with apache under the 'enable-mods-shared=all' option, under linux.
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
DeliriumServers
Customer, Basic Support
Jr. Member
*****
Posts: 61

delirium.martin@gmail.com a5m0deu5
View Profile WWW
« Reply #5 on: September 13, 2006, 09:06:39 PM »

does the adsense make up for your expenses or is it out of pocket?
Logged

es, i'm crazy
admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« Reply #6 on: September 13, 2006, 10:03:38 PM »

It covers the server expenses. And it would be really nice it if covered everything else.
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« Reply #7 on: September 14, 2006, 05:56:14 PM »

I'm going to have to revise the above numbers. After looking into the matter, it seems that the amount of unique IPs doing 206s, is significant and should be counted in -- correctly. Also, the average suite size was closer to 27MB than 33MB, if not less. And the number of canceled downloads will be small... So just go by unique IP requests -- will lose a few static IP numbers and ISP proxy requests.

Quote

[root@www devside.net]# grep "/download/std/" 200*/*/access_log.* | cut -d ' ' -f1 | sort -n |uniq -c | wc -l

74937

[root@www devside.net]# grep "/download/nocrypto/" 200*/*/access_log.* | cut -d ' ' -f1 | sort -n |uniq -c | wc -l

21907


That should represent the number of unique IPs doing downloads of the non-ssl + ssl-enabled suite as it was always under these directories, with nothing else there (I think). So the number of downloads is a bit more than 60k. More like 70k-97k. Hard to say exactly as I know its been named everything from www-ssl.rar to setup-v1.18-apache-2.2.3-win32.exe...

So I'm going to just say 70,000+ downloads and leave it there, as this is waisting my time. I can only get specific with current versions as I know the exact string to search for and the fact that the log format did not change (as it did once in the past).
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
Jorge
Customer, Basic Support
Guru
*****
Posts: 544

jorge_schrauwen@msn.com
View Profile WWW
« Reply #8 on: September 14, 2006, 06:11:34 PM »

I havent downloaded your suite in a while since i compile everything myself.

A quikc test download gave me and avg speed of around the 420kb/s... not bad since max i get is around 750kb/s i did have a few peaks of more than 420kb/s

And i'm in europe :)
Logged

admin
Administrator
Master of All Subjects
*****
Posts: 3272


View Profile WWW Email
« Reply #9 on: September 15, 2006, 01:52:09 PM »

Ok... I think I'm getting closer...

Code:

[root@www devside.net]# grep "/download/" 200*/*/access_log.* | cut -d ' ' -f1,7 | sort -n |uniq -c | cut -f2 | cut -d ' ' -f2 | sort -n | uniq -c | sort -n > download.nums


Quote

     28 /download/www-ssl.rar
     32 /download/std/setup-v1.17-apache-2.2.2-win32.exe?Cache
     33 /download/components/httpd-20/mod_deflate-zlib-1.2.3-win32.zip
     34 /download/components/httpd-20/mod_jk-1.2.18-win32.zip
     34 /download/components/mod_security-1.9.4-win32-x86.zip
     34 /download/nocrypto/pkg/www-1.12.exe
     35 /download/components/mod_deflate-zlib-1.2.3-win32-x86.zip
     36 /download/components/httpd-20/mod_security-1.9.4-win32.zip
     36 /download/msvcr70.rar?Cache
     37 /download/components/mod_perl-2.0.2-win32-x86.zip
     40 /download/crypto/www.rar
     41 /download/components/mod_aspdotnet-2.0.2003-snapshot-win32-x86.zip
     43 /download/crypto/www-ssl.rar
     44 /download/components/mod_aspdotnet-2.2.2003.zip
     55 /download/crypto/pkg/www-ssl.rar
     56 /download/std/setup-1.15.exe?Cache
     58 /download/ssl/secure/v1.18-www-ssl-opt.rar
     59 /download/components/httpd-20/mod_aspdotnet-2.0.2004.zip
     61 /download/nocrypto/pkg/www-1.09b.rar
     64 /download/components/httpd-22/mod_fcgid-1.10-win32.zip
     65 /download/std/setup-1.12.exe?Cache
     67 /download/ssl/www-ssl.rar
     69 /download/components/httpd-22/mod_deflate-zlib-1.2.3-win32.zip
     76 /download/components/httpd-22/mod_jk-1.2.18-win32.zip
     79 /download/std/v1.17-apache-2.2.2-win32-x86.rar
     81 /download/reqs/apache-2.2.3-projects.zip
     84 /download/crypto/pkg/www.rar
     86 /download/components/httpd-22/mod_security-1.9.4-win32.zip
     87 /download/ssl/secure/v1.17-apache-2.2.2-win32-x86-ssl.exe
     94 /download/std/www-ssl.rar

     96 /download/nocrypto/pkg/www-1.11.rar
    100 /download/components/httpd-22/mod_perl-2.0.2-win32.zip
    100 /download/nocrypto/pkg/www.rar?Cache
    134 /download/components/httpd-22/mod_aspdotnet-2.2.2004.zip
    138 /download/std/setup-1.16.exe?Cache
    139 /download/std/www-1.13.exe
    142 /download/std/v1.17-apache-2.2.2-win32-x86.exe
    143 /download/nocrypto/pkg/www-1.08.rar
    158 /download/std/setup-1.14.exe?Cache
    186 /download/nocrypto/pkg/www-1.10.rar
    203 /download/nocrypto/pkg/www-1.10.zip
    219 /download/std/v1.18-apache-2.2.3-win32.rar
    263 /download/nocrypto/pkg/setup-1.12.exe
    299 /download/components/mod_fcgid-1.09-win32.zip
    302 /download/components/GnuWin32tools.rar
    305 /download/std/www-1.13.rar
    309 /download/components/mod_deflate-zlib-1.2.3-win32.zip
    312 /download/nocrypto/pkg/www-1.10.exe
    313 /download/std/v1.18-apache-2.2.3-win32.exe
    322 /download/components/mod_jk-1.2.15-win32.zip
    341 /download/reqs/msvcr70.rar
    389 /download/components/mod_security-1.9.4-win32.zip
    402 /download/components/mod_aspdotnet-2.0.2003-snapshot-win32.zip
    416 /download/nocrypto/pkg/www-1.11.exe
    447 /download/ssl/secure/v1.17-apache-2.2.2-win32-x86-ssl.rar
    463 /download/components/php4apache2_2-apache2handler.zip
    470 /download/nocrypto/pkg/setup-1.10.exe
    490 /download/components/mod_perl-2.0.2-win32.zip

    616 /download/components/perl-5.8.8-win32.zip
    704 /download/std/v1.17-apache-2.2.2-win32.rar
    735 /download/std/setup-v1.17-apache-2.2.2-win32-x86.exe
    737 /download/std/www-1.12.exe
    757 /download/std/www-1.12.rar
    790 /download/std/www-1.15.rar
    793 /download/reqs/GnuWin32tools.rar
    836 /download/components/mysql.rar
    854 /download/std/v1.17-apache-2.2.2-win32.exe
    978 /download/components/php5apache2_2-apache2handler.zip
   1173 /download/std/www-1.15.exe
   1273 /download/std/www-1.16.rar
   1305 /download/std/setup-1.13.exe
   1368 /download/nocrypto/pkg/setup.exe
   1594 /download/std/www-1.14.rar
   1756 /download/crypto/secure/www-ssl.rar
   1806 /download/std/setup-v1.18-apache-2.2.3-win32.exe
   1845 /download/msvcr70.dll
   1883 /download/std/www-1.16.exe
   1998 /download/perl.rar
   2032 /download/std/www-1.14.exe
   2979 /download/ssl/secure/www-ssl.rar
   5030 /download/components/perl.rar
   6207 /download/std/setup-v1.17-apache-2.2.2-win32.exe
   7315 /download/msvcr70.rar
   7392 /download/std/setup-1.12.exe
   9668 /download/std/setup-1.15.exe
  11688 /download/nocrypto/pkg/www.rar
  15292 /download/reqs/msvcr70.dll
  19883 /download/std/setup-1.14.exe
  20876 /download/std/setup-1.16.exe


Though I'm starting to think it much more accurate to simply count the total bytes out, see the distributions of the above Suite versions, figure out the average suite size with that distribution, and divide the two. As bytes out for Suite will be 99% of the site bandwidth, this will count in everything, and the number of dropped d/ls will be insignificant to the figures for various reasons. Take out another 10k for perl, etc... and its should be right on. Though I think I'm just going to stick to 70,000, and only run for the Suite string starting with September -- v1.18.
[Some of those numbers are starting to look more like 90,000 -- and I would rather be safe than sorry.]

There is some really interesting figures in the above...

Maybe 10% at most d/l the manual install packages [Thats my que to get my installer up and running and provide an option under it for a non-install/unpack mode -- that will also take care of only having one Suite package]

D/Ls of ssl-enabled Suite is also 10% max [Soon I will open the ssl option to everyone and just have it in the base package -- again, that takes care of only having one package]

Eyeballing the figures...
v1.14 Dec 21 2004 [10months] [2.3k/month]
v1.15 Oct 04 2005 [4 months] [3.2k/month]
v1.16 Jan 30 2006 [5 months] [4.8k/month]
v1.17 Jun 01 2006 [3 months] [4k/month]
v1.18 Sep 01 2006 [15 days] [4.6k/month]
Logged

DeveloperSide.NET
Advanced PHP and MySQL Solutions for your Web Design and Development needs with Web.Developer Server Suite.
Pages: [1]
  Print  
 
Jump to:  

Powered by MySQL Powered by PHP Powered by SMF 1.1.9 | SMF © 2006-2009, Simple Machines LLC Valid XHTML 1.0! Valid CSS!