DeveloperSide.NET Forums

DeveloperSide.NET => Building a Web Server, for Linux => Topic started by: mikeq on March 21, 2004, 05:50:12 AM



Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mikeq on March 21, 2004, 05:50:12 AM
ever since i upgraded from 2.0.48 to 2.0.49 mod_deflate started
messing up all ".zip" files on my server. i have already set a rule to
exclude .zip from being gzipped (its the same rule i used for 2.0.48,
never had a problem) but apache 2.0.49 ignored this rule and caused
all .zip files to be corrupt after being downloaded.

i rolled back to 2.0.48 and everything's been working fine.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: Anonymous on March 21, 2004, 07:57:52 AM
zlib 1.1.4 or 1.2.1 ?


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mikeq on March 21, 2004, 08:27:03 AM
1.2.1

it works perfectly with 2.0.48.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: Anonymous on March 21, 2004, 06:54:42 PM
Nothing yet on the Usenet groups about it....
Nothing on the httpd-users list...
Nothing on the Apache bug database...

post your httpd.conf (please use the code function to format) and we will take a look at it...


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mikeq on March 21, 2004, 07:32:12 PM
the following options were used to compile apache (both 2.0.48 and 2.0.49)

Code:

CFLAGS="-I/usr/kerberos/include/ -L/usr/kerberos/lib" ./configure --prefix=/usr/local/apache --enable-so --enable-cgi --enable-info --enable-rewrite --enable-speling --enable-usertrack --with-z=/usr/local --enable-deflate --enable-mime-magic --enable-headers --enable-ssl


compilation returns no errors.

here's the mod_deflate configuration section from httpd.conf

Code:

<IfModule mod_deflate.c>
SetOutputFilter DEFLATE
DeflateCompressionLevel 6
SetEnvIfNoCase Request_URI \.(?:gif|jp?g|png|svg|js)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.(?:exe|t?gz|z|zip|bz2|rar|mp3|swf)$ no-gzip dont-vary
SetEnvIfNoCase Request_URI \.pdf$ no-gzip dont-vary

AddOutputFilterByType DEFLATE text/*

BrowserMatch ^Mozilla/4 gzip-only-text/html
BrowserMatch ^Mozilla/4\.0[678] no-gzip
BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

<IfModule mod_headers.c>
Header append Vary User-Agent env=!dont-vary
</IfModule>

</IfModule>


now under apache 2.0.48 everything works fine. all extensions defined as exlustions are not gzipped and files can be downloaded without a problem.

under apache 2.0.49 all files in .zip format on the server are corrupt after being downloaded. the problem is solved by commenting out all mod_deflate directives in httpd.conf

there's definitely a problem with 2.0.49.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mikeq on March 23, 2004, 10:10:36 PM
i have noticed that devside.net has not updated the apache building section of the tutorials. i guess there's a reason for this...


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on March 23, 2004, 11:12:44 PM
Perhaps you should try method #1...

http://www.devside.net/web/config/compression

Code:

<Location />
  <IfModule mod_deflate.c>
    # compress content with type html, text, and css
    AddOutputFilterByType DEFLATE text/html text/plain text/css
    <IfModule mod_headers.c>
      # properly handle requests coming from behind proxies
      Header append Vary User-Agent
    </IfModule>
  </IfModule>
</Location>


<IfModule mod_deflate.c>
  BrowserMatch ^Mozilla/4 gzip-only-text/html
  BrowserMatch ^Mozilla/4\.0[678] no-gzip
  BrowserMatch \bMSIE !no-gzip !gzip-only-text/html
</IfModule>


Actually, your original code has a problem...

Quote

Code:

SetOutputFilter DEFLATE
...
AddOutputFilterByType DEFLATE text/*


That does NOT seem correct.  You are setting the DEFLATE filter twice, and this might be the root of your problems.  It might have worked under 2.0.48, but under 2.0.49 this could cause problems.

Get rid of the first one (SetOutputFilter DEFLATE), and also define those "text/*", as I have found in the past that "text/*" has some problems.

Also, look at the httpd.conf section of...

http://www.devside.net/web/server/linux


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mikeq on March 24, 2004, 01:39:53 AM
thanks admin. the problem was resolved by using method#1.

yes my original configuration has that problem u pointed out. however it was not the cause of the problem. apache 2.0.49 simply wont work with method#2. i have tried the standard method#2 by using ur directives

Code:

<IfModule mod_deflate.c>
  # place filter 'DEFLATE' on all outgoing content
  SetOutputFilter DEFLATE
  # exclude uncompressible content via file type
  SetEnvIfNoCase Request_URI \.(?:gif|jpe?g|png|rar|zip)$ no-gzip
  <IfModule mod_headers.c>
    # properly handle requests coming from behind proxies
    Header append Vary User-Agent
  </IfModule>
</IfModule>


and the result was .zip files were still corrupted even tho it was explicly excluded from being gzipped. i guess its 2.0.49's bug.

i have one more question tho regarding

Code:

Header append Vary User-Agent


this works fine with all .html, .htm and .css files as defined by

Code:

AddOutputFilterByType DEFLATE text/html text/plain text/css


however it wont append this User-Agent header for my .php files. my .php files were obviously seen as "text/html" and they were indeed being gzipped. (it returns vary: accept-encoding) however User-Agent is missing. i fear that this may cause problems for my visitors as most of my web pages are .php files.

i tried adding "application/x-httpd-php" in AddOutputFilterByType and got no luck. it works fine with method#1 tho. (appended after accept-encoding)

thanks again!


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on March 24, 2004, 02:07:31 AM
The only function of the 'Vary: User-Agent' header field is to tell Proxies that the browser that a client is using and the content received might depend on the "User-Agent" (the client browser type).

"Header append Vary User-Agent " has nothing to do with DEFLATE, php, or anything else for that matter.  It is a safety feature for clients that go thru proxies.  As a proxy might change the User-Agent string by itself and your server will think that the changed User-Agent string is the browser type of the client...

Quote

However it wont append this User-Agent header for my .php files.


Sure it will.  But you can NOT append/modify a header field ONCE data has been sent to the client or if "hardcoded" HTML code is present in the php file...

Enable "output_buffering = on" under php.ini and have NO html code outside <?php ... code ... ?>

You can read up about this on php.net for a better understanding.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: Anonymous on March 24, 2004, 02:16:03 AM
thanks again admin. :)

Quote

Sure it will. But you can NOT append/modify a header field ONCE data has been sent to the client or if "hardcoded" HTML code is present in the php file...

Enable "output_buffering = on" under php.ini and have NO html code outside <?php ... code ... ?>


umm output_buffering is on and i tried it on a phpinfo.php file that contains only
Code:

<?php
phpinfo
&#40;&#41;;
?>



then i did a "HEAD (http://)mydomain.com/phpinfo.php" and still User-Agent is missing from the vary header. here's the output:

Code:


200 OK
Date: Wed, 24 Mar 2004 02:05:34 GMT
Server: Apache/2.0.49 (Unix) mod_ssl/2.0.49 OpenSSL/0.9.7d PHP/4.3.4
Vary: Accept-Encoding
Content-Type: text/html
Client-Date: Wed, 24 Mar 2004 02:05:34 GMT
Client-Response-Num: 1
X-Powered-By: PHP/4.3.4


actually when i did a 'HEAD http://www.devside.net/' it returns the same thing. -_- no User-Agent. however if its any other file say a .txt or .html then User-Agent will be present.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on March 24, 2004, 02:45:59 AM
By any chance, under your php.ini, is...

zlib.output_compression = On
output_handler = ob_gzhandler

...any one of these set?


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mike on March 24, 2004, 03:01:04 AM
nope. first one is off. 2nd one is empty.
output_buffering=4096

i do have zend optimizer 2.5.1 installed tho.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: mikeq on March 24, 2004, 03:03:33 AM
btw if i use the "compress all" mode

SetOutputFilter DEFLATE

then all .php files will have user-agent appended in the Vary header.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on July 27, 2004, 02:22:38 AM
There "was" a bug that caused PHP (at least back to v4.3.2) to replace the "Vary: User-Agent" Header when "adding" the "Accept-Encoding" part.

It looks like something is still wrong.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 09, 2004, 05:45:52 PM
Hi,

I have a similar problem. I'm using Apache 1.3.31 and PHP 4.3.8. RAR and ZIP files are corrupt when downloaded.

The md5sum for the file on the server was the same as the md5sum on my client's PC, so the file is being corrupted by Apache. My client has tried testing it using Firefox and Internet Explorer on Windows XP. He's not using any download managers.

Any ideas would be gratefully appreciated! :)

Matt
RetroWeb.net


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on September 09, 2004, 05:52:24 PM
If the file was "corrupt" the md5sum would have changed when going from server to client.

You cannot have a good file match a corrupted file's md5sum.

The problems in this thread are more than likely a result of a file being compressed twice, as a result of a configuration problem in Apache, or anything else down the line.  That, or your file is corrupt itself.

Test your file to make sure its good.  Turn off Apache's mod_deflate (or in your case mod_gzip ?) and see if it works.  Also check the headers, on the client/recieving side.

BTW, if you are using Apache to compress files, you did not turn ON any of PHP's compression directives under php.ini/cnf?


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 09, 2004, 11:06:09 PM
Quote from: "admin"
If the file was "corrupt" the md5sum would have changed when going from server to client.

You cannot have a good file match a corrupted file's md5sum.


The file was the same on the server as it was on my client's machine before he uploaded it. When he downloaded it back to his PC, it was corrupt and the md5sum had changed, thus proving that the corruption was caused in the process of downloading.

Quote from: "admin"
Test your file to make sure its good.  Turn off Apache's mod_deflate (or in your case mod_gzip ?) and see if it works.  Also check the headers, on the client/recieving side.

BTW, if you are using Apache to compress files, you did not turn ON any of PHP's compression directives under php.ini/cnf?


The file works fine before it's downloaded. We don't have mod_gzip installed. I'll try to get my client to check the headers.

We're not using any compression options in php.ini:

Code:
zlib.output_compression = Off
                                                                               
; You cannot specify additional output handlers if zlib.output_compression
; is activated here. This setting does the same as output_handler but in
; a different order.
;zlib.output_handler =



Any other ideas? Thanks for the help!

Matt :)


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on September 10, 2004, 12:33:47 AM
I take it the file gets corrupted no matter who downloads it?  Have you downloaded it yourself?

Is it a straight link?  No redirecting or PHP sendfile type of stuff?

You could try setting LogLevel under httpd.conf to debug for max output.  Maybe something will come up.

Also check the error_log


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 10, 2004, 12:53:59 AM
The problem is that when I downloaded it, I had no problems :)

It's a straight link - no PHP output or anything. I'll see what I can do with the logs - thanks :)

Matt


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on September 10, 2004, 07:11:23 AM
Then the problem is with your client and not Apache.  He could have all kinds of spyware, viruses, etc.. on his system.  Or maybe he just does not know what he is doing, in the first place.

Thats a really critical point to leave out, that only he has the problem.

He should try to download another file, similar extension and size, from another server.  Most of the GPL/open source progs out there should have a md5sum.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 10, 2004, 07:39:23 AM
Thing is, he isn't having any problems downloading RAR or ZIP files from other servers in the same manner.

And I figured maybe my case was just an exception.

Matt


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on September 11, 2004, 05:09:46 AM
Are you sure that your Apache is not loading mod_gzip ?  I read your usenet post, you are not the admin of this server, nor have root access?

This would be another critical point.
You should verify this part yourself.

If your Apache 1.3.xx is not running mod_gzip, you should get another sysadmin.  You would save 50% of your traffic.  Also, your sysadmin could just be lying to you, not wanting to deal with this problem.

And dont listen to those responding in that usenet post...  mod_gzip is not "3rd party" mod_deflate.  And mod_deflate has no problems.  Its the client browser versions that do.  And those are easily corrected for server-side.


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 11, 2004, 09:04:03 PM
Quote from: "admin"
Are you sure that your Apache is not loading mod_gzip ?  I read your usenet post, you are not the admin of this server, nor have root access?


I am the owner of RetroWeb.net, a web hosting company. I have root access to all my servers. I'm not an expert at Linux, which is why I pay sysadmins to handle that. I do have some Linux knowledge however (I run in on a couple of workstations at home, including my main workstation which I'm typing on now) so I help out where I can. Trying to find an answer to this problem is a good example.

Matt :)


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on September 11, 2004, 10:36:23 PM
All you have to do is view Apache's httpd.conf file to see if there is a "LoadModule" line with "mod_gzip or gzip_module" in it.

It could also have been compiled in, and you could view all the compiled in modules with...
httpd -l (lowercase L)

Or just check the HTTP headers for something like "Content-Encoding: gzip"


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 11, 2004, 10:50:42 PM
Quote from: "admin"
All you have to do is view Apache's httpd.conf file to see if there is a "LoadModule" line with "mod_gzip or gzip_module" in it.


I did that originally... that's how I knew it's not installed. :) It wasn't compiled in either.

Matt


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: admin on September 11, 2004, 10:51:07 PM
Either way, Apache without mod_gzip (or with it) would not selectively corrupt file.zip/rar only for your client (using IE and Firefox), each time and for no one else.

If your client can pull other files from your system, then he/she might be lying for some reason.  That or the client has something on his system that is causing this.

If your client cannot get any files with the zip/rar extension from your server, then the path the packets are taking could have a bad node in it.
Is the corruption consistent each time?  Are the md5sums the same for each download, or does he/she get a different hash each time?  What is the name of this file, and have you tried changing it, or the extension?


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 11, 2004, 11:55:16 PM
Quote from: "admin"
If your client can pull other files from your system, then he/she might be lying for some reason.  That or the client has something on his system that is causing this.


The way I see it is that you can never really know the exact circumstances of a problem someone else is having. He's using standard software (Windows XP, IE, Firefox, no download managers) but then there could be a multitude of other influential factors.

Quote from: "admin"
If your client cannot get any files with the zip/rar extension from your server, then the path the packets are taking could have a bad node in it.
Is the corruption consistent each time?  Are the md5sums the same for each download, or does he/she get a different hash each time?  What is the name of this file, and have you tried changing it, or the extension?


I haven't asked him to repeat the tests. The file ends in .rar and is composed of English characters, with an underscore in the middle. It's only about 10 characters long.

I'll see about repeating the tests.

Thanks,
Matt :)


Title: Apache 2.0.49 - Broken mod_deflate, anyone?
Post by: RetroWeb on September 12, 2004, 10:45:15 PM
Well, I fixed it :)

I commented out the bottom two lines:

 # AddEncoding allows you to have certain browsers (Mosaic/X 2.1+) uncompress
# information on the fly. Note: Not all browsers support this.
# Despite the name similarity, the following Add* directives have nothing
# to do with the FancyIndexing customization directives above.
#
# Matt commented out the two lines below on 9th Sept 2004
# AddEncoding x-compress Z
# AddEncoding x-gzip gz tgz

Thanks for all your help, and I hope this helps someone else with any similar problems :)
Matt