Fix FileDownloadUtils validation and implement checksum verification - #1133
Fix FileDownloadUtils validation and implement checksum verification#1133aalhossary wants to merge 6 commits into
Conversation
The download-validation helpers added in 7.0.0 (biojava#979, biojava#980) had several gaps that only surface once a caller passes a real hash URL or downloads from a server that can 404. All of them are fixed here, and hash verification is implemented rather than stubbed. Correctness fixes: * createValidationFiles(URL, ...) passed a literal Hash.UNKNOWN to its URLConnection overload instead of the caller's argument, so it could never write a hash file and threw IllegalArgumentException for any caller that supplied a hashURL. * Neither downloadFile nor createValidationFiles checked the HTTP status, so a 404 error page was written into the cache as though it were the requested file. Because the .size sidecar was then taken from that same error response, validateFile subsequently declared it valid. A new HttpStatusException lets callers tell "the resource is not there" apart from a transport failure, which matters for anything that tries several mirrors in turn. * downloadFile used FileChannel.transferFrom(rbc, 0, Long.MAX_VALUE), which is not guaranteed to drain a socket-backed channel and could silently truncate a download. Replaced with Files.copy, which loops to end of stream. * downloadFile leaked its temporary file on every failure path. * validateFile threw NullPointerException for a file with no parent directory, and again if listFiles() returned null; an empty .size file raised an unchecked NoSuchElementException that escaped the surrounding catch. * validateFile checked only the first hash sidecar it found, ignoring the rest. New functionality: * validateFile now really verifies MD5, SHA-1 and SHA-256 instead of throwing UnsupportedOperationException. Sidecars are written as bare lowercase hex and parsed tolerantly, so a file downloaded verbatim from a server in coreutils or BSD layout is also understood. A sidecar that cannot be parsed is skipped with a warning rather than failing an otherwise good download. * ETagPolicy lets an ETag that is a bare hex digest be recorded as a checksum without a second request. files.wwpdb.org and files.rcsb.org return the content MD5 as the ETag, so every download from the wwPDB archive now gets a real integrity check for free. The <mtime>-<size> ETags used by the EBI servers contain a dash and can never be misread as a digest; a test pins that. * downloadFileWithValidation downloads and validates over a single connection. The previous pattern opened one connection to read Content-Length and another to fetch the bytes, so the recorded size described a different response than the one written; if the resource changed in between, the cache entry was left permanently failing validation. Content is digested while streaming to a temp file and only moved into place once length and checksum check out. LocalPDBDirectory uses the new single-connection download, and its two-character directory hash is promoted to the reusable getMiddleHash(String). That hash deliberately counts from the end of the identifier so that both spellings of an entry land in the same bucket: 1cbs and pdb_00001cbs both give "cb", where counting from the start would file the extended form under "db". Defaults change slightly: the existing four-argument createValidationFiles overloads now use ETagPolicy.USE_IF_HEX_DIGEST, so callers start recording checksums where the server offers one. Targeted at 7.3.0.
Exercises the new status checking against the real wwPDB archive: downloadFile must throw HttpStatusException rather than writing the error page to the destination, and createValidationFiles must not record a .size for it. Without the second half the cached error page would pass validateFile, since its recorded size would match the error body exactly.
It is a general 'has the server got a newer copy' helper with nothing PDB-specific about it, and other caching code outside this package needs the same Last-Modified comparison. Protected access only reached subclasses and the io package itself.
|
Reviewer note: This PR should (when approved) should be merged before #1134 not after it; because this PR is a prerequisite and dependency of the other PR. |
download.cathdb.info now answers plain http with a 301 to https.
HttpURLConnection follows redirects within a protocol but deliberately will not
follow one that switches http to https, so the download never reached the real
file. downloadFileFromRemote read the response with a bare openStream() and no
status check, which meant the body of the 301 was written into the local cache
as though it were classification data. Parsing it produced no domains, and the
first sign of trouble was a NullPointerException much later, far from the cause:
CathDomainTest.test:40 NullPointer
Cannot invoke "CathDomain.toCanonical()" because "domain" is null
This has been failing on every pull request since the redirect appeared.
Two changes. CATH_DOWNLOAD_URL now uses https, which fixes the immediate
breakage. And the hand-rolled copy loop is replaced by
FileDownloadUtils.downloadFileWithValidation, so a non-2xx response throws
instead of being cached, and a redirect that changes protocol is logged
explicitly rather than passing silently. That is what turns this class of
failure from a mysterious NPE into an error naming the URL and the status.
The shared download also records the byte count it actually wrote, so CATH files
now get size validation they never had; download.cathdb.info sends no ETag, so
there is no checksum to record, but the size check alone would have caught a
truncated file.
CathDomainTest passes again, in about 11 seconds.
|
Added a commit here that fixes cause 1 of the 3 listed in #1135: It belongs in this PR rather than a separate one, because it is the first real consumer of the hardening added here. What it fixes
The URL change alone turns the test green. The reason the commit also swaps the hand-rolled copy loop for Scope, to be clear
So this does not turn CI green on its own — it removes one of the two test failures that currently fail every pull request, and fixes a genuine runtime bug that affects anyone calling Verification
Edited to point at the sub-issues, and to correct row 2: the ECOD failure was not a judgement call about download size but a format change upstream, now fixed by #1141. |
downloadChemCompRecord read the response body without ever checking the status. A 4xx or 5xx already failed safely, because getInputStream() throws for those, but a redirect did not: when the JDK declines to follow a 3xx - which it always does when the redirect changes http to https - getInputStream() returns the body of the redirect instead. That body is short but not empty, so the "did we read any lines" check accepted it, gzipped it, and stored it under the component's name. Every later lookup then read that back and failed to parse it, far from the request that caused it. This is the same failure that took the CATH downloader out when download.cathdb.info moved to https, and files.rcsb.org would do the same to us the day it stops serving plain http. Checking the status closes it. The download is left otherwise alone: it re-compresses the response as it writes, so the local .cif.gz is not a byte copy of the remote .cif and downloadFileWithValidation cannot stand in for it here.
Serves the responses from a local HttpServer rather than a real service. Aiming a test at a third party that happens to redirect today would make it fail on the day they stop, which is the coupling that made the build unreliable in the first place; this one needs no network at all. Three cases: a 301 the JDK will not follow because it changes protocol, which is the case that broke CATH and the one the previous code accepted; a 503; and a 200, to show the guard rejects bad responses rather than simply refusing to download. The 200 case deliberately ignores a parse failure on its minimal body. What is under test is the download path, and the response reaches the cache before anything tries to parse it, so whether the CIF is well formed says nothing about whether the guard behaved. The existing testWeDontCacheGarbage covers a 404, which was already safe: getInputStream() throws for 4xx, which is why that case never showed the bug.
|
Two more commits, applying the same guard to the chemical component download:
Why this one needed it too
That is precisely how CATH broke in What was deliberately not changedThe download re-compresses the response as it writes, so the local On the testIt serves its responses from a local 8 tests, green, offline. Where this PR now stands
|
The download-validation helpers added in 7.0.0 (#979, #980) have several gaps that only surface once a caller passes a real hash URL, or downloads from a server that can return 404. This fixes all of them and implements hash verification rather than leaving it stubbed.
Correctness fixes
createValidationFiles(URL, ...)passed a literalHash.UNKNOWNto itsURLConnectionoverload instead of the caller's argument, so it could never write a hash file and threwIllegalArgumentExceptionfor any caller that supplied ahashURL.downloadFilenorcreateValidationFileschecked the HTTP status, so a 404 error page was written into the cache as though it were the requested file. Because the.sizesidecar was then taken from that same error response,validateFilesubsequently declared it valid. A newHttpStatusExceptionlets callers tell "the resource is not there" apart from a transport failure, which matters for anything that tries several mirrors in turn.downloadFileusedFileChannel.transferFrom(rbc, 0, Long.MAX_VALUE), which is not guaranteed to drain a socket-backed channel and could silently truncate a download. Replaced withFiles.copy, which loops to end of stream.downloadFileleaked its temporary file on every failure path.validateFilethrewNullPointerExceptionfor a file with no parent directory, and again iflistFiles()returned null; an empty.sizefile raised an uncheckedNoSuchElementExceptionthat escaped the surrounding catch.validateFilechecked only the first hash sidecar it found, ignoring the rest.New functionality
validateFilenow really verifies MD5, SHA-1 and SHA-256. Sidecars are written as bare lowercase hex and parsed tolerantly, so a file downloaded verbatim from a server in coreutils or BSD layout is also understood. A sidecar that cannot be parsed is skipped with a warning rather than failing an otherwise good download.ETagPolicylets anETagthat is a bare hex digest be recorded as a checksum without a second request.files.wwpdb.organdfiles.rcsb.orgreturn the content MD5 as theETag— verified byte-exact — so every download from the wwPDB archive now gets a real integrity check for free. The<mtime>-<size>ETags used by the EBI servers contain a dash and can never be misread as a digest; a test pins that.downloadFileWithValidationdownloads and validates over a single connection. The previous pattern opened one connection to readContent-Lengthand another to fetch the bytes, so the recorded size described a different response than the one written; if the resource changed in between, the cache entry was left permanently failing validation.LocalPDBDirectoryuses the new single-connection download, and its two-character directory hash is promoted to a reusablegetMiddleHash(String).Behaviour change
The existing four-argument
createValidationFilesoverloads now default toETagPolicy.USE_IF_HEX_DIGEST, so callers start recording checksums where the server offers one.Actually the work is targeted at 7.3.0 rather than a patch release (check next PR). Therefore This small behavioural change should be acceptable.
Consumers fixed here
Fixes #1138 —
CathInstallationdownloaded overhttp, which now 301s tohttps;HttpURLConnectionwill not follow a redirect that changes protocol, so the body of the 301 was cached as classification data. Commited36d3aswitches the URL and routes the download through the status-checking path added here. Commits5ec48acand5d07241close the same hole inDownloadChemCompProvider, which is not broken today only because the RCSB ligand endpoint still serves plain http.Testing
37 tests in
FileDownloadUtilsTest, including the live wwPDB ETag-to-MD5 round trip and a 404 that must leave neither a cached file nor a.sizesidecar behind. Fullbiojava-coresuite: 522 tests, no failures.