Skip to content

Fix FileDownloadUtils validation and implement checksum verification - #1133

Open
aalhossary wants to merge 6 commits into
biojava:masterfrom
aalhossary:aa/filedownloadutils-hash-validation
Open

Fix FileDownloadUtils validation and implement checksum verification#1133
aalhossary wants to merge 6 commits into
biojava:masterfrom
aalhossary:aa/filedownloadutils-hash-validation

Conversation

@aalhossary

@aalhossary aalhossary commented Aug 15, 2026

Copy link
Copy Markdown
Member

The download-validation helpers added in 7.0.0 (#979, #980) have several gaps that only surface once a caller passes a real hash URL, or downloads from a server that can return 404. This fixes all of them and implements hash verification rather than leaving it stubbed.

Correctness fixes

  • createValidationFiles(URL, ...) passed a literal Hash.UNKNOWN to its URLConnection overload instead of the caller's argument, so it could never write a hash file and threw IllegalArgumentException for any caller that supplied a
    hashURL.
  • Neither downloadFile nor createValidationFiles checked the HTTP status, so a 404 error page was written into the cache as though it were the requested file. Because the .size sidecar was then taken from that same error response, validateFile subsequently declared it valid. A new HttpStatusException lets callers tell "the resource is not there" apart from a transport failure, which matters for anything that tries several mirrors in turn.
  • downloadFile used FileChannel.transferFrom(rbc, 0, Long.MAX_VALUE), which is not guaranteed to drain a socket-backed channel and could silently truncate a download. Replaced with Files.copy, which loops to end of stream.
  • downloadFile leaked its temporary file on every failure path.
  • validateFile threw NullPointerException for a file with no parent directory, and again if listFiles() returned null; an empty .size file raised an unchecked NoSuchElementException that escaped the surrounding catch.
  • validateFile checked only the first hash sidecar it found, ignoring the rest.

New functionality

  • validateFile now really verifies MD5, SHA-1 and SHA-256. Sidecars are written as bare lowercase hex and parsed tolerantly, so a file downloaded verbatim from a server in coreutils or BSD layout is also understood. A sidecar that cannot be parsed is skipped with a warning rather than failing an otherwise good download.
  • ETagPolicy lets an ETag that is a bare hex digest be recorded as a checksum without a second request. files.wwpdb.org and files.rcsb.org return the content MD5 as the ETag — verified byte-exact — so every download from the wwPDB archive now gets a real integrity check for free. The <mtime>-<size> ETags used by the EBI servers contain a dash and can never be misread as a digest; a test pins that.
  • downloadFileWithValidation downloads and validates over a single connection. The previous pattern opened one connection to read Content-Length and another to fetch the bytes, so the recorded size described a different response than the one written; if the resource changed in between, the cache entry was left permanently failing validation.
    LocalPDBDirectory uses the new single-connection download, and its two-character directory hash is promoted to a reusable getMiddleHash(String).

Behaviour change

The existing four-argument createValidationFiles overloads now default to ETagPolicy.USE_IF_HEX_DIGEST, so callers start recording checksums where the server offers one.
Actually the work is targeted at 7.3.0 rather than a patch release (check next PR). Therefore This small behavioural change should be acceptable.

Consumers fixed here

Fixes #1138CathInstallation downloaded over http, which now 301s to https; HttpURLConnection will not follow a redirect that changes protocol, so the body of the 301 was cached as classification data. Commit ed36d3a switches the URL and routes the download through the status-checking path added here. Commits 5ec48ac and 5d07241 close the same hole in DownloadChemCompProvider, which is not broken today only because the RCSB ligand endpoint still serves plain http.

Testing

37 tests in FileDownloadUtilsTest, including the live wwPDB ETag-to-MD5 round trip and a 404 that must leave neither a cached file nor a .size sidecar behind. Full biojava-core suite: 522 tests, no failures.

The download-validation helpers added in 7.0.0 (biojava#979, biojava#980) had several gaps
that only surface once a caller passes a real hash URL or downloads from a
server that can 404. All of them are fixed here, and hash verification is
implemented rather than stubbed.

Correctness fixes:

* createValidationFiles(URL, ...) passed a literal Hash.UNKNOWN to its
  URLConnection overload instead of the caller's argument, so it could never
  write a hash file and threw IllegalArgumentException for any caller that
  supplied a hashURL.
* Neither downloadFile nor createValidationFiles checked the HTTP status, so a
  404 error page was written into the cache as though it were the requested
  file. Because the .size sidecar was then taken from that same error response,
  validateFile subsequently declared it valid. A new HttpStatusException lets
  callers tell "the resource is not there" apart from a transport failure,
  which matters for anything that tries several mirrors in turn.
* downloadFile used FileChannel.transferFrom(rbc, 0, Long.MAX_VALUE), which is
  not guaranteed to drain a socket-backed channel and could silently truncate a
  download. Replaced with Files.copy, which loops to end of stream.
* downloadFile leaked its temporary file on every failure path.
* validateFile threw NullPointerException for a file with no parent directory,
  and again if listFiles() returned null; an empty .size file raised an
  unchecked NoSuchElementException that escaped the surrounding catch.
* validateFile checked only the first hash sidecar it found, ignoring the rest.

New functionality:

* validateFile now really verifies MD5, SHA-1 and SHA-256 instead of throwing
  UnsupportedOperationException. Sidecars are written as bare lowercase hex and
  parsed tolerantly, so a file downloaded verbatim from a server in coreutils
  or BSD layout is also understood. A sidecar that cannot be parsed is skipped
  with a warning rather than failing an otherwise good download.
* ETagPolicy lets an ETag that is a bare hex digest be recorded as a checksum
  without a second request. files.wwpdb.org and files.rcsb.org return the
  content MD5 as the ETag, so every download from the wwPDB archive now gets a
  real integrity check for free. The <mtime>-<size> ETags used by the EBI
  servers contain a dash and can never be misread as a digest; a test pins that.
* downloadFileWithValidation downloads and validates over a single connection.
  The previous pattern opened one connection to read Content-Length and another
  to fetch the bytes, so the recorded size described a different response than
  the one written; if the resource changed in between, the cache entry was left
  permanently failing validation. Content is digested while streaming to a temp
  file and only moved into place once length and checksum check out.

LocalPDBDirectory uses the new single-connection download, and its two-character
directory hash is promoted to the reusable getMiddleHash(String). That hash
deliberately counts from the end of the identifier so that both spellings of an
entry land in the same bucket: 1cbs and pdb_00001cbs both give "cb", where
counting from the start would file the extended form under "db".

Defaults change slightly: the existing four-argument createValidationFiles
overloads now use ETagPolicy.USE_IF_HEX_DIGEST, so callers start recording
checksums where the server offers one. Targeted at 7.3.0.
Exercises the new status checking against the real wwPDB archive: downloadFile
must throw HttpStatusException rather than writing the error page to the
destination, and createValidationFiles must not record a .size for it. Without
the second half the cached error page would pass validateFile, since its
recorded size would match the error body exactly.
It is a general 'has the server got a newer copy' helper with nothing
PDB-specific about it, and other caching code outside this package needs the
same Last-Modified comparison. Protected access only reached subclasses and
the io package itself.
@aalhossary

Copy link
Copy Markdown
Member Author

Reviewer note: This PR should (when approved) should be merged before #1134 not after it; because this PR is a prerequisite and dependency of the other PR.

download.cathdb.info now answers plain http with a 301 to https.
HttpURLConnection follows redirects within a protocol but deliberately will not
follow one that switches http to https, so the download never reached the real
file. downloadFileFromRemote read the response with a bare openStream() and no
status check, which meant the body of the 301 was written into the local cache
as though it were classification data. Parsing it produced no domains, and the
first sign of trouble was a NullPointerException much later, far from the cause:

    CathDomainTest.test:40 NullPointer
      Cannot invoke "CathDomain.toCanonical()" because "domain" is null

This has been failing on every pull request since the redirect appeared.

Two changes. CATH_DOWNLOAD_URL now uses https, which fixes the immediate
breakage. And the hand-rolled copy loop is replaced by
FileDownloadUtils.downloadFileWithValidation, so a non-2xx response throws
instead of being cached, and a redirect that changes protocol is logged
explicitly rather than passing silently. That is what turns this class of
failure from a mysterious NPE into an error naming the URL and the status.

The shared download also records the byte count it actually wrote, so CATH files
now get size validation they never had; download.cathdb.info sends no ETag, so
there is no checksum to record, but the size check alone would have caught a
truncated file.

CathDomainTest passes again, in about 11 seconds.
@aalhossary

aalhossary commented Aug 15, 2026

Copy link
Copy Markdown
Member Author

Added a commit here that fixes cause 1 of the 3 listed in #1135: ed36d3aFix CATH downloads: use https, and check the response before caching it.

It belongs in this PR rather than a separate one, because it is the first real consumer of the hardening added here.

What it fixes

download.cathdb.info now answers plain http with a 301 to https. HttpURLConnection follows redirects within a protocol but deliberately will not follow one that switches http to https, so our download never reached the real file. downloadFileFromRemote read the response with a bare openStream() and no status check, so the body of the 301 was written into the cache as though it were classification data. Parsing produced no domains, and the first symptom appeared much later and far from the cause:

CathDomainTest.test:40 NullPointer
  Cannot invoke "CathDomain.toCanonical()" because "domain" is null

The URL change alone turns the test green. The reason the commit also swaps the hand-rolled copy loop for downloadFileWithValidation is that the URL was only the trigger — accepting a redirect body as data is the actual defect, and without the status check the next service that changes a redirect reproduces the same silent corruption and the same puzzling NPE.

Scope, to be clear

# Cause in #1135 Status
1 CATH downloader follows a stale http:// URL that now 301s (#1138) fixed by ed36d3a
2 ECOD changed its distribution format (#1139) not addressed here; fixed by #1141
3 Master Build fails on an expired/absent SONAR_TOKEN (#1140) not addressed; needs someone with the SonarCloud credentials

So this does not turn CI green on its own — it removes one of the two test failures that currently fail every pull request, and fixes a genuine runtime bug that affects anyone calling CathInstallation today.

Verification

CathDomainTest fails on master and passes here, in about 11 seconds:

mvn verify -pl biojava-integrationtest -Dtest=CathDomainTest

Edited to point at the sub-issues, and to correct row 2: the ECOD failure was not a judgement call about download size but a format change upstream, now fixed by #1141.

downloadChemCompRecord read the response body without ever checking the status.
A 4xx or 5xx already failed safely, because getInputStream() throws for those,
but a redirect did not: when the JDK declines to follow a 3xx - which it always
does when the redirect changes http to https - getInputStream() returns the body
of the redirect instead. That body is short but not empty, so the "did we read
any lines" check accepted it, gzipped it, and stored it under the component's
name. Every later lookup then read that back and failed to parse it, far from
the request that caused it.

This is the same failure that took the CATH downloader out when
download.cathdb.info moved to https, and files.rcsb.org would do the same to us
the day it stops serving plain http.

Checking the status closes it. The download is left otherwise alone: it
re-compresses the response as it writes, so the local .cif.gz is not a byte copy
of the remote .cif and downloadFileWithValidation cannot stand in for it here.
Serves the responses from a local HttpServer rather than a real service. Aiming
a test at a third party that happens to redirect today would make it fail on the
day they stop, which is the coupling that made the build unreliable in the first
place; this one needs no network at all.

Three cases: a 301 the JDK will not follow because it changes protocol, which is
the case that broke CATH and the one the previous code accepted; a 503; and a
200, to show the guard rejects bad responses rather than simply refusing to
download.

The 200 case deliberately ignores a parse failure on its minimal body. What is
under test is the download path, and the response reaches the cache before
anything tries to parse it, so whether the CIF is well formed says nothing about
whether the guard behaved.

The existing testWeDontCacheGarbage covers a 404, which was already safe:
getInputStream() throws for 4xx, which is why that case never showed the bug.
@aalhossary

Copy link
Copy Markdown
Member Author

Two more commits, applying the same guard to the chemical component download:

Commit
5ec48ac Reject redirects and error responses in the chem comp download
5d07241 Test that a redirect or error is never cached as a chem comp definition

Why this one needed it too

downloadChemCompRecord read the response body without ever checking the status. A 4xx or 5xx already failed safely, because getInputStream() throws for those — which is why the existing testWeDontCacheGarbage (#703) passes and never revealed anything. A redirect is different: when the JDK declines to follow a 3xx, and it always declines when the redirect changes http to https, getInputStream() returns the body of the redirect instead. That body is short but not empty, so the existing "did we read any lines" check accepted it, gzipped it, and stored it under the component's name. Every later lookup then read that back and failed to parse it, far from the request that caused it.

That is precisely how CATH broke in ed36d3a, and files.rcsb.org would do the same to us the day it stops serving plain http. Chemical components are fetched constantly and cached indefinitely, so a poisoned entry there is worth more than the two lines it takes to prevent.

What was deliberately not changed

The download re-compresses the response as it writes, so the local .cif.gz is not a byte copy of the remote .cif. downloadFileWithValidation writes bytes verbatim and cannot stand in for it without changing the on-disk format, so this commit adds only the status check.

On the test

It serves its responses from a local HttpServer and needs no network. Pointing a test at a third party that happens to redirect today would make it fail the day they stop — the coupling that has kept this build red since December. Three cases: a 301 that changes protocol, a 503, and a 200 to confirm the guard rejects bad responses rather than refusing to download at all.

mvn test -pl biojava-structure -Dtest='TestChemCompRedirectNotCached,TestDownloadChemCompProvider'

8 tests, green, offline.

Where this PR now stands

FileDownloadUtils hardening, plus its first two consumers: CATH (ed36d3a, which fixes cause 1 of 3 in #1135) and chem comps (5ec48ac, 5d07241, pre-emptive — nothing is broken there today).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

CathInstallation downloads over http, which now redirects to https, and the redirect body is cached as classification data

1 participant