| Age | Commit message (Collapse) | Author |
|
This changes the behaviour for the first one or few nars the substitute script
downloads, with uncompressed and zstd compressed nars prefered rather than
picking by file size.
* guix/scripts/substitute.scm: (%default-fast-decompression?): Change to #t.
* tests/substitute.scm ("substitute, preferred nar URL is 404, other is 200"):
Adjust test.
Change-Id: I89202f084cd6b9d506bcb3d46f75de690c6986b5
|
|
To be consistent with other procedures that make network requests.
* guix/substitutes.scm (download-nar): Add #:keep-alive? option.
* guix/scripts/substitute.scm (process-substitution/fallback)
(process-substitution): Call download-nar with #:keep-alive? #t.
Change-Id: I83b27d0c3a0916d058fbbbeb7aa77dbb8a742768
|
|
From the substitutes script. This makes it possible to use download-nar in
the the Guile guix-daemon.
* guix/scripts/substitute.scm (%fetch-timeout): Move down to where it's now
used.
(%random-state, with-timeout, catch-system-error, http-response-error?,
download-nar): Move to…
* guix/substitutes.scm: …here.
Change-Id: I8c09bf4b33cb5c6d042057d4d9adeb36c24c11dc
|
|
This is in preparation for moving the download-nar procedure out of the
script.
As well as calling open-connection-for-uri/cached, with-cached-connection adds
a single retry to the expression passed in, in the case of a exception that
suggests there's a problem with the cached connection. This is important
because download-nar/http-fetch doesn't check if a connection used for
multiple requests should be closed (because the servers set the relevant
response header).
To make download-nar more generic, have it take open-connection-for-uri as a
keyword argument, and replicate the with-cached-connection single retry by
closing the port in the case of a network error, and recalling
open-connection-for-uri. This will work fine in the case when connection
caching is not in use, as well as when open-connection-for-uri/cached is used,
since open-connection-for-uri/cached will open a new connection if the cached
port is closed.
* guix/scripts/substitute.scm (kind-and-args-exception?): Remove and inline
where necessary.
(call-with-cached-connection): Remove procedure.
(with-cached-connection): Remove syntax rule.
(http-response-error?): New procedure.
(download-nar): Add new #:open-connection-for-uri keyword argument and use it,
also replace with-cached-connection.
(process-substitution/fallback,process-substitution): Pass
#:open-connection-for-uri open-connection-for-uri/cached to download-nar.
Change-Id: I277b1d8dfef79aa1711755b10b9944da7c19157c
|
|
Plus remove http-get-error? from network-error? as http-get-error? doesn't
indicate a network error.
* guix/scripts/substitute.scm (process-substitution/fallback)
(process-substitution): Use http-get-error? with network-error?.
(system-error?, network-error?): Move from here…
* guix/http-client.scm: …to here, and also don't use http-get-error?.
Change-Id: I61ee9e5fbf90ebb76a34aa8b9ec8f5d74f8a3c54
|
|
As this moves download-nar in a direction where it could be used outside the
substitute script.
* guix/scripts/substitute.scm (download-nar): Return expected and actual
hashes and move status-port output to guix-substitute.
(process-substitution/fallback): Remove port argument, and move output to port
to guix-substitute.
(process-substitution): Return hashes from download-nar or
process-substitution/fallback, plus the narinfo.
(guix-substitute): Don't pass the reply-port in to process-substitution and
implement the messages to the reply-port here.
Change-Id: Icbddb9a47620b3520cdd2e8095f37a99824c1ce0
|
|
Pulling the logic up to the script makes this code more portable and not
reliant on setting a global variable.
* guix/scripts/substitute.scm (%prefer-fast-decompression?): Rename to…
(%default-fast-decompression?): this.
(call-with-cpu-usage-monitoring): Use multiple values to return the results
from the thunk as well as the cpu usage.
(display-narinfo-data): Update accordingly.
(download-nar): Add fast-decompression? as a keyword argument, remove
code to set! it and monitor the cpu-usage.
(process-substitution, process-substitution/fallback): Accept and pass through
fast-decompression? to download-nar.
(guix-substitute): Move the cpu usage monitoring and fast decompression
switching logic here.
Change-Id: I4e80b457b55bcda8c0ff4ee224dd94a55e1b24fb
|
|
Rather than just the port and response-content-length. I'm looking at using
the response headers within the substitute script to work out when to close
the connection.
* guix/http-client.scm (http-fetch): Return the response as the second value,
rather than the response-content-length.
* guix/build/download-nar.scm (download-nar): Adapt accordingly.
* guix/build/download.scm (url-fetch): Adapt accordingly.
* guix/scripts/substitute.scm (process-substitution): Adapt accordingly.
* guix/scripts/challenge.scm (call-with-nar): Adapt accordingly.
Change-Id: I490ecf7cef1f5ebbf1e6ed026f6a8fc9dacc56be
|
|
To reduce the codepaths in download-nar.
* guix/scripts/substitute.scm (with-timeout): Accept a #f duration and don't
set a timeout.
(download-nar): Remove the if for fetch-timeout.
Change-Id: I4e944a425a8612e96659dd84dd0e315012f080ab
|
|
* guix/scripts/substitute.scm (process-query): Attempt to precise the 'update
substitutes' message.
Suggested-by: Tobias Geerinckx-Rice <me@tobias.gr>
Reported-by: Felix Lechner <felix.lechner@lease-up.com>
Reviewed-by: Ludovic Courtès <ludo@gnu.org>
Change-Id: I7b3f93f32b73a6a6175a035a09ab51a6e74e384b
|
|
This works around an occasional issue where substitution stops abruptly
due to “Error in the push function” from GnuTLS, as reported at
<https://issues.guix.gnu.org/71238> by Richard Sent.
* guix/scripts/substitute.scm (call-with-cached-connection): Add
‘error/push-error’ and ‘error/pull-error’ to the list of gnutls-error
values for which the connection is reopened.
Change-Id: Icf079dd10b16739f62fee15bc3d90bab77110576
|
|
Experience has shown that keeping too many entries increases disk usage
and, more importantly, leads to long delays when cleaning up the cache,
measured in minutes on slow or busy HDDs with hundreds of thousands of
cache entries, as is common on build machines. In those cases, the cost
of the cache outweighs its benefit.
* guix/scripts/substitute.scm (%narinfo-expired-cache-entry-removal-delay):
Reduce to 5 days.
(cached-narinfo-expiration-time)[max-ttl]: Reduce to 2 days.
Change-Id: Iab212f572ee9041be61716423a3c014f93fe81ed
|
|
These calls happen inside of with-error-handling, so the effect should be the
same, but this opens up the possibility of using this code in a program that
doesn't want to exit when one of these error conditions is met.
Change-Id: I15d963615d85d419559fa0f4333fa4dc1dfbfd3b
* guix/scripts/substitute.scm (download-nar, process-substitution): Use raise
formatted-message rather than leave.
Change-Id: Idd0880206b69e3903e19e0536b87d65a52c200d5
|
|
I don't think the approach of using SIGALARM here for the timeout will work
well in all cases (e.g. when using Guile Fibers), so make it possible to avoid
this.
* guix/scripts/substitute.scm (download-nar): Pass the fetch timeout in as an
option.
Change-Id: I8cbe6cdfa10cdaa7d41974cbea56a95f5efecfe6
|
|
Instead, display the warning from process-substitution and
process-substitution/fallback in the relevant places.
I'm looking at this because I want to make the substitute code less tied to
the script and usable in the Guile guix-daemon.
* guix/scripts/substitute.scm (network-error?): Move warning to…
(process-substitution/fallback, process-substitution): here.
Change-Id: I082b482c0e6ec7e02a8d437ba22dcefca5c40787
|
|
The aim here is to improve the user experience. There's anecdotal evidence
that the network performance for bordeaux is better compared to ci at least
for some users, and I don't know of any issues with rate limiting or access
restriction for bordeaux compared to ci. It also has IPv6 support.
Additionally, bordeaux generally had more substitutes than ci, particularly
for aarch64-linux and armhf-linux. This change will offer a very slight
speedup for those substitutes that only bordeaux has.
Bordeaux has been a default substitute server for nearly 3 years now and I
think this change is overdue. I'm also hopeful that we'll be able to build on
the testing regarding mirrors for bordeaux, and that'll allow potentially
improving the hosting setup (through providing more redundancy) and further
improving substitute fetching for users who currently have issues with
substitute access.
* config-daemon.ac: Switch substitute urls order.
* doc/guix.texi: Ditto.
* etc/guix-install.sh: Ditto.
* gnu/installer/newt/network.scm (wait-service-online): Ditto.
* guix/store.scm (%default-substitute-urls): Ditto.
Change-Id: I4f6d93ae1fc8b03d80b47b18b5749a51f1fde17b
Signed-off-by: Christopher Baines <mail@cbaines.net>
|
|
This allows ‘guix substitute’ to gracefully handle errors like:
TLS error in procedure 'write_to_session_record_port': Error in the push function
instead of exiting (“`guix substitute' died unexpectedly”).
* guix/scripts/substitute.scm (download-nar)[try-fetch]: Catch
‘network-error?’ too.
Reported-by: Ada Stevenson <adanskana@gmail.com>
Change-Id: I91b92183b0165832645ee37d50c13445f9322525
|
|
Fixes <https://issues.guix.gnu.org/67575>.
* guix/scripts/substitute.scm (process-substitution/fallback): Use
‘report-error’ instead of ‘leave’. Write status line to PORT.
* tests/substitute.scm ("substitute, narinfo is available but nar is
missing"): Adjust accordingly.
Change-Id: Ic7297dbd563c007111ec2167c8d52505a07d4822
|
|
* guix/scripts/substitute.scm (%options): Handle EPIPE errors when displaying
help.
|
|
Fixes <https://issues.guix.gnu.org/63443>.
Reported by Attila Lendvai <attila@lendvai.name>.
* guix/scripts/substitute.scm (catch-system-error): New macro.
(download-nar): Add call to 'delete-file-recursively'.
* tests/substitute.scm ("substitute, previous partial download around"):
New test.
|
|
This allows 'guix substitute' to shrink the cache a bit more, which
saves space and improves performance of cache-cleanup phases since fewer
entries need to be traversed.
* guix/scripts/substitute.scm (cached-narinfo-expiration-time): Define
'max-ttl' and use it as an upper bound.
|
|
If a substitute server advertises in its narinfo, for example, both a
/zstd and a /lzip URL but the /zstd URL is unreachable, try the /lzip
URL.
Fixes <https://issues.guix.gnu.org/63634>.
* guix/narinfo.scm (narinfo-preferred-uris): New procedure.
(narinfo-best-uri): Rebase on top of it.
* guix/scripts/substitute.scm (download-nar)[try-fetch]: New procedure.
Use 'narinfo-preferred-uris' and 'try-fetch' to attempt all the URLs of
NARINFO.
* tests/substitute.scm (request-substitution): Remove 'parameterize'.
Delete DESTINATION.
("substitute, preferred nar URL is 404, other is 200"): New test.
|
|
Rethrowing with 'throw' doesn't work as intended when the exception
being rethrown is a SRFI-34 exception.
Fixes <https://issues.guix.gnu.org/55820>.
* guix/scripts/substitute.scm (kind-and-args-exception?): New variable.
(call-with-cached-connection): Rewrite using 'guard' instead of 'catch'
and 'raise' instead of 'throw'.
(system-error?): Use 'kind-and-args-exception?' instead of local
definition.
|
|
Previously, 'guix substitute' would fail abruptly with something like:
guix substitute: warning: while fetching https://ci.guix.gnu.org/nar/lzip/…-example: server is somewhat slow
guix substitute: warning: try `--no-substitutes' if the problem persists
guix substitute: error: connect*: Connection timed out
substitution of /gnu/store/…-example failed
* guix/scripts/substitute.scm (network-error?): Add ETIMEDOUT.
|
|
This was obtained by setting up this environment:
guix shell -D guix --with-input=guile@3.0.9=guile-next \
--with-commit=guile-next=e2ed33ef0445c867fe56c247054aa67e834861f2
-- make -j5
then adding 'unused-module' to (@@ (guix build compiler) %warnings),
building, and checking all the "unused module" warnings and removing
those that were definitely unused.
|
|
* guix/scripts/substitute.scm (find-daemon-option): Arrange to call
'daemon-options' only once.
|
|
* guix/scripts/substitute.scm (open-connection-for-uri/cached):
Fix docstring.
|
|
Fixes <https://issues.guix.gnu.org/57978>
Reported by Attila Lendvai <attila@lendvai.name>.
Previously, if a narinfo was available but its corresponding nar was
missing (for instance because the narinfo was cached and the server
became unreachable in the meantime), 'guix substitute --substitute'
would try to download the nar from its preferred location and abort when
that fails. This change forces one retry with each of the URLs.
* guix/scripts/substitute.scm (download-nar): Do not catch
'http-get-error?' exceptions.
(system-error?, network-error?, process-substitution/fallback): New
procedures.
(process-substitution): Call 'process-substitution/fallback' upon
'network-error?'.
* tests/substitute.scm ("substitute, first URL has narinfo but lacks nar, second URL unauthorized")
("substitute, first URL has narinfo but nar is 404, both URLs authorized")
("substitute, first URL has narinfo but nar is 404, one URL authorized")
("substitute, narinfo is available but nar is missing"): New tests.
|
|
* guix/scripts/substitute.scm (download-nar): New procedure, with most
of the code moved from...
(process-substitution): ... here. Call it.
|
|
* guix/scripts/substitute.scm (display-narinfo-data)
(open-connection-for-uri/cached)
(process-substitution): Use SRFI-71 instead of SRFI-11.
|
|
Running `guix archive --authorize` sets /etc/guix/acl to 600
via with-atomic-file-output via mkstemp!.
Then running `guix substitute --help/--version` fails on "permission denied".
While "guix substitute" is an internal tool, the options --help and --version
exist and you should be able to run those from the command line.
* guix/scripts/substitute.scm (guix-substitute): Earlier check for
--help or --version.
Signed-off-by: Ludovic Courtès <ludo@gnu.org>
|
|
* guix/scripts/substitute.scm (show-help): Accept exactly one OPTION.
|
|
The short option was listed in the help-text, but not recognized.
|
|
* guix/scripts/substitute.scm (call-with-cached-connection): Adjust
comment.
|
|
In addition to substitutes from ci.guix.gnu.org. There are more changes that
can be made in the future, but these changes seem like a good start.
* config-daemon.ac (guix_substitute_urls): Add https://bordeaux.guix.gnu.org.
* guix/scripts/substitute.scm (%default-substitute-urls): Add
http://bordeaux.guix.gnu.org.
* guix/store.scm (%default-substitute-urls): Add bordeaux.guix.gnu.org.
* doc/guix.texi: Adjust accordingly.
* doc/contributing.texi: Adjust accordingly.
|
|
The process-substitution procedure is opening two distinct connections. The
first one when looking for narinfo by calling lookup-narinfo and the other one
when fetching nar files.
Cache the connection when looking for narinfos so that process-substitution
only opens one connection.
* guix/scripts/substitute.scm (lookup-narinfo): Cache connection by using
open-connection-for-uri/cached.
|
|
The default Guile web server implementation supports the keep alive
mechanism. However, in our custom http-write implementation, the connection
is unconditionally close after sending nar files.
To prevent that, when supported, add the client port to the server poll set so
that further requests can be handled without closing the connection.
* guix/scripts/publish.scm (nar-response-port): Rename it into ...
(nar-compressed-port): ... this procedure. Operate directly on a given PORT.
(http-write): Add keep-alive support when sending nar files.
* guix/scripts/substitute.scm (process-substitution): Pass the download size
to the progress-report-port procedure so that it doesn't block reading from
the input port when keep-alive is supported.
|
|
Partly fixes <https://bugs.gnu.org/47867>.
Reported by Florian Pelz <pelzflorian@pelzflorian.de>.
In GnuTLS up to 3.7.1 included, GNUTLS_E_AGAIN and GNUTLS_E_INTERRUPTED
are not handled by 'write_to_session_record_port' and could be thrown at
the caller. This patch works around that by dropping connections
altogether and restarting when this happens.
* guix/http-client.scm (false-if-networking-error): Swallow ERROR/AGAIN
and ERROR/INTERRUPTED.
* guix/scripts/substitute.scm (call-with-cached-connection): Likewise.
|
|
This avoids the situation where error messages would unintentionally go
to stderr and be wrongfully interpreted as a reply by the daemon.
Fixes <https://bugs.gnu.org/46362>.
This is a followup to ee3226e9d54891c7e696912245e4904435be191c.
* guix/scripts/substitute.scm (display-narinfo-data): Add 'port'
parameter and honor it.
(process-query): Likewise.
(process-substitution): Likewise.
(%error-to-file-descriptor-4?, with-redirected-error-port): Remove.
(%reply-file-descriptor): New variable.
(guix-substitute): Remove use of 'with-redirected-error-port'. Define
'reply-port' and pass it to 'process-query' and 'process-substitution'.
* nix/libstore/build.cc (SubstitutionGoal::handleChildOutput): Swap
'builderOut' and 'fromAgent'.
* nix/libstore/local-store.cc (LocalStore::getLineFromSubstituter):
Likewise.
* tests/substitute.scm <top level>: Set '%reply-file-descriptor'
rather than '%error-to-file-descriptor-4?'.
|
|
The immediate effect is that, with '--verbosity=1', only two lines are
displayed for each substitute, instead of two lines followed by an empty
line.
* guix/scripts/substitute.scm (process-substitution): Emit a single
newline upon completion when PRINT-BUILD-TRACE? is true.
|
|
This is a follow-up of bc3896db25c788c181c7bcd65754e7cd378e9d9f.
* guix/scripts/substitute.scm (%local-substitute-urls): Test for "true"
instead of "yes".
|
|
This stems from the observation that substitute download can be
CPU-bound when high-speed networks are in use:
https://lists.gnu.org/archive/html/guix-devel/2020-12/msg00177.html
* guix/narinfo.scm (decompresses-faster?): New procedure.
(narinfo-best-uri): Add #:fast-decompression?.
* guix/scripts/substitute.scm (%prefer-fast-decompression?): New
variable.
(call-with-cpu-usage-monitoring): New procedure.
(with-cpu-usage-monitoring): New macro.
(display-narinfo-data, process-substitution): Pass #:fast-decompression?
to 'narinfo-best-uri'.
(process-substitution): Wrap 'restore-file' call in
'with-cpu-usage-monitoring'. Set '%prefer-fast-decompression?'.
|
|
The call-with-connection-error-handling was added in
20c08a8a45d0f137ead7c05e720456b2aea44402, but that error handling was
previously inside of open-connection-for-uri/maybe, which is related
to (call-)with-cached-connection which was used in process-substitution, but
only actually used with call-with-cached-connection when used in
fetch-narinfos.
There's some handling for similar errors within with-networking, which is used
within process-substitution.
* guix/scripts/substitute.scm (process-substitution): Remove
call-with-connection-error-handling call.
|
|
In f50f5751fff4cfc6d5abba9681054569694b7a5c, the way fetch was called within
process-substitution was changed. As call-with-cached-connection actually
includes important error handling for the opening of a HTTP request, this
change removed some error handling. This commit adds that back.
Fixes <https://bugs.gnu.org/47157>.
* guix/scripts/substitute.scm (call-with-cached-connection): New procedure.
(with-cached-connection): New syntax rule.
(process-substitution): Retry once for some errors when making HTTP requests
to fetch substitutes.
|
|
Rather than always outputting to (current-error-port) in
lookup-narinfos (which is called from within lookup-narinfos/diverse), take a
procedure which should return a progress reporter, and defer any output to
that.
As this is now general purpose code, make the default behaviour to output
nothing. Maintain the current behaviour of the substitute script by moving the
progress reporter implementation there, and passing it in when calling
lookup-narinfos/diverse.
These changes should be generally useful, but I'm particularly looking at
getting guix weather to do progress reporting differently, with this new
flexibility.
* guix/substitutes.scm (fetch-narinfos): Take a procedure to make a
progress-reporter, and use that rather than the hardcoded behaviour.
(lookup-narinfos): Add #:make-progress-reporter keyword argument, and pass
this through to fetch-narinfos.
(lookup-narinfos/diverse): Add a #:make-progress-reporter keyword argument,
and pass this through to lookup-narinfos.
* guix/scripts/substitute.scm (process-query): Pass a progress-reporter to
lookup-narinfos/diverse.
|
|
This means there's a module for working with substitutes, rather than all the
code sitting in the script. The need for this can be seen with the weather and
challenge scripts, that now don't have to use code from the substitute script,
but can instead use the substitute module.
The separation here between the actual functionality of the substitute script
and the underlying functionality used both there and elsewhere should make
maintenance easier moving forward.
This commit just moves code, none of the code should have been changed
significantly.
* guix/scripts/substitute.scm (%narinfo-cache-directory, %narinfo-ttl,
%narinfo-negative-ttl, %narinfo-transient-error-ttl, %unreachable-hosts): Move
variables to guix/substitutes.scm.
(narinfo-cache-file, cached-narinfo, cache-narinfo!, narinfo-request,
read-to-eof, call-with-connection-error-handling, fetch-narinfos,
lookup-narinfos, lookup-narinfos/diverse): Move procedures to
guix/substitutes.scm.
* guix/substitutes.scm: New file.
* Makefile.am: Add it.
* guix/narinfo.scm: Remove redundant module.
* guix/scripts/challenge.scm: Change (guix scripts substitute) to (guix
substitutes).
* guix/scripts/weather.scm: Change (guix scripts substitute) to (guix
substitutes).
|
|
Fixes <https://bugs.gnu.org/46362>.
Reported by Zhu Zihao <all_but_last@163.com>.
Previously, backtraces due to uncaught exceptions would always to go
file descriptor 2; the daemon would read it and error out with something
like:
error: got unexpected path `Backtrace:' from substituter
This patch fixes that by ensuring backtraces are properly displayed on
file descriptor 4.
* guix/scripts/substitute.scm (with-redirected-error-port): New macro.
(guix-substitute): Use 'with-redirected-error-port' instead of 'parameterize'.
|
|
This is similar to commit a168c3e4f8d580f70e1c26bcdfc5b8378b2fa42d.
* guix/scripts/substitute.scm (with-networking): Use
'with-throw-handler' instead of 'catch'.
|
|
This is part of trying to reduce the interdependency of code within the
substitute module.
This commit addresses some of the error handling that was performed through
open-connection-for-uri/maybe. The new approach is to use
call-with-connection-error-handling, and wrap calls to http-multiple-get and
http-fetch with that procedure, which takes care of handling connection
errors.
I think this is even slightly more rigerous than the previous setup, because
this approach handles connection errors that occur when http-multiple-get
reconnects to a host.
* guix/scripts/substitute.scm (open-connection-for-uri/maybe): Transform in to
call-with-connection-error-handling.
(fetch-narinfos): Use call-with-connection-error-handling.
(process-query): Replace open-connection-for-uri/maybe with
open-connection-for-uri/cached.
(open-connection-for-uri/cached): Set a default timeout, matching the
behaviour in open-connection-for-uri/maybe.
(process-substitution): Use call-with-connection-error-handling.
|
|
At least by default. Instead, make the open-connection procedure a parameter,
and make the default guix:open-connection-for-uri. Do so similarly for
lookup-narinfos and lookup-narinfos/diverse which work towards calling
fetch-narinfos.
This means this code can be moved to a different module, without having
use/move the connection caching code.
* guix/scripts/substitute.scm (fetch-narinfos): Add #:open-connection
argument, and call http-multiple-get with it.
(lookup-narinfos) Add #:open-connection argument, and call fetch-narinfos with
it.
(lookup-narinfos/diverse): Add #:open-connection argument, and call
lookup-narinfos with it.
(process-query): Call lookup-narinfos/diverse with #:open-connection
open-connection-for-uri/maybe.
|