packages/lftp
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Attempt to fix SFTP Too many out-of-order packets.

Browse Source
This commit is contained in:
liningjie 2024-09-29 11:01:48 +08:00
parent cd37e0ae32
commit 8d42936ac2
2 changed files with 82 additions and 1 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

74
backport-Attempt-to-fix-SFTP-Too-many-out-of-order-.patch Normal file
View File

@ -0,0 +1,74 @@
From fdb81537a2f854cf5e2b9dd95c7e5542bb5cd420 Mon Sep 17 00:00:00 2001
From: Paul Dennis <padenn@github.com>
Date: Mon, 29 Jul 2024 16:43:02 -0400
Subject: [PATCH] Attempt to fix SFTP issue #636 Too many out-of-order packets.
lftp supports making many SFTP DATA requests in parallel when downloading a file. Responses
with data payloads from an sftp server can sometimes arrive in a different order from the
original requests.
The out of order data payload (packets) are added to the `ooo_chain` to be processed when the expected
(but late) next requested packet arrives.
A single packet being delayed while many parallel requests continue to be made can sometimes result in the
`Too many out-of-order packets` message if the `ooo_chain` buffer of 64 entries is exceeded.
This fix checks the remaining capacity of the `ooo_chain` and pending requests (`RespQueueSize()`)
and will avoid sending new requests for more data until the `ooo_chain` is processed or `RespQueueSize()`
shrinks. Adding this constraint does mean that `sftp:max-packets-in-flight` will be limited to
the current `ooo_chain` length of 64.
I was able to reliably reproduce the issue with a 4gb download and verify the fix.
---
src/SFtp.cc | 8 ++++++--
src/SFtp.h | 1 +
2 files changed, 7 insertions(+), 2 deletions(-)
diff --git a/src/SFtp.cc b/src/SFtp.cc
index 6c818dfa..5879a631 100644
--- a/src/SFtp.cc
+++ b/src/SFtp.cc
@@ -344,6 +344,7 @@ void SFtp::Init()
size_write=0x8000;
use_full_path=false;
flush_timer.Set(0,500);
+ max_out_of_order=64;
}
SFtp::SFtp() : SSH_Access("SFTP:")
@@ -892,7 +893,7 @@ void SFtp::HandleExpect(Expect *e)
{
LogNote(9,"put a packet with id=%d on out-of-order chain (need_pos=%lld packet_pos=%lld)",
reply->GetID(),(long long)(pos+file_buf->Size()),(long long)r->pos);
- if(ooo_chain.count()>=64)
+ if(ooo_chain.count()>=max_out_of_order)
{
LogError(0,"Too many out-of-order packets");
Disconnect();
@@ -1192,7 +1193,10 @@ int SFtp::Read(Buffer *buf,int size)
{
// keep some packets in flight.
int limit=(entity_size>=0?max_packets_in_flight:max_packets_in_flight_slow_start);
- if(RespQueueSize()<limit && !file_buf->Eof())
+ int ooo_queue_available=max_out_of_order-ooo_chain.count();
+ int current_in_flight=RespQueueSize();
+ if(RespQueueSize()<limit && !file_buf->Eof()
+ && current_in_flight < ooo_queue_available)
{
// but don't request much after possible EOF.
if(entity_size<0 || request_pos<entity_size || RespQueueSize()<2)
diff --git a/src/SFtp.h b/src/SFtp.h
index 50888b37..2099440c 100644
--- a/src/SFtp.h
+++ b/src/SFtp.h
@@ -755,6 +755,7 @@ private:
int size_read;
int size_write;
bool use_full_path;
+ int max_out_of_order;
protected:
void SetError(int code,const Packet *reply);
--
2.33.0

9
lftp.spec
View File

@ -1,7 +1,7 @@
Summary: A sophisticated file transfer program
Name: lftp
Version: 4.9.1
Release: 4
Release: 5
License: GPLv3+
URL: http://lftp.yar.ru/
Source0: http://lftp.yar.ru/ftp/%{name}-%{version}.tar.xz
@ -12,6 +12,7 @@ Requires: %{name}-help
Patch0: lftp-4.0.9-date_fmt.patch
Patch1: quit-while-source-file-increased.patch
Patch2: backport-Attempt-to-fix-SFTP-Too-many-out-of-order-.patch
%description
LFTP is a sophisticated file transfer program supporting a number of
@ -85,6 +86,12 @@ echo "%{_libdir}/lftp/%{version}" > %{buildroot}/etc/ld.so.conf.d/%{name}-%{_arc
%changelog
* Sun Oct 13 2024 liningjie <liningjie@xfusion.com> - 4.9.1-5
- Type:bugfix
- ID:NA
- SUG:NA
- DESC:Attempt to fix SFTP Too many out-of-order packets.
* Fri Oct 29 2021 zengweifeng <zwfeng@huawei.com> - 4.9.1-4
- Type:bugfix
- ID:NA