Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Support requests, bug reports, etc. go here. Dedicated servers / VDS hosting only
hosemn
Bear Rating Trainee
Bear Rating Trainee
Posts: 2
Joined: 12 Aug 2015, 18:27

Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby hosemn » 12 Aug 2015, 19:23

My instance of tt-rss was not properly following the "Krebs On Security" feed (http://krebsonsecurity.com/feed/), which it had no trouble following for a couple of years before. I checked the status of the feed in Preferences, and the error it was getting was: file_get_contents(http://www.krebsonsecurity.com/feed/): failed to open stream: HTTP request failed!

Side note: I was a little disappointed that tt-rss didn't make more of an effort to notify me that the feed was failing to parse. It wasn't showing up red in my tree on the left since I had read all of the articles and had hidden feeds which had zero unread items. Since it had zero unread items, it didn't show up in the list so I couldn't see it was failing. That's a separate issue, and more of a UX discussion than a technical "it's broken" discussion. Back to our regularly scheduled bug report:

I was (and am) able to read the feed itself through my browser, and also from curl at the command line of my server that serves my instance of tt-rss, so I believed it wasn't a routing, DNS, or (Krebs's) server-side issue. I investigated with Wireshark and ZAP, and from what I could tell, it looked like Krebs's nginx reverse proxy was refusing connections from me only when my clients made requests using HTTP/1.0 protocol. This was 100% reproducible.

So, I looked around, and apparently the default behavior of file_get_contents() is to use HTTP/1.0 protocol unless the context specifies to use protocol version HTTP/1.1.

I modified the file /include/functions.php to adjust the code inside the fetch_file_contents() function, changing this:

Code: Select all

if (!$post_query && $timestamp) {
    $context = stream_context_create(array(
        'http' => array(
            'method' => 'GET',
            'header' => "If-Modified-Since: ".gmdate("D, d M Y H:i:s \\G\\M\\T\r\n", $timestamp)
        )));
}
 else {
    $context = NULL;

into this:

Code: Select all

if (!$post_query && $timestamp) {
    $context = stream_context_create(array(
        'http' => array(
            'method' => 'GET',
            'protocol_version'=> 1.1,
            'header' => "If-Modified-Since: ".gmdate("D, d M Y H:i:s \\G\\M\\T\r\n", $timestamp)
        )));
}
 else {
    $context = stream_context_create(array(
        'http' => array(
            'method' => 'GET',
            'protocol_version'=> 1.1
        
)));


This resolved my issue, and didn't break any of my other feeds. I have only 57 feeds, though, so I don't know if that's a solid enough smoke test to say this is a robust solution. I believe that some fault does lie with Krebs's hosting or IT staff for configuring a reverse proxy to actively reject (with a friggin TCP RST) clients that want to talk HTTP/1.0, but at the same time I'm not sure how angry we can be at them since HTTP/1.1 has been around since 1999 and HTTP/2 is coming down the pike.

Do you see any issue with adjusting the tt-rss code base to have it actively command PHP to use HTTP/1.1 when using file_get_contents() to fetch a URL?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby fox » 12 Aug 2015, 19:42

this makes sense although you really should be using curl

format this change as a git format-patch and attach it here, i'll apply it

either that or gitlab etc

>Side note: I was a little disappointed that tt-rss didn't make more of an effort to notify me that the feed was failing to parse. It wasn't showing up red in my tree on the left since I had read all of the articles and had hidden feeds which had zero unread items. Since it had zero unread items, it didn't show up in the list so I couldn't see it was failing. That's a separate issue, and more of a UX discussion than a technical "it's broken" discussion. Back to our regularly scheduled bug report:

i could probably add a visible knob somewhere although im not sure if its such a good idea because feed fetching may just fail intermittently and tt-rss doesn't count sequential update errors

JustAMacUser
Bear Rating Overlord
Bear Rating Overlord
Posts: 373
Joined: 20 Aug 2013, 23:13

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby JustAMacUser » 12 Aug 2015, 21:19

What if feeds with errors always appeared in the feed lists, even if "hide with no unread" was checked? That way people would know and the UI wouldn't really need any other changes.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby fox » 12 Aug 2015, 21:25

what if the category is collapsed

i dunno

User avatar
HeikoAdams
Bear Rating Master
Bear Rating Master
Posts: 101
Joined: 19 Mar 2013, 00:17

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby HeikoAdams » 12 Aug 2015, 21:43

I felt free to create the attached patch
Attachments
use_http11.patch
Patch to use http 1.1
(879 Bytes) Downloaded 75 times

JustAMacUser
Bear Rating Overlord
Bear Rating Overlord
Posts: 373
Joined: 20 Aug 2013, 23:13

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby JustAMacUser » 12 Aug 2015, 21:44

Oh, yeah. Plus, like hosemn, I don't have a lot of feeds. But if a user has a lot of feeds it serves to reason that there will always be feeds with errors, which could be annoying.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby fox » 12 Aug 2015, 21:47

i think that's how it worked at some point btw now that i think about it, maybe this was lost with the switch to dojo

i did a quick change and it seems to work ok, although i didn't test the nested categories

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby fox » 12 Aug 2015, 21:48

HeikoAdams wrote:I felt free to create the attached patch


cool

hosemn
Bear Rating Trainee
Bear Rating Trainee
Posts: 2
Joined: 12 Aug 2015, 18:27

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby hosemn » 12 Aug 2015, 22:43

HeikoAdams, thanks for the patch.

Fox, I think there are a few options for alerting the user to failed feeds, which you've already started to explore:

1. Show failed feeds in the tree even if all items are read (I believe you made mention earlier to already putting a commit in to move this functionality into production, no?)

Cons: You're right, this could get annoying for users with many many feeds, as there is a high probability that at any given moment a feed will have some transient error.

2. Maybe show a button in the top navigation bar, and it could only show up when one or more feeds are having an error, and you could click on it to bring up the "Feeds with errors" window and see what's up.

Cons: Again, for users with many feeds, this might always be present, and could still get annoying.

3. Track feed failures on a short-term basis and if a feed fails some number of times in a row (5, 10, 20) only then have the feed show up in the tree marked "red" for error or bring up the "Feeds with errors" button in the top pane.

Cons: Might require a new database table, and definitely requires more code, so I'm not going to bitch too hard about asking for something like this. For real. I've been using tt-rss since Google Reader was canned, and this is like the first time I ever had a problem that I had to work for more than 15 minutes to figure out. I'm impressed.

Lastly, Fox, I didn't realize that curl was recommended. I've been using for a while without curl, so it just never occurred to me. It's also not mentioned in the installation notes, the curl extention for php wasn't a default option from my distro, and tt-rss worked out of the box without it. Thanks for pointing out that I should switch. FWIW, I grabbed the curl extension for php from my distros repo now, but I still thank you for considering my bug and already getting the updated code into the codebase. A quick git reset --hard and a git pull and I'm right back to the bleeding edge. You're the man.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Use of HTTP/1.0 interfering with retrieving a feed (with possible solution)

Postby fox » 12 Aug 2015, 22:48

1. i think keeping error'd feeds visible is a good enough compromise which doesn't add new ui elements.

2. i think it's out on the wiki there somewhere that curl is recommended. it's not really a big deal. some plugins depend on curl, that's about it.

e: https://tt-rss.org/gitlab/fox/tt-rss/wi ... ilityNotes here i guess


Return to “Support”

Who is online

Users browsing this forum: No registered users and 5 guests