Feed entries are always re-imported

Support requests, bug reports, etc. go here. Dedicated servers / VDS hosting only
feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Feed entries are always re-imported

Postby feader » 14 May 2013, 21:22

The feed http://www.iswintercoming.com/feed.php?f=5 shows a strange behaviour: Its entries are always re-imported. Before I reported it here, I checked the feed data with curl, and saw something that could have caused it

Code: Select all

$ curl http://www.iswintercoming.com/feed.php?f=5
[…]
<id>http://www.iswintercoming.com/viewtopic.php?t=1137&amp;p=44684&amp;sid=011f5f6529a1591eacc99af1037ff535#p44684</id>
[…]
$ curl http://www.iswintercoming.com/feed.php?f=5
[…]
<id>http://www.iswintercoming.com/viewtopic.php?t=1137&amp;p=44684&amp;sid=086ae08efa935c6b5a6054bc00bc29f7#p44684</id>
[…]

So, best course of action is to tell the content provider that sids in feed ids are a bad idea, or am I missing something and should investigate further?
Last edited by feader on 14 May 2013, 22:47, edited 1 time in total.

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Feed entries are always re-imported

Postby craywolf » 14 May 2013, 22:07

The feed doesn't supply a GUID for entries. If the GUID is missing, Link is used instead. So if the Link changes, it will be seen as a new and different entry. The feed author needs to either supply a GUID or use links that don't change.

The sid parameter disappeared from the feed as I was refreshing it, so I guess you contacted the site and they fixed it?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Feed entries are always re-imported

Postby fox » 14 May 2013, 22:10

Yeah the feed is broken in the worst possible way imaginable. Great job to whoever coded the generator (lol session id, seriously).

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: Feed entries are always re-imported

Postby feader » 14 May 2013, 22:51

craywolf wrote:The sid parameter disappeared from the feed as I was refreshing it, so I guess you contacted the site and they fixed it?

:?: I still see the sid (so many sids in that feed, must be a significant percentage of the content :shock: ), but I have contacted the administrator right now. We'll see what happens.

craywolf
Mr. Awesome
Posts: 97
Joined: 19 Mar 2013, 18:07

Re: Feed entries are always re-imported

Postby craywolf » 15 May 2013, 01:33

A quick test (by loading the URL in an Incognito window) shows the SIDs are present on an initial load of the page, but are no longer present on refresh. The page also sets several cookies. If the cookies are deleted and the page refreshed, the SIDs reappear. They disappear again on reload.

Since curl doesn't support cookies, the page will always contain a SID when retrieved that way.

What a fucked up feed.

Worst case, you can probably write up a plugin that strips the SID from the link attribute.

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: Feed entries are always re-imported

Postby feader » 18 May 2013, 04:42

So, since these guys probably don't even know what a feed is, I decided to create a plugin. Since I already found another use case, I may as well try to remove the hard coded values from it :mrgreen:

A few questions that arised:
    • Since the plugin whose code I "borrowed" implements IHandler, mine also does (in a very similar way 8) ). What is this good for, and do I need to implement it?
    • I hooked in at HOOK_FEED_FETCHED, which I guess I must (?), so what is a nice and clean way to get the feed URL so I can match against it? I could try to use the feed parser, or is there a nice RegExp out there I could use for that?
    • Is logging available for plugins, or is it better to do this on your own?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Feed entries are always re-imported

Postby fox » 18 May 2013, 09:24

1. IHandler is a thing that responds to http requests from the user, you shouldn't need to implement that.

2. I guess it's a valid point. I'll add the necessary stuff so it would be easier:

https://github.com/gothfox/Tiny-Tiny-RS ... 062fab3e64

3. Sure, you should be able to invoke Logger just like every other part of tt-rss, you can check include/errorhandler.php for usage example. Or you can use user_error().

feader
Bear Rating Master
Bear Rating Master
Posts: 160
Joined: 26 Dec 2012, 20:03

Re: Feed entries are always re-imported

Postby feader » 20 May 2013, 15:06

Thank you, this one is just for you, so would everybody else please look away:
Image
I still have a few questions though. Does this

Code: Select all

if ($override_url) $fetch_url = $override_url;

actually happen, and if so, how can the user find out what $override_url is?
With the logging, by making an embarrassing mistake I noticed that I won't have to do much since much is already caught by ttRSS. (BTW, how do you do this in PHP, do you catch exceptions at some point, or plug into signal handlers?). So all I have to notice & report at the moment is probably malformed JSON or an error with reg exes. I thought it would be a nice touch if I reported activity with some

Code: Select all

_debug("...", $debug_enabled);

or is it better if I didn't?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Feed entries are always re-imported

Postby fox » 20 May 2013, 15:19

That's quite a bear you got there.

>actually happen, and if so, how can the user find out what $override_url is?

Going by invocations of update_rss_feed() it seems that it's a pale remnant of some long since removed thing, so you shouldn't worry about it. I'll remove the parameter from trunk.

>BTW, how do you do this in PHP, do you catch exceptions at some point, or plug into signal handlers

lol exceptions in php


Return to “Support”

Who is online

Users browsing this forum: No registered users and 5 guests