Update Interval

Support requests, bug reports, etc. go here. Dedicated servers / VDS hosting only
Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Update Interval

Postby Striker21 » 02 Mar 2016, 14:44

Hi all,

I'm trying to wrap my head around update intervals. I've a default installation running on an Ubuntu 14.04 VPS with 4GB memory. I've started up update_daemon2.php and that seems to be running all fine, with multiple sub processes (at least two).

My update interval preference is 30 minutes and all my feeds are set to use the default interval. Now looking in my DB of about 900 feeds, the latest "last_update_started" indicates 11:32 as in the time of writing, in the bottom of the list I have 05:12. Now my VPS memory and CPU seems not to be stressing too much and updates of many sites seems to happen within a minute, while other times there can be a few minutes gap.

Is this all normal? Is there a way to work towards shortening the gap between the oldest update 05:12 and newest 11:32? 5h+ hours sound like a lot as some of my feeds are rather time sensitive certain times of the day I optimally need all feeds to be highly up-to-date.

Chris

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 02 Mar 2016, 15:00

start with the stickied threads

also, last_update_started is not the field you should be looking for (check "last_updated").

Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Re: Update Interval

Postby Striker21 » 02 Mar 2016, 17:11

Yeah, I thought I had started with sticky threads, but maybe not :) even did a search for update interval but could not find much, checked FAQ etc. but again maybe I missed something. I do sometimes miss the obvious.

In regards to the right DB field, the last_updated correspond partly with the other one logically. Right now the time difference on last_updated is from 08:17 this morning until 14.03 this afternoon, so there is a similar 5h gap (attached images from the DB jut in case). I'll continue to look for hints as to what I might have missed a sticky as to better understand this being normal or what I can do to lower the gap.

Expanding on my set-up just in case; VPS; Ubuntu 14.04.3, MySQL, mariaDB 10.1.11, TT-RSS v16.1 (from here) with soon 900 feeds .

tiny1start.jpg
tiny1start.jpg (184.98 KiB) Viewed 8975 times

tiny1end.jpg
tiny1end.jpg (192.19 KiB) Viewed 8975 times


Chris

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 02 Mar 2016, 18:03

what does it show in the UI when you hover over one of those feeds

also all timestamps in the database are UTC just in case

Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Re: Update Interval

Postby Striker21 » 02 Mar 2016, 18:26

UTC expected so that's all fine. If I hover over the feeds in the UI it shows similar times, right now while hovering over a selection of 20-30 feeds I see times from 10:30 to 14:30 which is 50 minutes ago at the moment. Double clicking the 10:30 one (Yahoo News) updated it with some 13 new entries that happened after 10:30 so my challenge is the rather big delay of receiving and processing them. Thoughts?


Added some webmin dashboard snapshots in case of relevance as I'm no linux pro as to spot if something is wrong. Though it is running clearly, and updating, just taking a too long :)
Attachments
tiny2.JPG
tiny2.JPG (45.95 KiB) Viewed 8922 times
tiny1.JPG
tiny1.JPG (45.21 KiB) Viewed 8922 times
tiny4.JPG
tiny4.JPG (88.52 KiB) Viewed 8922 times

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 02 Mar 2016, 18:54

wait

50 minutes ago or 5 hours ago? this is an important difference

check the feed which has a minimum value of last_updated

Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Re: Update Interval

Postby Striker21 » 02 Mar 2016, 19:36

That was just random hovering some feeds, so yes some of them was updated 50 minutes ago and some others 2-3-4-5 hours ago at that point in time (which matches what the DB indicates update timing wise for me).
Last edited by Striker21 on 02 Mar 2016, 19:44, edited 2 times in total.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 02 Mar 2016, 19:42

well, tbh i dunno

you do have a lot of feeds but that technically shouldn't break things

maybe default of 2 worker tasks is not enough to timely process everything because of some other bottleneck (network or cpu), you can try increasing worker amount by adding define('MAX_JOBS', 4) to config.php while decreasing batch limit by a reasonable amount (DAEMON_FEED_LIMIT, default 500). try playing around with those two values and see if it helps.

Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Re: Update Interval

Postby Striker21 » 02 Mar 2016, 19:44

And just to add: the UI says 11:52 on my oldest, and the DB also says 11:52, so no mismatch. So right now its processing feeds that have not been checked for about 4 hours as it looks and it never does not close that gap (as to get closer to a 30 minute ratio).

Update: okay, will try to set 4 and play with that number. The VPS does not seems to be stressing as far as I can judge, neither CPU or Memory wise as you can see from the screens I attached to some degree. I rarely see the CPU peak sometimes at 40% just to be down again a second later.

Does the changes of workers etc. require me to restart anything?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 02 Mar 2016, 19:52

you have to restart the daemon, yes

>The VPS does not seems to be stressing as far as I can judge,

the idea here is to add parallelism thus smaller batches, more workers

Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Re: Update Interval

Postby Striker21 » 02 Mar 2016, 20:25

Q: Feeds with errors, do they keep retrying more often than others as to potentially be causing frequent slowdowns or how are they processed?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 02 Mar 2016, 20:28

no

Striker21
Bear Rating Trainee
Bear Rating Trainee
Posts: 42
Joined: 27 Oct 2015, 00:30

Re: Update Interval

Postby Striker21 » 03 Mar 2016, 00:25

I upped it to 5 workers and 225 feeds, well not sure I did it right... added this to the config. Restarted and still felt it was just two workers, so changed the default in the update_daemon2.php file to 5 though did not find a similar variable for the feed_limit. Restarted again at there was at least 5 workers now. They have been running for hours and it seems the gap now is about 2h in total between oldest and newest update.

Code: Select all

   define('MAX_JOBS', '5');
   define('DAEMON_FEED_LIMIT', '225');


Would love to figure this out further, I take it that in normal circumstances even 2 workers should have managed and definitively 5, though that there is something affecting my system that is not normal.

rcaputo
Bear Rating Trainee
Bear Rating Trainee
Posts: 6
Joined: 22 Feb 2016, 06:14

Re: Update Interval

Postby rcaputo » 03 Mar 2016, 04:32

Sounds like every connection is involving a 60-90 second timeout. This sort of thing can happen if your TTRSS host's resolver returns IPv6 addresses first but only routes IPv4. TTRSS' updater may be waiting for the IPv6 connections to fail before falling back to IPv4.

The solution would be to configure your resolver to prefer IPv4 addresses. Or to fix your IPv6 routing.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: Update Interval

Postby fox » 03 Mar 2016, 12:12

btw i find the lack of daemon logs in this thread disturbing


Return to “Support”

Who is online

Users browsing this forum: No registered users and 2 guests