1.9 to 1.10 - update performance regression

Support requests, bug reports, etc. go here. Dedicated servers / VDS hosting only
paulez
Bear Rating Trainee
Bear Rating Trainee
Posts: 3
Joined: 26 Sep 2013, 21:43

1.9 to 1.10 - update performance regression

Postby paulez » 26 Sep 2013, 21:50

Hi all,

I am running TT-RSS on a Amazon EC2 t1.micro instance (the smallest one). It is not powerfull but tt-rss was running fine on it.

I upgraded from version 1.9 to 1.10 and now the update daemon is taking all the CPU time (100% cpu usage for many days on monitoring, it was around 40% before).

There was no other change at that time, and the high cpu usage occurred just after the upgrade. I am using two workers for the update daemon (with update_daemon2.php). And they both take 50% of CPU time.

I had to change DAEMON_SLEEP_INTERVAL to a larger value (18000) to let my instance breathe and give some cpu time to other processes.

If you have a way to profile the update daemon I can try.

Any idea on what could cause this ?

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: 1.9 to 1.10 - update performance regression

Postby fox » 26 Sep 2013, 22:07

e: general lol @ "my software uses cpu while working, oh woe is me whatever am I going to do"

don't update feeds?

e: see also http://tt-rss.org/redmine/issues/779 I guess

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: 1.9 to 1.10 - update performance regression

Postby sleeper_service » 26 Sep 2013, 23:03

paulez wrote:I upgraded from version 1.9 to 1.10 and now the update daemon is taking all the CPU time (100% cpu usage for many days on monitoring, it was around 40% before).


I don't see any change in the average cpu usage on my system from before and after I updated to the current trunk, or indeed for quite a period before.

take that for what it's worth.

AngryChris
Bear Rating Master
Bear Rating Master
Posts: 135
Joined: 08 Apr 2013, 02:42

Re: 1.9 to 1.10 - update performance regression

Postby AngryChris » 26 Sep 2013, 23:36

I've not monitored CPU usage, but the fan in my server goes high speed a lot more often now, concurrent with feed update runs. I've wondered why and now I know! (I have no plans to "fix" it, however).

Pentium 4 HT 3.00 GHz, 3.5 GB of RAM, Ubuntu Server 12.04.3. I don't know how that compares to a Raspberry Pi. Yes, I know the hardware is old, but it still runs like a champ and does what I need.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: 1.9 to 1.10 - update performance regression

Postby fox » 26 Sep 2013, 23:54

When the daemon is using cpu it is either parsing (while using all available cpu time) or waiting on the network. If your network works faster or caching is more efficient, it will use use the cpu more because it can parse more feeds per minute.

In the end though if averaged over the longer period the amount of cpu time used should be about the same because the daemon still has to process the same amount of feeds. If it can go faster, the exact same batch will be processed faster (and it will go to sleep faster), that's the only difference there is.

You can probably make it go slower but I don't really see any purpose in it unless you actively dislike your feeds being updated in a timely fashion.

xtaz
Bear Rating Master
Bear Rating Master
Posts: 174
Joined: 24 Dec 2009, 16:48

Re: 1.9 to 1.10 - update performance regression

Postby xtaz » 27 Sep 2013, 00:02

Other than the language detection that fox mentioned, is it possible this kind of thing is now being reported because of the changes to the MySQL schema to change three of the tables to InnoDB? Maybe on crap VMs or shared hosting the engine change now means the update daemon is waiting for MySQL to get round to doing something. Solution is to use PostgreSQL....

AngryChris
Bear Rating Master
Bear Rating Master
Posts: 135
Joined: 08 Apr 2013, 02:42

Re: 1.9 to 1.10 - update performance regression

Postby AngryChris » 27 Sep 2013, 00:25

I'm using PostgreSQL and the aforementioned server where the CPU fans are running more often. If the updater is doing language detection, it will consume more CPU time as it will take longer to run (time to parse XML + load into database vs. time to parse XML + detect language + load into database). It's likely that the the fans come on because the CPU load remains at 100% for a longer period of time, allowing more heat to be generated. It doesn't bother me at all, and if it did, I would just move the machine to the floor (it's sitting on the other end of my table here).

I would guess the 3 caching tables updated to InnoDB in the recent release are having no impact whatsoever.

paulez
Bear Rating Trainee
Bear Rating Trainee
Posts: 3
Joined: 26 Sep 2013, 21:43

Re: 1.9 to 1.10 - update performance regression

Postby paulez » 27 Sep 2013, 01:29

I use Postgresql on a separate instance, so database performance issue shouldn't impact update daemon cpu (if it is not doing active wait on db queries).

The cpu usage increase was definitely triggered by the update, is quite obvious in the graphs.

Image

And the update logs :

Code: Select all

Start-Date: 2013-09-24  19:11:11
Commandline: apt-get dist-upgrade
Install: init-system-helpers:amd64 (1.11, automatic)
Upgrade: tt-rss:amd64 (1.9+dfsg-1, 1.10+dfsg-1)
End-Date: 2013-09-24  19:11:57


I known that EC2 t1.micro is not really fast, but as you can get it for free a lot of people are using, so if tt-rss works on it perfectly (as it did before) that may be great for a lot of people !

I am totally willing to help if you have some pointers to find what is consuming this much cpu time, as I am not a php hacker, I know only xdebug but I don't know how to use it on a standalone daemon.**

I'll try to disable language detection to see if it improves the situation.

AngryChris
Bear Rating Master
Bear Rating Master
Posts: 135
Joined: 08 Apr 2013, 02:42

Re: 1.9 to 1.10 - update performance regression

Postby AngryChris » 27 Sep 2013, 08:01

I did some log parsing looking for the duration of all update runs using task 0. I grab the PID of that task and then look for the first and last timestamps that task appears in the log for each run. I then calculate a seconds duration between them. I then used these durations to calculate an average number of seconds per run of task 0 pre and post upgrade.

On version 1.9, the average run time per cycle for task 0 was 25.577 seconds.
On version 1.10, the average run time per cycle for task 0 is 40.796 seconds.

This is an increase run time per cycle of 15.219 seconds or 59.502%. If it's safe to assume that over the course of a day, work is evenly distributed between task 0 and task 1, the overall amount of time it takes the updater to complete has increased by about 60%. If language detection is highly CPU intensive, and it's spending ~37% of its time in that task, then the overall CPU utilization will have increased by a much greater degree. Since I don't have metrics captured from before the upgrade for CPU utilization, I can only go by the length of time it takes each update cycle to complete with the assumption that the extra time is spent using the new language parsing feature.

Again, this is on a Pentium 4 HT 3.00 GHz, 3.5 GB of RAM, Ubuntu Server 12.04.3, PostgreSQL 9.1.9.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: 1.9 to 1.10 - update performance regression

Postby fox » 27 Sep 2013, 08:23

Well, the only thing language detection is used for is proper hyphenation, which is not critical or anything. I guess I can rethink #779 and add a knob for it to be disabled.

e: so if you disable detection the speed picks up? I don't remember changing anything else in the update process.

e2: for a comparison, on my tt-rss system, detecting language for a typical long post takes 0.007s tops.

hrk
Bear Rating Disaster
Bear Rating Disaster
Posts: 75
Joined: 24 Apr 2013, 12:39

Re: 1.9 to 1.10 - update performance regression

Postby hrk » 27 Sep 2013, 10:52

fox wrote:the only thing language detection is used for is proper hyphenation


Image

What the... is hyphenation used for in the first place? :shock:

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: 1.9 to 1.10 - update performance regression

Postby fox » 27 Sep 2013, 11:35

because it makes text easier to read, duh

Sent from my GT-I9300 using Tapatalk 4

hrk
Bear Rating Disaster
Bear Rating Disaster
Posts: 75
Joined: 24 Apr 2013, 12:39

Re: 1.9 to 1.10 - update performance regression

Postby hrk » 27 Sep 2013, 12:21

fox wrote:because it makes text easier to read, duh


... err, ok, that's personal taste and off topic. I really meant: what's the use of hyphenation in TT-RSS? Language detection during update, so that when browsing the web-interface articles get changed to a hyphenated version of themselves? Based on what width? Window width? Span width read through jQuery? :shock: I haven't had the chance to update to 1.10 yet, so I couldn't notice anything.

User avatar
fox
^ me reading your posts ^
Posts: 6318
Joined: 27 Aug 2005, 22:53
Location: Saint-Petersburg, Russia
Contact:

Re: 1.9 to 1.10 - update performance regression

Postby fox » 27 Sep 2013, 12:47

It is used when rendering article text in browsers compatible with css hyphens property (chrome has this broken, everything else works I think).

paulez
Bear Rating Trainee
Bear Rating Trainee
Posts: 3
Joined: 26 Sep 2013, 21:43

Re: 1.9 to 1.10 - update performance regression

Postby paulez » 27 Sep 2013, 13:16

I just disable language detection and CPU usage is back to normal.

Code: Select all

361,362c361,362
<       //$lang = new Text_LanguageDetect();
<       //$lang->setNameMode(2);
---
>       $lang = new Text_LanguageDetect();
>       $lang->setNameMode(2);
575c575
<             /*$entry_language = $lang->detect($entry_title . " " . $entry_content, 1);
---
>             $entry_language = $lang->detect($entry_title . " " . $entry_content, 1);
584,585c584
<             }*/
<             $entry_language = "";
---
>    


Return to “Support”

Who is online

Users browsing this forum: No registered users and 3 guests