How To Use Trunk Version?

Development-related discussion, including bundled plugins
gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 08 Jun 2013, 00:23

I note that running 'update.php --update-schema' when an update proves necessary now prompts you to backup your database and "Type 'yes' to continue.".


That's good to know... and reinforces my belief that there isn't much to be gained by including additional automation there.

I did decide to play around with a match/merge on config.php-dist and config.php --- just cleaning it up now. Worse case if something is screwed up you just need to go in
and manually edit - versus the schema database voodoo...

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 08 Jun 2013, 09:14

http://tso.bzb.us/2013/06/update-script ... rsion.html

OK, I've posted a revision which includes logfiles and schema/config.php change notifications. I've also added an additional script which does a backup and update to config.php. It creates a temporary baseline of config.php-dist and uses it for a reference point after the git pull; in addition to grabbing the local modifications from the existing config.php. The new config.php-dist is considered authoritative and will override the local changes if an item in question doesn't match the temporary baseline. That will probably happen extremely rarely, but if it does, you'll get a nudge to manually fix things. If the baseline matches config.php-dist after the update, then local modifications will be preserved and carried forward to the new config.php.

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: How To Use Trunk Version?

Postby sleeper_service » 08 Jun 2013, 12:44

gbcox wrote:http://tso.bzb.us/2013/06/update-script-for-ttrss-trunk-version.html

OK, I've posted a revision which includes logfiles and schema/config.php change notifications.


I'm curious, as I am at times, after checking and finding that there's a config.php update, or a schema update, why do you try and start the updater service? cuz, it won't run when there's been a change to the config or schema. shouldn't that be a conditional?

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 08 Jun 2013, 21:14

I'm curious, as I am at times, after checking and finding that there's a config.php update, or a schema update, why do you try and start the updater service? cuz, it won't run when there's been a change to the config or schema. shouldn't that be a conditional?


Yeah, if config.php changes, then services should be restarted. I checked and I am restarting after the cfg_o_matic call, but if folks don't have it, then I can add a message telling them if config.php requires changes, they gotta restart services.

Not sure on the schema though... on the last change I had the update service already going and it seemed to work fine. Does the schema update change anything in the directory? I thought that only affected the db, and if that is the case why would you require the service to restart?

Athanasius
Bear Rating Trainee
Bear Rating Trainee
Posts: 38
Joined: 02 Apr 2013, 21:01

Re: How To Use Trunk Version?

Postby Athanasius » 08 Jun 2013, 23:41

In my mind it's a case of not wanting the update service to be adding/changing data during any schema update, yes, even if the latter locks things. It just seems safer to not risk any race conditions/corruption there.

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 08 Jun 2013, 23:47

Athanasius wrote:In my mind it's a case of not wanting the update service to be adding/changing data during any schema update, yes, even if the latter locks things. It just seems safer to not risk any race conditions/corruption there.


I don't believe that can happen. If a schema update is pending, I believe you must complete the schema update before you can access the database.

Here is the code from update_daemon2.php :
if ($schema_version != SCHEMA_VERSION) {
die("Schema version is wrong, please upgrade the database.\n");

But, it is a bit sloppy to start it if it isn't going to do anything anyway. I'll change it to exclude it if there is a schema update and put in a message that it needs to be done manually.
I also removed Sphinx from automatic start if there is a schema update. If anyone thinks a index rebuild is also required, let me know - but I'm thinking not.

Thanks again for all the feedback, I sure there is something else I haven't considered.

Athanasius
Bear Rating Trainee
Bear Rating Trainee
Posts: 38
Joined: 02 Apr 2013, 21:01

Re: How To Use Trunk Version?

Postby Athanasius » 09 Jun 2013, 02:06

gbcox wrote:
Athanasius wrote:In my mind it's a case of not wanting the update service to be adding/changing data during any schema update, yes, even if the latter locks things. It just seems safer to not risk any race conditions/corruption there.


I don't believe that can happen. If a schema update is pending, I believe you must complete the schema update before you can access the database.

Here is the code from update_daemon2.php :
if ($schema_version != SCHEMA_VERSION) {
die("Schema version is wrong, please upgrade the database.\n");

More to the point, now I check the code, update.php, which update_daemon2.php calls as needed in its infinite loop also does this schema check. Still, you could get past that check before the update and be in the middle of other code just as something in the schema update causes DB changes. I'll stick with not relying on the underlying code being always 100% correct. It's not like it's difficult to stop and start the daemon.

My own upgrade scheme is to run a chain of shell commands as root, which basically just stops the daemon, su's to tt-rss, and then restarts the daemon. And at the stage it's su'd I run a script that does some git manipulation (I still use my own separate local branch which keeps my config changes and any out-of-tree plugins and themes), and once that is done it does a schema update. So if I get the 'yes' prompt from that I can then separately run a quick (extra, I have 30 days worth of daily anyway) backup, 'yes' the prompt and continue. Log out of tt-rss once that's done and the daemon is restarted. Simples (for me).

Eh, for completeness. First the quick shell line, which I should actually just make into a script:

Code: Select all

invoke-rc.d tt-rss-feeds-update stop ; su - tt-rss ; invoke-rc.d tt-rss-feeds-update start

That is making use of my hand-crafted (I didn't know there were any Debian packages for tt-rss at the time) init.d script:

Code: Select all

#! /bin/bash
### BEGIN INIT INFO
# Provides:          tt-rss-feeds-update
# Required-Start:    $local_fs $network $named $postgresql
# Required-Stop:     $local_fs $network $named $postgresql
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: tt-rss feeds update
# Description:       Tiny Tiny RSS feeds update daemon, to update feeds
#                    in the background.
### END INIT INFO

# Do NOT "set -e"

# PATH should only include /usr/* if it runs after the mountnfs.sh script
PATH=/sbin:/usr/sbin:/bin:/usr/bin
DESC="tt-rss feeds update"
NAME=tt-rss-feeds-update
DAEMON=/home/users/tt-rss/sites/live/update_daemon2.php
#DAEMON_ARGS=""
PIDFILE=/var/run/$NAME.pid
SCRIPTNAME=/etc/init.d/$NAME

# Exit if the package is not installed
[ -x "$DAEMON" ] || exit 0

# Fallback options values, we use these when
# the /etc/default/git-daemon file does not exist
RUN=no
USER=tt-rss
GROUP=tt-rss
TTRSS_HOME=/home/users/tt-rss/sites/live

# Read configuration variable file if it is present
[ -r /etc/default/$NAME ] && . /etc/default/$NAME

# If ADVANCED_OPTS is empty, use a default setting
#if [ "x$ADVANCED_OPTS" == "x" ];
#then
#       ADVANCED_OPTS="--base-path=$REPOSITORIES $REPOSITORIES"
#fi

DAEMON_ARGS=""

# Load the VERBOSE setting and other rcS variables
. /lib/init/vars.sh

# Define LSB log_* functions.
# Depend on lsb-base (>= 3.0-6) to ensure that this file is present.
. /lib/lsb/init-functions

#
# Function that starts the daemon/service
#
do_start()
{
        # Return
        #   0 if daemon has been started
        #   1 if daemon was already running
        #   2 if daemon could not be started
        start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --chuid ${USER}:${GROUP} --chdir ${TTRSS_HOME} --exec $DAEMON --test > /dev/null \
                || return 1
        start-stop-daemon --start --quiet --pidfile $PIDFILE --make-pidfile --background --chuid ${USER}:${GROUP} --chdir ${TTRSS_HOME} --exec $DAEMON -- \
                $DAEMON_ARGS \
                || return 2
        # Add code here, if necessary, that waits for the process to be ready
        # to handle requests from services started subsequently which depend
        # on this one.  As a last resort, sleep for some time.
}

#
# Function that stops the daemon/service
#
do_stop()
{
        # Return
        #   0 if daemon has been stopped
        #   1 if daemon was already stopped
        #   2 if daemon could not be stopped
        #   other if a failure occurred
        start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
        RETVAL="$?"
        [ "$RETVAL" = 2 ] && return 2
        # Wait for children to finish too if this is a daemon that forks
        # and if the daemon is only ever run from this initscript.
        # If the above conditions are not satisfied then add some other code
        # that waits for the process to drop all resources that could be
        # needed by services started subsequently.  A last resort is to
        # sleep for some time.
        start-stop-daemon --stop --quiet --oknodo --retry=0/30/KILL/5 --exec $DAEMON
        [ "$?" = 2 ] && return 2
        # Many daemons don't delete their pidfiles when they exit.
        rm -f $PIDFILE
        return "$RETVAL"
}

#
# Function that sends a SIGHUP to the daemon/service
#
do_reload() {
        #
        # If the daemon can reload its configuration without
        # restarting (for example, when it is sent a SIGHUP),
        # then implement that here.
        #
        start-stop-daemon --stop --signal 1 --quiet --pidfile $PIDFILE --name $NAME
        return 0
}

case "$1" in
  start)
        log_daemon_msg "Starting $DESC" "$NAME"
        do_start
        case "$?" in
                0|1) log_end_msg 0 ;;
                2) log_end_msg 1 ;;
        esac
        ;;
  stop)
        log_daemon_msg "Stopping $DESC" "$NAME"
        do_stop
        case "$?" in
                0|1) log_end_msg 0 ;;
                2) log_end_msg 1 ;;
        esac
        ;;
  #reload|force-reload)
        #
        # If do_reload() is not implemented then leave this commented out
        # and leave 'force-reload' as an alias for 'restart'.
        #
        #log_daemon_msg "Reloading $DESC" "$NAME"
        #do_reload
        #log_end_msg $?
        #;;
  restart|force-reload)
        #
        # If the "reload" option is implemented then remove the
        # 'force-reload' alias
        #
        log_daemon_msg "Restarting $DESC" "$NAME"
        do_stop
        case "$?" in
          0|1)
                do_start
                case "$?" in
                        0) log_end_msg 0 ;;
                        1) log_end_msg 1 ;; # Old process is still running
                        *) log_end_msg 1 ;; # Failed to start
                esac
                ;;
          *)
                # Failed to stop
                log_end_msg 1
                ;;
        esac
        ;;
  *)
        #echo "Usage: $SCRIPTNAME {start|stop|restart|reload|force-reload}" >&2
        echo "Usage: $SCRIPTNAME {start|stop|restart|force-reload}" >&2
        exit 3
        ;;
esac

:


And the ~tt-rss/bin/update-tt-rss-git:

Code: Select all

#!/bin/bash

cd ~/sites/tt-rss_git
git status
if [ "$?" != "0" ];
then
        echo "Uncommited changes?  Bailing"
        exit $?
fi
git checkout master
echo "'git checkout master' - done"
read
git pull
echo "'git pull' - done"
read
git fetch --tags
echo "'git fetch --tags' - done"
read
git checkout b-miggy
echo "'git checkout b-miggy' - done"
read
git merge master
echo "'git merge master' - done"
read
git tag -a -m "Miggy `date +%Y%m%d` - merge from upstream master" miggy-`date +%Y%m%d`
chmod -R og+rX . 2> /dev/null
echo '~/sites/tt-rss_git/cache'
ls -al ~/sites/tt-rss_git/cache
echo "Permissions fix - done"
read
./update.php --update-schema
echo "DB Schema update, if necessary - done"


The 'read's are so I can bail if anything threw errors (like a merge not going in cleanly). I also eyeball if config.php-dist has changed and merge necessary bits into config.php if needs be. I also try to keep an eye out for mass change of plugins which might indicate having to update out of tree ones. There's some paranoia about file perms in there too, as git does like to mess with those (I'm experimenting with 'umask 02' to avoid those issues).

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: How To Use Trunk Version?

Postby sleeper_service » 09 Jun 2013, 02:35

gbcox wrote:Not sure on the schema though... on the last change I had the update service already going and it seemed to work fine. Does the schema update change anything in the directory? I thought that only affected the db, and if that is the case why would you require the service to restart?

um, really?

stop update, do a pull, in the pull is a schema change. now, update has a new data layout for the database, you're starting the update, and without the checks that fox built in, you'd be trying to insert a new data structure into an old database....

fortunately, his software is smart enough to say "waaaaait a minute, you haven't updated the schema on the database, I'm not going to work" and dies.

that's why you do a schema update *before* you restart the update process.

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 09 Jun 2013, 03:47

stop update, do a pull, in the pull is a schema change. now, update has a new data layout for the database, you're starting the update, and without the checks that fox built in, you'd be trying to insert a new data structure into an old database....


Yeah, I knew the update-service had the code to not do the update - so it wasn't going to affect anything. It was sloppy as I mentioned above, so I've already changed it.

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 09 Jun 2013, 04:03

Still, you could get past that check before the update and be in the middle of other code just as something in the schema update causes DB changes. I'll stick with not relying on the underlying code being always 100% correct. It's not like it's difficult to stop and start the daemon.


Yes, as I mentioned in my previous reply, I decided it was sloppy so changed it - I'm not sure I agree with not relying on underlying code... if you didn't you'd have to write everything from scratch, which I think is counter-productive.

I also eyeball if config.php-dist has changed and merge necessary bits into config.php if needs be

Originally, I was going to do all that manually, and after considering comments from sleeper-service I thought I'd take a whack at automation with cfg_o_matic - which rebuilds config.php each time from the new config.php-dist...

Regarding the contrib-plugins, I think that is why they were moved out of the trunk... I threw them back in, which could cause grief I guess... I really only use the googleplus one, maybe I should change the code to allow folks to pick and choose... something I need to consider...

Again, thanks for the comments... makes me rethink stuff... which is good...

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: How To Use Trunk Version?

Postby sleeper_service » 09 Jun 2013, 05:32

gbcox wrote:
stop update, do a pull, in the pull is a schema change. now, update has a new data layout for the database, you're starting the update, and without the checks that fox built in, you'd be trying to insert a new data structure into an old database....


Yeah, I knew the update-service had the code to not do the update - so it wasn't going to affect anything. It was sloppy as I mentioned above, so I've already changed it.


I noticed, but it didn't look like you really understoond *why*... glad that you did.

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: How To Use Trunk Version?

Postby sleeper_service » 09 Jun 2013, 06:00

gbcox wrote:Originally, I was going to do all that manually, and after considering comments from sleeper-service I thought I'd take a whack at automation with cfg_o_matic - which rebuilds config.php each time from the new config.php-dist...

Again, thanks for the comments... makes me rethink stuff... which is good...


you can do the config update like this:

Code: Select all

#!/bin/sh
diff config.php-dist-old config.php-dist > config.php.changes
if [ -s config.php.changes ]; then 
  cp config.php-dist config.php
  cp config.php-dist config.php-dist-old
  patch < config.php.patch
  diff -c config.php-dist config.php > config.php.patch
  echo config changes:
  cat config.php.changes
fi


before your firstupdate, do this: cp config.php-dist config.php-dist-old and diff -c config.php-dist config.php > config.php.patch

then, you can just run that bit of code, call it what you like, or better just put it in the script you use to do the git pulls.

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 09 Jun 2013, 08:12

before your firstupdate, do this: cp config.php-dist config.php-dist-old and diff -c config.php-dist config.php > config.php.patch


I don't think patch will work for this. You've got three files which are different and you have to match, merge on each line using different sets of rules (Comments, define)... in any event, I followed your instructions and it bombed... :shock:
My code completely rebuilds config.php from the new dist - matching on define statements and ignoring the comments or lack thereof in config.php as the case may be. Local changes to defines are replaced by the new dist if it wasn't in the old dist. Not sure if patch can do all that. Additionally, it give a stoplight display of the status, green, yellow, red and blue.

Here is what I did for a test case.
copied config.php to config.php-dist
edited config.php-dist first few lines to replace dbtypes with "xxxxx"
edited config.php and deleted a bunch of comment lines
then: diff -c config.php-dist config.php > config.php.patch
now I edited config.php-dist adding some define statements, changing others, added comments
then followed the rest of your instructions

User avatar
sleeper_service
Bear Rating Overlord
Bear Rating Overlord
Posts: 884
Joined: 30 Mar 2013, 23:50
Location: Dallas, Texas

Re: How To Use Trunk Version?

Postby sleeper_service » 09 Jun 2013, 08:42

gbcox wrote:I don't think patch will work for this. You've got three files

actually, it works fine. you've only got two files, the *new* config.php-dist and the changes that need to be made to it, (the .patch file) which is generated at the end, and with the second command I said to use. the patch file only has your local customizations in it.
gbcox wrote:which are different and you have to match, merge on each line using different sets of rules (Comments, define)... in any event, I followed your instructions and it bombed... :shock:
My code completely rebuilds config.php from the new dist - matching on define statements and ignoring the comments or lack thereof in config.php as the case may be. Local changes to defines are replaced by the new dist if it wasn't in the old dist. Not sure if patch can do all that. Additionally, it give a stoplight display of the status, green, yellow, red and blue.

all the 'rebuilding from scratch' that you do isn't necessary, all you have to do is apply your local customizations to the source file. those can be found with a diff -c config.php-dist config.php. then when a new *dist file comes in, you can simply patch those changes into a copy of the new dist file. poof, instant customized dist file.
gbcox wrote:Here is what I did for a test case.
copied config.php to config.php-dist
edited config.php-dist first few lines to replace dbtypes with "xxxxx"
edited config.php and deleted a bunch of comment lines
then: diff -c config.php-dist config.php > config.php.patch
now I edited config.php-dist adding some define statements, changing others, added comments
then followed the rest of your instructions


yeah, I notice, you *didn't* do what I said to do to get started. not surprised you managed to make it 'bomb'.

at any rate, I know you like to "be uber cautious" (like starting up the update daemon without updating the schema on the db ;)) so, if you don't like my simple script, that's quite all right with me.

gbcox
Bear Rating Master
Bear Rating Master
Posts: 149
Joined: 25 Apr 2013, 04:52

Re: How To Use Trunk Version?

Postby gbcox » 09 Jun 2013, 09:21

all the 'rebuilding from scratch' that you do isn't necessary, all you have to do is apply your local customizations to the source file. those can be found with a diff -c config.php-dist config.php. then when a new *dist file comes in, you can simply patch those changes into a copy of the new dist file. poof, instant customized dist file.


More like kaboom... patch wasn't intended for what you're trying to do.


Return to “Development”

Who is online

Users browsing this forum: Google [Bot] and 1 guest